You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Server error: ConnectTimeout: HTTPConnectionPool #177
16:34:46.561Z ERROR elastalert-server:
ProcessController: ERROR:root:Traceback (most recent call last):
File "/opt/elastalert/elastalert/elastalert.py", line 1173, in run_all_rules
num_matches = self.run_rule(rule, endtime, self.starttime)
File "/opt/elastalert/elastalert/elastalert.py", line 870, in run_rule
if not self.run_query(rule, rule['starttime'], endtime):
File "/opt/elastalert/elastalert/elastalert.py", line 602, in run_query
data = self.get_hits(rule, start, end, index, scroll)
File "/opt/elastalert/elastalert/elastalert.py", line 331, in get_hits
if self.current_es.is_atleastsixsix():
File "elastalert/init.py", line 69, in is_atleastsixsix
major, minor = map(int, self.es_version.split(".")[:2])
File "elastalert/init.py", line 43, in es_version
self._es_version = self.info()['version']['number']
File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.2-py2.7.egg/elasticsearch/client/utils.py", line 84, in _wrapped
return func(*args, params=params, **kwargs)
File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.2-py2.7.egg/elasticsearch/client/init.py", line 259, in info
return self.transport.perform_request("GET", "/", params=params)
File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.2-py2.7.egg/elasticsearch/transport.py", line 353, in perform_request
timeout=timeout,
File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.2-py2.7.egg/elasticsearch/connection/http_requests.py", line 138, in perform_request
raise ConnectionTimeout("TIMEOUT", str(e), e)
ConnectionTimeout: ConnectionTimeout caused by - ConnectTimeout(HTTPConnectionPool(host='xxx.xxx.xxx.xxx', port=9200): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7efc1794ced0>, 'Connection to xxx.xxx.xxx.xxx timed out. (connect timeout=20)')))
ERROR:root:Uncaught exception running rule poolqq: ConnectionTimeout caused by - ConnectTimeout(HTTPConnectionPool(host='xxx.xxx.xxx.xxx', port=9200): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7efc1794ced0>, 'Connection to xxx.xxx.xxx.xxx timed out. (connect timeout=20)')))
If this error occurs on my server, the alarm will not be issued again, and only by restarting the server can the alarm be triggered again
However, 9200 is normal. May I ask why?
FROM bitsensor/elastalert:3.0.0-beta.0
USER root
RUN apk update && \
apk add bash curl && \
rm -rf /var/cache/apk/*
ADD elastalert/bin/elastalert-start.sh /usr/local/bin/
ADD elastalert/bin/elastic_search_status.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/elastalert-start.sh & \
chmod +x /usr/local/bin/elastic_search_status.sh
USER node
ENTRYPOINT ["/usr/local/bin/elastalert-start.sh"]
elastalert/bin/elastic_search_status.sh
#!/bin/bash
set -e
if [ $# -gt 0 ]; then
ES_URL="$1"
elif [[ -n $ELASTICSEARCH_URL ]]; then
ES_URL="$ELASTICSEARCH_URL"
elif [[ -n $ES_HOST ]] && [[ -n $ES_PORT ]]; then
ES_URL="http://$ES_HOST:$ES_PORT"
else
ES_URL="http://elasticsearch:9200"
fi
until [[ "$(curl -fsSL "$ES_URL/_cat/health?h=status" | sed -r 's/^[[:space:]]+|[[:space:]]+$//g')" =~ ^(yellow|green)$ ]]; do
# printf '+' >&2
sleep 1
done
echo "Elasticsearch is up and healthy at "$ES_URL"" >&2
elastalert/bin/elastalert-start.sh
#!/bin/bash
set -e
echo "Giving Elasticsearch at $ELASTICSEARCH_URL time to start..."
elastic_search_status.sh
echo "Starting ElastAlert!"
npm start
# NOTE: This config is used when testing a rule
# The elasticsearch hostname for metadata writeback
# Note that every rule can have its own elasticsearch host
es_host: elasticsearch
# The elasticsearch port
es_port: 9200
# This is the folder that contains the rule yaml files
# Any .yaml file will be loaded as a rule
rules_folder: rules
# How often ElastAlert will query elasticsearch
# The unit can be anything from weeks to seconds
run_every:
seconds: 5
# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
minutes: 1
# Optional URL prefix for elasticsearch
#es_url_prefix: elasticsearch
# Connect with TLS to elasticsearch
#use_ssl: True
# Verify TLS certificates
#verify_certs: True
# GET request with body is the default option for Elasticsearch.
# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
# See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
# for details
#es_send_get_body_as: GET
# Option basic-auth username and password for elasticsearch
#es_username: someusername
#es_password: somepassword
# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: elastalert_status
# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
days: 2
elastalert/config/elastalert.yml
# The elasticsearch hostname for metadata writeback
# Note that every rule can have its own elasticsearch host
es_host: elasticsearch
# The elasticsearch port
es_port: 9200
# This is the folder that contains the rule yaml files
# Any .yaml file will be loaded as a rule
rules_folder: rules
# How often ElastAlert will query elasticsearch
# The unit can be anything from weeks to seconds
run_every:
seconds: 5
# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
minutes: 1
# Optional URL prefix for elasticsearch
#es_url_prefix: elasticsearch
# Connect with TLS to elasticsearch
#use_ssl: True
# Verify TLS certificates
#verify_certs: True
# GET request with body is the default option for Elasticsearch.
# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
# See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
# for details
#es_send_get_body_as: GET
# Option basic-auth username and password for elasticsearch
#es_username: someusername
#es_password: somepassword
# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: elastalert_status
# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
days: 2
$ docker logs -f elastalert
Giving Elasticsearch at time to start...
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
Elasticsearch is up and healthy at http://elasticsearch:9200
Starting ElastAlert!
> @bitsensor/elastalert@3.0.0-beta.0 start /opt/elastalert-server
> sh ./scripts/start.sh
13:00:34.886Z INFO elastalert-server: Config: No config.dev.json file was found in /opt/elastalert-server/config/config.dev.json.
13:00:34.897Z INFO elastalert-server: Config: Proceeding to look for normal config file.
13:00:34.922Z INFO elastalert-server: Config: A config file was found in /opt/elastalert-server/config/config.json. Using that config.
13:00:35.013Z INFO elastalert-server: Router: Listening for GET request on /.
13:00:35.014Z INFO elastalert-server: Router: Listening for GET request on /status.
13:00:35.015Z INFO elastalert-server: Router: Listening for GET request on /status/control/:action.
13:00:35.015Z INFO elastalert-server: Router: Listening for GET request on /status/errors.
13:00:35.016Z INFO elastalert-server: Router: Listening for GET request on /rules.
13:00:35.018Z INFO elastalert-server: Router: Listening for GET request on /rules/:id.
13:00:35.027Z INFO elastalert-server: Router: Listening for POST request on /rules/:id.
13:00:35.027Z INFO elastalert-server: Router: Listening for DELETE request on /rules/:id.
13:00:35.031Z INFO elastalert-server: Router: Listening for GET request on /templates.
13:00:35.032Z INFO elastalert-server: Router: Listening for GET request on /templates/:id.
13:00:35.035Z INFO elastalert-server: Router: Listening for POST request on /templates/:id.
13:00:35.035Z INFO elastalert-server: Router: Listening for DELETE request on /templates/:id.
13:00:35.038Z INFO elastalert-server: Router: Listening for POST request on /test.
13:00:35.041Z INFO elastalert-server: Router: Listening for GET request on /config.
13:00:35.041Z INFO elastalert-server: Router: Listening for POST request on /config.
13:00:35.042Z INFO elastalert-server: Router: Listening for POST request on /download.
13:00:35.043Z INFO elastalert-server: Router: Listening for GET request on /metadata/:type.
13:00:35.043Z INFO elastalert-server: Router: Listening for GET request on /mapping/:index.
13:00:35.044Z INFO elastalert-server: Router: Listening for POST request on /search/:index.
13:00:35.048Z INFO elastalert-server: ProcessController: Starting ElastAlert
13:00:35.048Z INFO elastalert-server: ProcessController: Creating index
13:00:40.962Z INFO elastalert-server:
ProcessController: Elastic Version: 7.8.0
Reading Elastic 6 index mappings:
Reading index mapping 'es_mappings/6/silence.json'
Reading index mapping 'es_mappings/6/elastalert_status.json'
Reading index mapping 'es_mappings/6/elastalert.json'
Reading index mapping 'es_mappings/6/past_elastalert.json'
Reading index mapping 'es_mappings/6/elastalert_error.json'
New index elastalert_status created
Done!
13:00:40.962Z INFO elastalert-server: ProcessController: Index create exited with code 0
13:00:40.971Z INFO elastalert-server: ProcessController: Starting elastalert with arguments [none]
13:00:41.006Z INFO elastalert-server: ProcessController: Started Elastalert (PID: 229)
13:00:41.022Z INFO elastalert-server: Server: Server listening on port 3030
13:00:41.041Z INFO elastalert-server: Server: Websocket listening on port 3333
13:00:41.042Z INFO elastalert-server: Server: Server started
13:00:42.949Z INFO elastalert-server:
ProcessController: 0 rules loaded
13:00:53.024Z INFO elastalert-server: Routes: Successfully handled GET request for '/'.
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e3d3620a31b9 elastalert:3.0.0-beta.0 "/usr/local/bin/elas…" About a minute ago Up About a minute (healthy) 0.0.0.0:3030->3030/tcp, 0.0.0.0:3333->3333/tcp elastalert
d6ab98d03836 docker.elastic.co/kibana/kibana:7.8.0 "/usr/local/bin/dumb…" About a minute ago Up About a minute (healthy) 0.0.0.0:5601->5601/tcp kibana
14f1e131c0b9 docker.elastic.co/elasticsearch/elasticsearch:7.8.0 "/tini -- /usr/local…" About a minute ago Up About a minute (healthy) 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp elasticsearch
Ok, thank you for your timely reply. I deployed the official Docker, and there was no problem at the beginning (the alarm was normal). Within a week or a few days, the above error would appear, and then no alarm would be sent
Ok, thank you for your timely reply. I deployed the official Docker, and there was no problem at the beginning (the alarm was normal). Within a week or a few days, the above error would appear, and then no alarm would be sent
The mirror is bitsensor/elastalert: 3.0.0 - beta. 1
Elasticsearch version 7.2.0
Elastalert Server is available in the virtual machine
Elasticsearch Cluster in Aliyun (2 8C16G)
There's nothing wrong with Kibana when an error occurs
There's enough storage space
Python3 is used to run
I'm aware that Elasticsearch, Kibana and ElastAlert are running on the same server.
It seems to be connectivity issue. Can you connect to ES with curl -v http://localhost:9200 from docker host?
Is that server up and running with anything besides Elasticsearch, Kibana and ElatAlert installed?
Could you tell me the information of docker-compose.yml and the configuration file when docker container is started?
Since the error message is connection timeout, the size of the index to be searched may be very large, but I can not clearly say that it is only the information provided so far.
I found a setting in the ElastAlert documentation to change the timeout period.
Add es_conn_timeout to elastalert.yml of ElastAlert and set the setting value to a large number (default is 20)
Try restarting the Docker container for ElastAlert.
I'm very sorry, there is still an error, timeout time is 50, the reason for the problem is again, when timeout occurs, the alarm will not be triggered again, can you configure the retry mechanism?, I delete index again and restart the program, and everything is fine again,
16:01:30.692Z ERROR elastalert-server:
ProcessController: ERROR:root:Traceback (most recent call last):
File "/opt/elastalert/elastalert/elastalert.py", line 1173, in run_all_rules
num_matches = self.run_rule(rule, endtime, self.starttime)
File "/opt/elastalert/elastalert/elastalert.py", line 870, in run_rule
if not self.run_query(rule, rule['starttime'], endtime):
File "/opt/elastalert/elastalert/elastalert.py", line 602, in run_query
data = self.get_hits(rule, start, end, index, scroll)
File "/opt/elastalert/elastalert/elastalert.py", line 331, in get_hits
if self.current_es.is_atleastsixsix():
File "elastalert/init.py", line 69, in is_atleastsixsix
major, minor = map(int, self.es_version.split(".")[:2])
File "elastalert/init.py", line 43, in es_version
self._es_version = self.info()['version']['number']
File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.2-py2.7.egg/elasticsearch/client/utils.py", line 84, in _wrapped
return func(*args, params=params, **kwargs)
File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.2-py2.7.egg/elasticsearch/client/init.py", line 259, in info
return self.transport.perform_request("GET", "/", params=params)
File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.2-py2.7.egg/elasticsearch/transport.py", line 353, in perform_request
timeout=timeout,
File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.2-py2.7.egg/elasticsearch/connection/http_requests.py", line 138, in perform_request
raise ConnectionTimeout("TIMEOUT", str(e), e)
ConnectionTimeout: ConnectionTimeout caused by - ConnectTimeout(HTTPConnectionPool(host='xxx.xxx.xxx.xxx', port=9200): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7fb16464b710>, 'Connection to xxx.xxx.xxx.xxx timed out. (connect timeout=50)')))
ERROR:root:Uncaught exception running rule poolv: ConnectionTimeout caused by - ConnectTimeout(HTTPConnectionPool(host='xxx.xxx.xxx.xxx', port=9200): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7fb16464b710>, 'Connection to xxx.xxx.xxx.xxx timed out. (connect timeout=50)')))
The following error occurs when using the johnsusek/elastalert-server:latest startup:
08:56:06.576Z INFO elastalert-server: ProcessController: Creating index
08:56:08.914Z INFO elastalert-server:
ProcessController: Elastic Version:7
Mapping used for string:{'type': 'keyword'}
Index elastalert_status already exists. Skipping index creation.
08:56:08.915Z INFO elastalert-server: ProcessController: Index create exited with code 0
08:56:08.916Z INFO elastalert-server: ProcessController: Starting elastalert with arguments [none]
08:56:08.931Z INFO elastalert-server: ProcessController: Started Elastalert (PID: 47)
08:56:08.935Z INFO elastalert-server: Server: Server listening on port 3030
08:56:08.937Z INFO elastalert-server: Server: Websocket listening on port 3333
08:56:08.938Z INFO elastalert-server: Server: Server started
08:56:11.534Z ERROR elastalert-server:
ProcessController: Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/opt/elastalert/elastalert/elastalert.py", line 1929, in
08:56:11.535Z ERROR elastalert-server:
ProcessController: sys.exit(main(sys.argv[1:]))
File "/opt/elastalert/elastalert/elastalert.py", line 1925, in main
client.start()
File "/opt/elastalert/elastalert/elastalert.py", line 1106, in start
08:56:11.538Z ERROR elastalert-server:
ProcessController: self.run_all_rules()
File "/opt/elastalert/elastalert/elastalert.py", line 1158, in run_all_rules
self.send_pending_alerts()
File "/opt/elastalert/elastalert/elastalert.py", line 1534, in send_pending_alerts
08:56:11.538Z ERROR elastalert-server:
ProcessController: pending_alerts = self.find_recent_pending_alerts(self.alert_time_limit)
File "/opt/elastalert/elastalert/elastalert.py", line 1526, in find_recent_pending_alerts
08:56:11.540Z ERROR elastalert-server:
ProcessController: size=1000)
File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/client/utils.py", line 84, in _wrapped
08:56:11.604Z ERROR elastalert-server: ProcessController: ElastAlert exited with code 1
08:56:11.605Z INFO elastalert-server: Server: Stopping server
08:56:11.605Z INFO elastalert-server: ProcessController: ElastAlert is not running
08:56:11.606Z INFO elastalert-server: Server: Server stopped. Bye!
The following error occurs when using the johnsusek/elastalert-server:latest startup:
08:56:06.576Z INFO elastalert-server: ProcessController: Creating index
08:56:08.914Z INFO elastalert-server:
ProcessController: Elastic Version:7
Mapping used for string:{'type': 'keyword'}
Index elastalert_status already exists. Skipping index creation.
08:56:08.915Z INFO elastalert-server: ProcessController: Index create exited with code 0
08:56:08.916Z INFO elastalert-server: ProcessController: Starting elastalert with arguments [none]
08:56:08.931Z INFO elastalert-server: ProcessController: Started Elastalert (PID: 47)
08:56:08.935Z INFO elastalert-server: Server: Server listening on port 3030
08:56:08.937Z INFO elastalert-server: Server: Websocket listening on port 3333
08:56:08.938Z INFO elastalert-server: Server: Server started
08:56:11.534Z ERROR elastalert-server:
ProcessController: Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/opt/elastalert/elastalert/elastalert.py", line 1929, in
08:56:11.535Z ERROR elastalert-server:
ProcessController: sys.exit(main(sys.argv[1:]))
File "/opt/elastalert/elastalert/elastalert.py", line 1925, in main
client.start()
File "/opt/elastalert/elastalert/elastalert.py", line 1106, in start
08:56:11.538Z ERROR elastalert-server:
ProcessController: self.run_all_rules()
File "/opt/elastalert/elastalert/elastalert.py", line 1158, in run_all_rules
self.send_pending_alerts()
File "/opt/elastalert/elastalert/elastalert.py", line 1534, in send_pending_alerts
08:56:11.538Z ERROR elastalert-server:
ProcessController: pending_alerts = self.find_recent_pending_alerts(self.alert_time_limit)
File "/opt/elastalert/elastalert/elastalert.py", line 1526, in find_recent_pending_alerts
08:56:11.540Z ERROR elastalert-server:
ProcessController: size=1000)
File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/client/utils.py", line 84, in _wrapped
08:56:11.604Z ERROR elastalert-server: ProcessController: ElastAlert exited with code 1
08:56:11.605Z INFO elastalert-server: Server: Stopping server
08:56:11.605Z INFO elastalert-server: ProcessController: ElastAlert is not running
08:56:11.606Z INFO elastalert-server: Server: Server stopped. Bye!
It is difficult to answer without information about what you did. Because it’s not Esper
FROM johnsusek/elastalert-server:latest
USER root
RUN apk update && \
apk add bash curl && \
rm -rf /var/cache/apk/*
ADD elastalert/bin/elastalert-start.sh /usr/local/bin/
ADD elastalert/bin/elastic_search_status.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/elastalert-start.sh & \
chmod +x /usr/local/bin/elastic_search_status.sh
USER node
ENTRYPOINT ["/usr/local/bin/elastalert-start.sh"]
docker-compose up -d
docker logs -f elastalert
Giving Elasticsearch at time to start...
Elasticsearch is up and healthy at http://elasticsearch:9200
Starting ElastAlert!
> @bitsensor/elastalert@0.0.14 start /opt/elastalert-server
> sh ./scripts/start.sh
13:14:45.276Z INFO elastalert-server: Config: No config.dev.json file was found in /opt/elastalert-server/config/config.dev.json.
13:14:45.281Z INFO elastalert-server: Config: Proceeding to look for normal config file.
13:14:45.282Z INFO elastalert-server: Config: A config file was found in /opt/elastalert-server/config/config.json. Using that config.
13:14:45.288Z INFO elastalert-server: Router: Listening for GET request on /.
13:14:45.289Z INFO elastalert-server: Router: Listening for GET request on /status.
13:14:45.289Z INFO elastalert-server: Router: Listening for GET request on /status/errors.
13:14:45.289Z INFO elastalert-server: Router: Listening for GET request on /rules.
13:14:45.291Z INFO elastalert-server: Router: Listening for GET request on /rules/:id*.
13:14:45.291Z INFO elastalert-server: Router: Listening for POST request on /rules/:id*.
13:14:45.291Z INFO elastalert-server: Router: Listening for DELETE request on /rules/:id*.
13:14:45.291Z INFO elastalert-server: Router: Listening for GET request on /templates.
13:14:45.292Z INFO elastalert-server: Router: Listening for GET request on /templates/:id*.
13:14:45.292Z INFO elastalert-server: Router: Listening for POST request on /templates/:id*.
13:14:45.292Z INFO elastalert-server: Router: Listening for DELETE request on /templates/:id*.
13:14:45.293Z INFO elastalert-server: Router: Listening for PUT request on /folders/:type/:path*.
13:14:45.293Z INFO elastalert-server: Router: Listening for DELETE request on /folders/:type/:path*.
13:14:45.293Z INFO elastalert-server: Router: Listening for POST request on /test.
13:14:45.293Z INFO elastalert-server: Router: Listening for POST request on /silence/:path*.
13:14:45.293Z INFO elastalert-server: Router: Listening for GET request on /config.
13:14:45.293Z INFO elastalert-server: Router: Listening for POST request on /config.
13:14:45.294Z INFO elastalert-server: Router: Listening for POST request on /download.
13:14:45.294Z INFO elastalert-server: Router: Listening for GET request on /metadata/elastalert.
13:14:45.294Z INFO elastalert-server: Router: Listening for GET request on /metadata/elastalert_status.
13:14:45.294Z INFO elastalert-server: Router: Listening for GET request on /metadata/silence.
13:14:45.294Z INFO elastalert-server: Router: Listening for GET request on /metadata/elastalert_error.
13:14:45.295Z INFO elastalert-server: Router: Listening for GET request on /metadata/past_elastalert.
13:14:45.295Z INFO elastalert-server: Router: Listening for GET request on /indices.
13:14:45.295Z INFO elastalert-server: Router: Listening for GET request on /mapping/:index.
13:14:45.295Z INFO elastalert-server: Router: Listening for POST request on /search/:index.
13:14:45.295Z INFO elastalert-server: Router: Listening for GET request on /config.
13:14:45.299Z INFO elastalert-server: ProcessController: Starting ElastAlert
13:14:45.299Z INFO elastalert-server: ProcessController: Creating index
13:14:49.019Z INFO elastalert-server:
ProcessController: Elastic Version: 7.8.0
Reading Elastic 6 index mappings:
Reading index mapping 'es_mappings/6/silence.json'
Reading index mapping 'es_mappings/6/elastalert_status.json'
Reading index mapping 'es_mappings/6/elastalert.json'
Reading index mapping 'es_mappings/6/past_elastalert.json'
Reading index mapping 'es_mappings/6/elastalert_error.json'
New index elastalert_status created
Done!
13:14:49.019Z INFO elastalert-server: ProcessController: Index create exited with code 0
13:14:49.020Z INFO elastalert-server: ProcessController: Starting elastalert with arguments [none]
13:14:49.031Z INFO elastalert-server: ProcessController: Started Elastalert (PID: 44)
13:14:49.035Z INFO elastalert-server: Server: Server listening on port 3030
13:14:49.036Z INFO elastalert-server: Server: Websocket listening on port 3333
13:14:49.036Z INFO elastalert-server: Server: Server started
13:15:11.959Z INFO elastalert-server: Routes: Successfully handled GET request for '/'.
FROM johnsusek/elastalert-server:latest
USER root
RUN apk update && \
apk add bash curl && \
rm -rf /var/cache/apk/*
ADD elastalert/bin/elastalert-start.sh /usr/local/bin/
ADD elastalert/bin/elastic_search_status.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/elastalert-start.sh & \
chmod +x /usr/local/bin/elastic_search_status.sh
USER node
ENTRYPOINT ["/usr/local/bin/elastalert-start.sh"]
docker-compose up -d
docker logs -f elastalert
Giving Elasticsearch at time to start...
Elasticsearch is up and healthy at http://elasticsearch:9200
Starting ElastAlert!
> @bitsensor/elastalert@0.0.14 start /opt/elastalert-server
> sh ./scripts/start.sh
13:14:45.276Z INFO elastalert-server: Config: No config.dev.json file was found in /opt/elastalert-server/config/config.dev.json.
13:14:45.281Z INFO elastalert-server: Config: Proceeding to look for normal config file.
13:14:45.282Z INFO elastalert-server: Config: A config file was found in /opt/elastalert-server/config/config.json. Using that config.
13:14:45.288Z INFO elastalert-server: Router: Listening for GET request on /.
13:14:45.289Z INFO elastalert-server: Router: Listening for GET request on /status.
13:14:45.289Z INFO elastalert-server: Router: Listening for GET request on /status/errors.
13:14:45.289Z INFO elastalert-server: Router: Listening for GET request on /rules.
13:14:45.291Z INFO elastalert-server: Router: Listening for GET request on /rules/:id*.
13:14:45.291Z INFO elastalert-server: Router: Listening for POST request on /rules/:id*.
13:14:45.291Z INFO elastalert-server: Router: Listening for DELETE request on /rules/:id*.
13:14:45.291Z INFO elastalert-server: Router: Listening for GET request on /templates.
13:14:45.292Z INFO elastalert-server: Router: Listening for GET request on /templates/:id*.
13:14:45.292Z INFO elastalert-server: Router: Listening for POST request on /templates/:id*.
13:14:45.292Z INFO elastalert-server: Router: Listening for DELETE request on /templates/:id*.
13:14:45.293Z INFO elastalert-server: Router: Listening for PUT request on /folders/:type/:path*.
13:14:45.293Z INFO elastalert-server: Router: Listening for DELETE request on /folders/:type/:path*.
13:14:45.293Z INFO elastalert-server: Router: Listening for POST request on /test.
13:14:45.293Z INFO elastalert-server: Router: Listening for POST request on /silence/:path*.
13:14:45.293Z INFO elastalert-server: Router: Listening for GET request on /config.
13:14:45.293Z INFO elastalert-server: Router: Listening for POST request on /config.
13:14:45.294Z INFO elastalert-server: Router: Listening for POST request on /download.
13:14:45.294Z INFO elastalert-server: Router: Listening for GET request on /metadata/elastalert.
13:14:45.294Z INFO elastalert-server: Router: Listening for GET request on /metadata/elastalert_status.
13:14:45.294Z INFO elastalert-server: Router: Listening for GET request on /metadata/silence.
13:14:45.294Z INFO elastalert-server: Router: Listening for GET request on /metadata/elastalert_error.
13:14:45.295Z INFO elastalert-server: Router: Listening for GET request on /metadata/past_elastalert.
13:14:45.295Z INFO elastalert-server: Router: Listening for GET request on /indices.
13:14:45.295Z INFO elastalert-server: Router: Listening for GET request on /mapping/:index.
13:14:45.295Z INFO elastalert-server: Router: Listening for POST request on /search/:index.
13:14:45.295Z INFO elastalert-server: Router: Listening for GET request on /config.
13:14:45.299Z INFO elastalert-server: ProcessController: Starting ElastAlert
13:14:45.299Z INFO elastalert-server: ProcessController: Creating index
13:14:49.019Z INFO elastalert-server:
ProcessController: Elastic Version: 7.8.0
Reading Elastic 6 index mappings:
Reading index mapping 'es_mappings/6/silence.json'
Reading index mapping 'es_mappings/6/elastalert_status.json'
Reading index mapping 'es_mappings/6/elastalert.json'
Reading index mapping 'es_mappings/6/past_elastalert.json'
Reading index mapping 'es_mappings/6/elastalert_error.json'
New index elastalert_status created
Done!
13:14:49.019Z INFO elastalert-server: ProcessController: Index create exited with code 0
13:14:49.020Z INFO elastalert-server: ProcessController: Starting elastalert with arguments [none]
13:14:49.031Z INFO elastalert-server: ProcessController: Started Elastalert (PID: 44)
13:14:49.035Z INFO elastalert-server: Server: Server listening on port 3030
13:14:49.036Z INFO elastalert-server: Server: Websocket listening on port 3333
13:14:49.036Z INFO elastalert-server: Server: Server started
13:15:11.959Z INFO elastalert-server: Routes: Successfully handled GET request for '/'.
“
ADD elastalert/bin/elastalert-start.sh /usr/local/bin/
ADD elastalert/bin/elastic_search_status.sh /usr/local/bin/
”
Where are these two files?I didn't find
FROM johnsusek/elastalert-server:latest
USER root
RUN apk update && \
apk add bash curl && \
rm -rf /var/cache/apk/*
ADD elastalert/bin/elastalert-start.sh /usr/local/bin/
ADD elastalert/bin/elastic_search_status.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/elastalert-start.sh & \
chmod +x /usr/local/bin/elastic_search_status.sh
USER node
ENTRYPOINT ["/usr/local/bin/elastalert-start.sh"]
docker-compose up -d
docker logs -f elastalert
Giving Elasticsearch at time to start...
Elasticsearch is up and healthy at http://elasticsearch:9200
Starting ElastAlert!
> @bitsensor/elastalert@0.0.14 start /opt/elastalert-server
> sh ./scripts/start.sh
13:14:45.276Z INFO elastalert-server: Config: No config.dev.json file was found in /opt/elastalert-server/config/config.dev.json.
13:14:45.281Z INFO elastalert-server: Config: Proceeding to look for normal config file.
13:14:45.282Z INFO elastalert-server: Config: A config file was found in /opt/elastalert-server/config/config.json. Using that config.
13:14:45.288Z INFO elastalert-server: Router: Listening for GET request on /.
13:14:45.289Z INFO elastalert-server: Router: Listening for GET request on /status.
13:14:45.289Z INFO elastalert-server: Router: Listening for GET request on /status/errors.
13:14:45.289Z INFO elastalert-server: Router: Listening for GET request on /rules.
13:14:45.291Z INFO elastalert-server: Router: Listening for GET request on /rules/:id*.
13:14:45.291Z INFO elastalert-server: Router: Listening for POST request on /rules/:id*.
13:14:45.291Z INFO elastalert-server: Router: Listening for DELETE request on /rules/:id*.
13:14:45.291Z INFO elastalert-server: Router: Listening for GET request on /templates.
13:14:45.292Z INFO elastalert-server: Router: Listening for GET request on /templates/:id*.
13:14:45.292Z INFO elastalert-server: Router: Listening for POST request on /templates/:id*.
13:14:45.292Z INFO elastalert-server: Router: Listening for DELETE request on /templates/:id*.
13:14:45.293Z INFO elastalert-server: Router: Listening for PUT request on /folders/:type/:path*.
13:14:45.293Z INFO elastalert-server: Router: Listening for DELETE request on /folders/:type/:path*.
13:14:45.293Z INFO elastalert-server: Router: Listening for POST request on /test.
13:14:45.293Z INFO elastalert-server: Router: Listening for POST request on /silence/:path*.
13:14:45.293Z INFO elastalert-server: Router: Listening for GET request on /config.
13:14:45.293Z INFO elastalert-server: Router: Listening for POST request on /config.
13:14:45.294Z INFO elastalert-server: Router: Listening for POST request on /download.
13:14:45.294Z INFO elastalert-server: Router: Listening for GET request on /metadata/elastalert.
13:14:45.294Z INFO elastalert-server: Router: Listening for GET request on /metadata/elastalert_status.
13:14:45.294Z INFO elastalert-server: Router: Listening for GET request on /metadata/silence.
13:14:45.294Z INFO elastalert-server: Router: Listening for GET request on /metadata/elastalert_error.
13:14:45.295Z INFO elastalert-server: Router: Listening for GET request on /metadata/past_elastalert.
13:14:45.295Z INFO elastalert-server: Router: Listening for GET request on /indices.
13:14:45.295Z INFO elastalert-server: Router: Listening for GET request on /mapping/:index.
13:14:45.295Z INFO elastalert-server: Router: Listening for POST request on /search/:index.
13:14:45.295Z INFO elastalert-server: Router: Listening for GET request on /config.
13:14:45.299Z INFO elastalert-server: ProcessController: Starting ElastAlert
13:14:45.299Z INFO elastalert-server: ProcessController: Creating index
13:14:49.019Z INFO elastalert-server:
ProcessController: Elastic Version: 7.8.0
Reading Elastic 6 index mappings:
Reading index mapping 'es_mappings/6/silence.json'
Reading index mapping 'es_mappings/6/elastalert_status.json'
Reading index mapping 'es_mappings/6/elastalert.json'
Reading index mapping 'es_mappings/6/past_elastalert.json'
Reading index mapping 'es_mappings/6/elastalert_error.json'
New index elastalert_status created
Done!
13:14:49.019Z INFO elastalert-server: ProcessController: Index create exited with code 0
13:14:49.020Z INFO elastalert-server: ProcessController: Starting elastalert with arguments [none]
13:14:49.031Z INFO elastalert-server: ProcessController: Started Elastalert (PID: 44)
13:14:49.035Z INFO elastalert-server: Server: Server listening on port 3030
13:14:49.036Z INFO elastalert-server: Server: Websocket listening on port 3333
13:14:49.036Z INFO elastalert-server: Server: Server started
13:15:11.959Z INFO elastalert-server: Routes: Successfully handled GET request for '/'.
“
ADD elastalert/bin/elastalert-start.sh /usr/local/bin/
ADD elastalert/bin/elastic_search_status.sh /usr/local/bin/
”
Where are these two files?I didn't find
Activity
nsano-rururu commentedon Jun 23, 2020
It is a sample
Elasticsearch 7.8.0
Kibana 7.8.0
bitsensor/elastalert:3.0.0-beta.0
docker-compose.yml
kibana/config/kibana.yml
es/config/elasticsearch.yml
Dockerfiles/Dockerfile-elastalert
elastalert/bin/elastic_search_status.sh
elastalert/bin/elastalert-start.sh
elastalert/config/config.json
elastalert/config/elastalert-test.yml
elastalert/config/elastalert.yml
kakaNo1 commentedon Jun 24, 2020
Ok, thank you for your timely reply. I deployed the official Docker, and there was no problem at the beginning (the alarm was normal). Within a week or a few days, the above error would appear, and then no alarm would be sent
xuanyuanaosheng commentedon Jun 24, 2020
@nsano-rururu The elastalert now is only support the python 3.
nsano-rururu commentedon Jun 24, 2020
・Which Docker image are you using?
Babel 6 → 7
bug fix
・Cloud (AWS1, Azure, etc.), physical server?
・What are the server specifications? (CPU, number of CPU cores, memory, etc.)
・Are you using Kibana without problems when an error occurs?
・Is Elasticsearch running on docker ps?
・Did you check docker logs to see if there are any errors in the Elasticsearch container?
・How much memory is available by executing the "free -h" command when an error occurs?
nsano-rururu commentedon Jun 24, 2020
Yeah, right,
kakaNo1 commentedon Jun 28, 2020
The mirror is bitsensor/elastalert: 3.0.0 - beta. 1
Elasticsearch version 7.2.0
Elastalert Server is available in the virtual machine
Elasticsearch Cluster in Aliyun (2 8C16G)
There's nothing wrong with Kibana when an error occurs
There's enough storage space
Python3 is used to run
nsano-rururu commentedon Jun 28, 2020
I've never run ElastAlert on an Elasticsearch Cluster, so my current knowledge may not tell...
nsano-rururu commentedon Jun 28, 2020
I'm aware that Elasticsearch, Kibana and ElastAlert are running on the same server.
It seems to be connectivity issue. Can you connect to ES with curl -v http://localhost:9200 from docker host?
nsano-rururu commentedon Jun 28, 2020
Is that server up and running with anything besides Elasticsearch, Kibana and ElatAlert installed?
Could you tell me the information of docker-compose.yml and the configuration file when docker container is started?
nsano-rururu commentedon Jun 28, 2020
Since the error message is connection timeout, the size of the index to be searched may be very large, but I can not clearly say that it is only the information provided so far.
nsano-rururu commentedon Jun 28, 2020
I found a setting in the ElastAlert documentation to change the timeout period.
Add es_conn_timeout to elastalert.yml of ElastAlert and set the setting value to a large number (default is 20)
Try restarting the Docker container for ElastAlert.
kakaNo1 commentedon Jun 29, 2020
Thank you for your timely reply. I will try today. My server is running in K8S, do you need to check YAML
kakaNo1 commentedon Jun 29, 2020
I set it up in config.yaml
Es_conn_timeout: 50
Let's see if we can make the same mistakes tomorrow
kakaNo1 commentedon Jul 3, 2020
I'm very sorry, there is still an error, timeout time is 50, the reason for the problem is again, when timeout occurs, the alarm will not be triggered again, can you configure the retry mechanism?, I delete index again and restart the program, and everything is fine again,
16:01:30.692Z ERROR elastalert-server:
ProcessController: ERROR:root:Traceback (most recent call last):
File "/opt/elastalert/elastalert/elastalert.py", line 1173, in run_all_rules
num_matches = self.run_rule(rule, endtime, self.starttime)
File "/opt/elastalert/elastalert/elastalert.py", line 870, in run_rule
if not self.run_query(rule, rule['starttime'], endtime):
File "/opt/elastalert/elastalert/elastalert.py", line 602, in run_query
data = self.get_hits(rule, start, end, index, scroll)
File "/opt/elastalert/elastalert/elastalert.py", line 331, in get_hits
if self.current_es.is_atleastsixsix():
File "elastalert/init.py", line 69, in is_atleastsixsix
major, minor = map(int, self.es_version.split(".")[:2])
File "elastalert/init.py", line 43, in es_version
self._es_version = self.info()['version']['number']
File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.2-py2.7.egg/elasticsearch/client/utils.py", line 84, in _wrapped
return func(*args, params=params, **kwargs)
File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.2-py2.7.egg/elasticsearch/client/init.py", line 259, in info
return self.transport.perform_request("GET", "/", params=params)
File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.2-py2.7.egg/elasticsearch/transport.py", line 353, in perform_request
timeout=timeout,
File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.2-py2.7.egg/elasticsearch/connection/http_requests.py", line 138, in perform_request
raise ConnectionTimeout("TIMEOUT", str(e), e)
ConnectionTimeout: ConnectionTimeout caused by - ConnectTimeout(HTTPConnectionPool(host='xxx.xxx.xxx.xxx', port=9200): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7fb16464b710>, 'Connection to xxx.xxx.xxx.xxx timed out. (connect timeout=50)')))
nsano-rururu commentedon Jul 3, 2020
I need to see if I have the same issue with a Docker image that uses a newer version of ElastAlert
servercentral/elastalert・・・ElastAlert 0.2.1
https://hub.docker.com/r/servercentral/elastalert
johnsusek/elastalert-server・・・ElastAlert 0.2.4
https://hub.docker.com/r/johnsusek/elastalert-server
kakaNo1 commentedon Jul 6, 2020
Hello, I noticed that the official server mirror is Bitsensor /elastalert
May I ask that the two you recommend are the same?
kakaNo1 commentedon Jul 6, 2020
The following error occurs when using the johnsusek/elastalert-server:latest startup:
08:56:06.576Z INFO elastalert-server: ProcessController: Creating index
08:56:08.914Z INFO elastalert-server:
ProcessController: Elastic Version:7
Mapping used for string:{'type': 'keyword'}
Index elastalert_status already exists. Skipping index creation.
08:56:08.915Z INFO elastalert-server: ProcessController: Index create exited with code 0
08:56:08.916Z INFO elastalert-server: ProcessController: Starting elastalert with arguments [none]
08:56:08.931Z INFO elastalert-server: ProcessController: Started Elastalert (PID: 47)
08:56:08.935Z INFO elastalert-server: Server: Server listening on port 3030
08:56:08.937Z INFO elastalert-server: Server: Websocket listening on port 3333
08:56:08.938Z INFO elastalert-server: Server: Server started
08:56:11.534Z ERROR elastalert-server:
ProcessController: Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/opt/elastalert/elastalert/elastalert.py", line 1929, in
08:56:11.535Z ERROR elastalert-server:
ProcessController: sys.exit(main(sys.argv[1:]))
File "/opt/elastalert/elastalert/elastalert.py", line 1925, in main
client.start()
File "/opt/elastalert/elastalert/elastalert.py", line 1106, in start
08:56:11.538Z ERROR elastalert-server:
ProcessController: self.run_all_rules()
File "/opt/elastalert/elastalert/elastalert.py", line 1158, in run_all_rules
self.send_pending_alerts()
File "/opt/elastalert/elastalert/elastalert.py", line 1534, in send_pending_alerts
08:56:11.538Z ERROR elastalert-server:
ProcessController: pending_alerts = self.find_recent_pending_alerts(self.alert_time_limit)
File "/opt/elastalert/elastalert/elastalert.py", line 1526, in find_recent_pending_alerts
08:56:11.540Z ERROR elastalert-server:
ProcessController: size=1000)
File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/client/utils.py", line 84, in _wrapped
08:56:11.541Z ERROR elastalert-server:
ProcessController: return func(*args, params=params, **kwargs)
TypeError: search() got an unexpected keyword argument 'doc_type'
08:56:11.604Z ERROR elastalert-server: ProcessController: ElastAlert exited with code 1
08:56:11.605Z INFO elastalert-server: Server: Stopping server
08:56:11.605Z INFO elastalert-server: ProcessController: ElastAlert is not running
08:56:11.606Z INFO elastalert-server: Server: Server stopped. Bye!
nsano-rururu commentedon Jul 6, 2020
Bitsensor/elastalert will no longer be updated
nsano-rururu commentedon Jul 6, 2020
It is difficult to answer without information about what you did. Because it’s not Esper
nsano-rururu commentedon Jul 6, 2020
kakaNo1 commentedon Jul 7, 2020
“
ADD elastalert/bin/elastalert-start.sh /usr/local/bin/
ADD elastalert/bin/elastic_search_status.sh /usr/local/bin/
”
Where are these two files?I didn't find
nsano-rururu commentedon Jul 7, 2020
I can not do it. I have no time to bite. I'm sorry. goodbye.
nsano-rururu commentedon Jul 7, 2020
The file is in the first answer.