Build Error <mock requires Python '>=3.6' but the running Python is 2.7.15>

Hi,

I'm having trouble building the elastalert docker container on Ubuntu 18.04 using the instructions at https://github.com/bitsensor/elastalert. Initial builds failed with same error message mentioned in #147 but even after modifying the dockerfile to have "FROM alpine:3.8 as py-ea" instead of "FROM alpine:latest as py-ea" as recommended, builds are still failing but with a different error:

mock requires Python '>=3.6' but the running Python is 2.7.15
You are using pip version 10.0.1, however version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
The command '/bin/sh -c sed -i 's/jira>=1.0.10/jira>=1.0.10,<1.0.15/g' setup.py && python setup.py install && pip install -r requirements.txt' returned a non-zero code: 1
Makefile:6: recipe for target 'build' failed
make: *** [build] Error 1

Any help or insight on how to resolve this will be greatly appreciated.

@bcisse

Having the same issue myself on Ubuntu 18.04 after making the change "FROM alpine:latest as py-ea -> FROM alpine:3.8 as py-ea"

Did you have any luck resolving @bcisse ?

I was able to build version v0.2.1 of the elastalert docker image by modifying the dockerfile to install python3 as follows:

*****Dockerfile *****
FROM alpine:latest as py-ea
ARG ELASTALERT_VERSION=v0.2.1
...
CHANGE:
RUN apk add --update --no-cache ca-certificates openssl-dev openssl python2-dev python2 py2-pip py2-yaml libffi-dev gcc musl-dev wget && \

TO:
RUN apk add --update --no-cache ca-certificates openssl-dev openssl python3-dev python3 py3-pip py3-yaml libffi-dev gcc musl-dev wget &&
...
CHANGE:
COPY --from=py-ea /usr/lib/python2.7/site-packages /usr/lib/python2.7/site-packages

TO:
COPY --from=py-ea /usr/lib/python3.8/site-packages /usr/lib/python3.8/site-packages

RUN sed -i 's/jira>=1.0.10/jira>=1.0.10,<1.0.15/g' setup.py &&
python3 setup.py install &&
pip3 install -r requirements.txt

When I try to start a container using the image, it fails however after a few seconds. The elastalert indices are created in elasticsearch but container fails shortly after with following error:

16:37:18.170Z INFO elastalert-server: Config: No config.dev.json file was found in /opt/elastalert-server/config/config.dev.json.
16:37:18.173Z INFO elastalert-server: Config: Proceeding to look for normal config file.
16:37:18.174Z INFO elastalert-server: Config: A config file was found in /opt/elastalert-server/config/config.json. Using that config.
16:37:18.185Z INFO elastalert-server: Router: Listening for GET request on /.
16:37:18.186Z INFO elastalert-server: Router: Listening for GET request on /status.
16:37:18.186Z INFO elastalert-server: Router: Listening for GET request on /status/control/:action.
16:37:18.186Z INFO elastalert-server: Router: Listening for GET request on /status/errors.
16:37:18.186Z INFO elastalert-server: Router: Listening for GET request on /rules.
16:37:18.188Z INFO elastalert-server: Router: Listening for GET request on /rules/:id.
16:37:18.189Z INFO elastalert-server: Router: Listening for POST request on /rules/:id.
16:37:18.189Z INFO elastalert-server: Router: Listening for DELETE request on /rules/:id.
16:37:18.189Z INFO elastalert-server: Router: Listening for GET request on /templates.
16:37:18.189Z INFO elastalert-server: Router: Listening for GET request on /templates/:id.
16:37:18.189Z INFO elastalert-server: Router: Listening for POST request on /templates/:id.
16:37:18.193Z INFO elastalert-server: Router: Listening for DELETE request on /templates/:id.
16:37:18.194Z INFO elastalert-server: Router: Listening for POST request on /test.
16:37:18.194Z INFO elastalert-server: Router: Listening for GET request on /config.
16:37:18.194Z INFO elastalert-server: Router: Listening for POST request on /config.
16:37:18.194Z INFO elastalert-server: Router: Listening for POST request on /download.
16:37:18.194Z INFO elastalert-server: Router: Listening for GET request on /metadata/:type.
16:37:18.195Z INFO elastalert-server: Router: Listening for GET request on /mapping/:index.
16:37:18.195Z INFO elastalert-server: Router: Listening for POST request on /search/:index.
16:37:18.199Z INFO elastalert-server: ProcessController: Starting ElastAlert
16:37:18.199Z INFO elastalert-server: ProcessController: Creating index
16:37:21.448Z INFO elastalert-server:
ProcessController: Elastic Version:7
Mapping used for string:{'type': 'keyword'}

16:37:21.449Z ERROR elastalert-server:
ProcessController: Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/opt/elastalert/elastalert/create_index.py", line 275, in
main()
File "/opt/elastalert/elastalert/create_index.py", line 252, in main
es.indices.put_mapping(index=index, doc_type='elastalert', body=es_mapping)
File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/client/utils.py", line 84, in _wrapped
return func(*args, params=params, **kwargs)
File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/client/indices.py", line 339, in put_mapping
"PUT", _make_path(index, "_mapping", doc_type), params=params, body=body
File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/transport.py", line 353, in perform_request
timeout=timeout,
File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/connection/http_requests.py", line 143, in perform_request
self._raise_error(response.status_code, raw_data)
File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/connection/base.py", line 168, in _raise_error
status_code, error_message, additional_info
elasticsearch.exceptions.RequestError: RequestError(400, u'illegal_argument_exception', u'Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true.')

16:37:21.449Z ERROR elastalert-server: ProcessController: Index create exited with code 1
16:37:21.449Z WARN elastalert-server: ProcessController: ElastAlert will start but might not be able to save its data!
16:37:21.450Z INFO elastalert-server: ProcessController: Starting elastalert with arguments [none]
16:37:21.459Z INFO elastalert-server: ProcessController: Started Elastalert (PID: 36)
16:37:21.461Z INFO elastalert-server: Server: Server listening on port 3030
16:37:21.462Z INFO elastalert-server: Server: Websocket listening on port 3333
16:37:21.462Z INFO elastalert-server: Server: Server started
16:37:22.338Z ERROR elastalert-server:
ProcessController: Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main

16:37:22.338Z ERROR elastalert-server:
ProcessController: "main", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/opt/elastalert/elastalert/elastalert.py", line 1929, in
sys.exit(main(sys.argv[1:]))
File "/opt/elastalert/elastalert/elastalert.py", line 1925, in main

16:37:22.338Z ERROR elastalert-server:
ProcessController: client.start()
File "/opt/elastalert/elastalert/elastalert.py", line 1106, in start

16:37:22.338Z ERROR elastalert-server:
ProcessController: self.run_all_rules()
File "/opt/elastalert/elastalert/elastalert.py", line 1158, in run_all_rules

16:37:22.339Z ERROR elastalert-server:
ProcessController: self.send_pending_alerts()
File "/opt/elastalert/elastalert/elastalert.py", line 1534, in send_pending_alerts

16:37:22.339Z ERROR elastalert-server:
ProcessController: pending_alerts = self.find_recent_pending_alerts(self.alert_time_limit)
File "/opt/elastalert/elastalert/elastalert.py", line 1526, in find_recent_pending_alerts

16:37:22.340Z ERROR elastalert-server:
ProcessController: size=1000)
File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/client/utils.py", line 84, in _wrapped

16:37:22.340Z ERROR elastalert-server:
ProcessController: return func(*args, params=params, **kwargs)
TypeError: search() got an unexpected keyword argument 'doc_type'

16:37:22.384Z ERROR elastalert-server: ProcessController: ElastAlert exited with code 1
16:37:22.384Z INFO elastalert-server: Server: Stopping server
16:37:22.384Z INFO elastalert-server: ProcessController: ElastAlert is not running
16:37:22.384Z INFO elastalert-server: Server: Server stopped. Bye!

In my testing I saw this error a few times. fwiw, I have been using #157 and it works great on python3. I would also prehaps prefer a base ubuntu. See the commits for a possible solution

For those who want to stick to python 2.7 the below works

FROM alpine:3.8 as py-ea
ARG ELASTALERT_VERSION=v0.1.39
ENV ELASTALERT_VERSION=${ELASTALERT_VERSION}
# URL from which to download Elastalert.
ARG ELASTALERT_URL=https://github.com/Yelp/elastalert/archive/$ELASTALERT_VERSION.zip
ENV ELASTALERT_URL=${ELASTALERT_URL}
# Elastalert home directory full path.
ENV ELASTALERT_HOME /opt/elastalert

WORKDIR /opt

RUN apk add --update --no-cache ca-certificates openssl-dev openssl python2-dev python2 py2-pip py2-yaml libffi-dev gcc musl-dev wget && \
# Download and unpack Elastalert.
    wget -O elastalert.zip "${ELASTALERT_URL}" && \
    unzip elastalert.zip && \
    rm elastalert.zip && \
    mv e* "${ELASTALERT_HOME}"

WORKDIR "${ELASTALERT_HOME}"

# Install Elastalert.
# see: https://github.com/Yelp/elastalert/issues/1654
RUN sed -i 's/jira>=1.0.10/jira>=1.0.10,<1.0.15/g' setup.py && sed -i 's/mock>=2.0.0/mock==3.0.5/g' setup.py  && \
    python setup.py install && \
    pip install apscheduler>=3.3.0 aws-requests-auth>=0.3.0 blist>=1.3.6 boto3>=1.4.4 cffi>=1.11.5 configparser>=3.5.0 croniter>=0.3.16 elasticsearch>=7.0.0 envparse>=0.2.0 exotel>=0.1.3 "jira>=1.0.10,<1.0.15" jsonschema>=3.0.2 mock==3.0.5 prison>=0.1.2 py-zabbix==1.1.3 PyStaticConfiguration>=0.10.3 "python-dateutil>=2.6.0,<2.7.0" python-magic>=0.4.15 PyYAML>=5.1 requests>=2.0.0 stomp.py==4.1.23 texttable>=0.8.8 thehive4py>=1.4.4 twilio==6.0.0

FROM node:alpine
LABEL maintainer="BitSensor <dev@bitsensor.io>"
# Set timezone for this container
ENV TZ Etc/UTC

RUN apk add --update --no-cache curl tzdata python2 make libmagic

COPY --from=py-ea /usr/lib/python2.7/site-packages /usr/lib/python2.7/site-packages
COPY --from=py-ea /opt/elastalert /opt/elastalert
COPY --from=py-ea /usr/bin/elastalert* /usr/bin/

WORKDIR /opt/elastalert-server
COPY . /opt/elastalert-server

RUN npm install --production --quiet
COPY config/elastalert.yaml /opt/elastalert/config.yaml
COPY config/elastalert-test.yaml /opt/elastalert/config-test.yaml
COPY config/config.json config/config.json
COPY rule_templates/ /opt/elastalert/rule_templates
COPY elastalert_modules/ /opt/elastalert/elastalert_modules

EXPOSE 3030
ENTRYPOINT ["npm", "start"]

@jibsonline

@jibsonline Using your config I get this error. Am I missing something ?

Installed /usr/lib/python2.7/site-packages/croniter-0.3.37-py2.7.egg
Searching for configparser>=3.5.0
Reading https://pypi.org/simple/configparser/
Downloading https://files.pythonhosted.org/packages/3f/e7/9518720c56396179f8c63d7b5924c8463ed423828e54329be7f8cde5c364/configparser-5.0.1.tar.gz#sha256=005c3b102c96f4be9b8f40dafbd4997db003d07d1caa19f37808be8031475f2a
Best match: configparser 5.0.1
Processing configparser-5.0.1.tar.gz
Writing /tmp/easy_install-DYi9wW/configparser-5.0.1/setup.cfg
Running configparser-5.0.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-DYi9wW/configparser-5.0.1/egg-dist-tmp-rLCXdh
error: Setup script exited with error: 'egg_base' must be a directory name (got src)
The command '/bin/sh -c sed -i 's/jira>=1.0.10/jira>=1.0.10,<1.0.15/g' setup.py && sed -i 's/mock>=2.0.0/mock==3.0.5/g' setup.py && python setup.py install && pip install apscheduler>=3.3.0 aws-requests-auth>=0.3.0 blist>=1.3.6 boto3>=1.4.4 cffi>=1.11.5 configparser>=3.5.0 croniter>=0.3.16 elasticsearch>=7.0.0 envparse>=0.2.0 exotel>=0.1.3 "jira>=1.0.10,<1.0.15" jsonschema>=3.0.2 mock==3.0.5 prison>=0.1.2 py-zabbix==1.1.3 PyStaticConfiguration>=0.10.3 "python-dateutil>=2.6.0,<2.7.0" python-magic>=0.4.15 PyYAML>=5.1 requests>=2.0.0 stomp.py==4.1.23 texttable>=0.8.8 thehive4py>=1.4.4 twilio==6.0.0' returned a non-zero code: 1
Makefile:6: recipe for target 'build' failed
make: *** [build] Error 1

Navigation Menu

Repository navigation

Build Error <mock requires Python '>=3.6' but the running Python is 2.7.15> #161

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Build Error <mock requires Python '>=3.6' but the running Python is 2.7.15> #161

Description

Activity

Renny8 commented on Feb 24, 2020

bcisse commented on Feb 24, 2020

andrewbierbaum commented on Feb 24, 2020

jibsonline commented on Mar 3, 2020

saroyaj commented on Jan 6, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions