For CeleryExecutor, why do we need to install mysql? #66

Activity

1. mistercrunch commented on Jun 23, 2015

   

   mistercrunch

   on Jun 23, 2015

   Member

   More actions

   stack trace pls
2. r39132 commented on Jun 23, 2015

   

   r39132

   on Jun 23, 2015

   ContributorAuthor

   More actions

   Sorry. This was reported by a colleague of mine and he does not have the stack trace. It is possibly pilot error. Closing unless we can reproduce.
3. r39132

   closed this as completedon Jun 23, 2015
4. dermorz commented on Jul 24, 2015

   

   dermorz

   on Jul 24, 2015

   More actions

   I think it was the stack trace below. This happens if you run

   $ airflow worker
   

   on a freshly installed airflow with the defaults

   broker_url = sqla+mysql://airflow:airflow@localhost:3306/airflow
   celery_result_backend = db+mysql://airflow:airflow@localhost:3306/airflow
   

   in the configuration file (with no mysql server running). The Exception does not raise if those values are properly set to a running rabbitmq instance.

    -------------- celery@airflow v3.1.18 (Cipater)
   ---- **** -----
   --- * ***  * -- Linux-3.2.0-4-amd64-x86_64-with-debian-7.5
   -- * - **** ---
   - ** ---------- [config]
   - ** ---------- .> app:         airflow.executors.celery_executor:0x345a550
   - ** ---------- .> transport:   sqla+mysql://airflow:airflow@localhost:3306/airflow
   - ** ---------- .> results:     db+mysql://airflow:airflow@localhost:3306/airflow
   - *** --- * --- .> concurrency: 16 (prefork)
   -- ******* ----
   --- ***** ----- [queues]
    -------------- .> default          exchange=celery(direct) key=celery
   
   
   [2015-07-24 13:37:36,009: ERROR/MainProcess] Unrecoverable error: ImportError('No module named MySQLdb',)
   Traceback (most recent call last):
     File "/home/vagrant/env/local/lib/python2.7/site-packages/celery/worker/__init__.py", line 206, in start
       self.blueprint.start(self)
     File "/home/vagrant/env/local/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
       step.start(parent)
     File "/home/vagrant/env/local/lib/python2.7/site-packages/celery/bootsteps.py", line 374, in start
       return self.obj.start()
     File "/home/vagrant/env/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 278, in start
       blueprint.start(self)
     File "/home/vagrant/env/local/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
       step.start(parent)
     File "/home/vagrant/env/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 608, in start
       c.connection, on_decode_error=c.on_decode_error,
     File "/home/vagrant/env/local/lib/python2.7/site-packages/celery/app/amqp.py", line 375, in __init__
       **kw
     File "/home/vagrant/env/local/lib/python2.7/site-packages/kombu/messaging.py", line 363, in __init__
       self.revive(self.channel)
     File "/home/vagrant/env/local/lib/python2.7/site-packages/kombu/messaging.py", line 375, in revive
       self.declare()
     File "/home/vagrant/env/local/lib/python2.7/site-packages/kombu/messaging.py", line 385, in declare
       queue.declare()
     File "/home/vagrant/env/local/lib/python2.7/site-packages/kombu/entity.py", line 515, in declare
       self.queue_declare(nowait, passive=False)
     File "/home/vagrant/env/local/lib/python2.7/site-packages/kombu/entity.py", line 541, in queue_declare
       nowait=nowait)
     File "/home/vagrant/env/local/lib/python2.7/site-packages/kombu/transport/virtual/__init__.py", line 446, in queue_declare
       self._new_queue(queue, **kwargs)
     File "/home/vagrant/env/local/lib/python2.7/site-packages/kombu/transport/sqlalchemy/__init__.py", line 80, in _new_queue
       self._get_or_create(queue)
     File "/home/vagrant/env/local/lib/python2.7/site-packages/kombu/transport/sqlalchemy/__init__.py", line 68, in _get_or_create
       obj = self.session.query(self.queue_cls) \
     File "/home/vagrant/env/local/lib/python2.7/site-packages/kombu/transport/sqlalchemy/__init__.py", line 63, in session
       _, Session = self._open()
     File "/home/vagrant/env/local/lib/python2.7/site-packages/kombu/transport/sqlalchemy/__init__.py", line 54, in _open
       engine = self._engine_from_config()
     File "/home/vagrant/env/local/lib/python2.7/site-packages/kombu/transport/sqlalchemy/__init__.py", line 49, in _engine_from_config
       return create_engine(conninfo.hostname, **transport_options)
     File "/home/vagrant/env/local/lib/python2.7/site-packages/sqlalchemy/engine/__init__.py", line 386, in create_engine
       return strategy.create(*args, **kwargs)
     File "/home/vagrant/env/local/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 75, in create
       dbapi = dialect_cls.dbapi(**dbapi_args)
     File "/home/vagrant/env/local/lib/python2.7/site-packages/sqlalchemy/dialects/mysql/mysqldb.py", line 92, in dbapi
       return __import__('MySQLdb')
   ImportError: No module named MySQLdb
   Starting flask
    * Running on http://0.0.0.0:8793/ (Press CTRL+C to quit)
   
5. mistercrunch commented on Jul 24, 2015

   

   mistercrunch

   on Jul 24, 2015

   Member

   More actions

   Seems like the expected behavior. Please reopen if you don't think it is.
6. mistercrunch

   reopened this on Jul 24, 2015
7. mistercrunch

   closed this as completedon Jul 24, 2015
8. mobuchowski

   added a commit that references this issue on Jan 4, 2022

   initial commit; add bigquery extractor (apache#66)

   ...

   4c2a200