RabbitMQ startup fails due to missing wait/retry

Describe the bug

The RabbitMQ container fails to start up properly.

To Reproduce

test_container = RabbitMqContainer(
        image="rabbitmq:3.9.10",
    )
test_container.start()  # the readiness_probe will fail

Runtime environment
uname -a
Darwin MABP-C02FW18CMD6V 22.4.0 Darwin Kernel Version 22.4.0: Mon Mar 6 21:00:17 PST 2023; root:xnu-8796.101.5~3/RELEASE_X86_64 x86_64

python --version
Python 3.8.11

docker info

 Server:
 Containers: 39
  Running: 0
  Paused: 0
  Stopped: 39
 Images: 112
 Server Version: 20.10.20
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
 runc version: 5fd4c4d144137e991c4acebb2146ab1483a97925
 init version: 
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.15.82-0-virt
 Operating System: Alpine Linux v3.16
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 3.845GiB
 Name: colima
 ID: 4PMF:ODB3:U555:IXBH:6ZQU:OZST:ASMT:HRMX:77AE:2ZW2:VYQS:VYPL
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

pika 1.3.1
testcontainers 3.7.1

I am happy to contribute a PR. Assuming that the current tests pass just fine in the pipeline. Are there any tests to simulate a slower container startup?

Thanks for reporting. Could you please provide all the information requested in the issue template?

I can not share all python packages, but I think that's not necessary anyhow.

Our fix to make it work was

from pika.connection import Parameters
Parameters.DEFAULT_CONNECTION_ATTEMPTS = 10
test_container.start()

Two more questions: Could you please also share the traceback? Does this test work on your machine?

testcontainers-python/rabbitmq/testcontainers/rabbitmq/__init__.py

Lines 17 to 24 in 8a3a958

    
                   .. doctest:: 
        
                       >>> import pika 
        
                       >>> from testcontainers.rabbitmq import RabbitMqContainer 
        
                       >>> with RabbitMqContainer("rabbitmq:3.9.10") as rabbitmq: 
        
                       ...    connection = pika.BlockingConnection(rabbitmq.get_connection_params()) 
        
                       ...    channel = connection.channel()

I came across the same problem.
I am using DinD in a self hosted GitLab Instance. Gitrunner runs on a linux vm, docker version is 24.0.2.
Python venv for the job: pytest, testcontainers-rabbitmq

The Job output for running a basic publish/receive test:

$ pytest -s tests/test_debug_ci_cd/test_rabbitmq.py
============================= test session starts ==============================
platform linux -- Python 3.10.9, pytest-7.4.0, pluggy-1.2.0 -- /usr/local/bin/python
cachedir: .pytest_cache
rootdir: /builds/jda/debug_ci_cd
configfile: setup.cfg
collecting ... collected 1 item
tests/test_debug_ci_cd/test_rabbitmq.py::test_send Pulling image rabbitmq:latest
Container started: 6a67cf3f6ac0
Waiting to be ready...
Waiting to be ready...
Connection attempt '1' of '121' failed: Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/testcontainers/core/waiting_utils.py", line 49, in wrapper
    return wrapped(*args, **kwargs)
  File "/builds/jda/debug_ci_cd/tests/test_debug_ci_cd/test_rabbitmq.py", line 87, in readiness_probe
    connection = pika.BlockingConnection(self.get_connection_params())
  File "/usr/local/lib/python3.10/site-packages/pika/adapters/blocking_connection.py", line 360, in __init__
    self._impl = self._create_connection(parameters, _impl_class)
  File "/usr/local/lib/python3.10/site-packages/pika/adapters/blocking_connection.py", line 451, in _create_connection
    raise self._reap_last_connection_workflow_error(error)
pika.exceptions.AMQPConnectionError
Waiting to be ready...
...

Connection attempt '5' of '121' failed: Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/testcontainers/core/waiting_utils.py", line 49, in wrapper
    return wrapped(*args, **kwargs)
  File "/builds/jda/debug_ci_cd/tests/test_debug_ci_cd/test_rabbitmq.py", line 87, in readiness_probe
    connection = pika.BlockingConnection(self.get_connection_params())
  File "/usr/local/lib/python3.10/site-packages/pika/adapters/blocking_connection.py", line 360, in __init__
    self._impl = self._create_connection(parameters, _impl_class)
  File "/usr/local/lib/python3.10/site-packages/pika/adapters/blocking_connection.py", line 451, in _create_connection
    raise self._reap_last_connection_workflow_error(error)
pika.exceptions.AMQPConnectionError
Waiting to be ready...
Waiting to be ready...
PASSED
============================== 1 passed in 12.34s ==============================
Cleaning up project directory and file based variables
00:01
Job succeeded

Another fix is to add pika.exceptions.AMQPConnectionError to the @wait_container_is_ready decorator of the readiness_probe method.
Connection is then established after ~ 6 attempts.
Also I changed loglevel to debug for testcontainers.core.waiting_utils. (So that the failed connection attempts are shown in job output)

fix: rabbitmq readiness probe (#375)

Navigation Menu

Repository navigation

RabbitMQ startup fails due to missing wait/retry #348

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	.. doctest::

	>>> import pika
	>>> from testcontainers.rabbitmq import RabbitMqContainer

	>>> with RabbitMqContainer("rabbitmq:3.9.10") as rabbitmq:
	... connection = pika.BlockingConnection(rabbitmq.get_connection_params())
	... channel = connection.channel()

RabbitMQ startup fails due to missing wait/retry #348

Description

Activity

tillahoffmann commented on May 8, 2023

tharwan commented on May 9, 2023

tillahoffmann commented on May 9, 2023

jonathandannenberg commented on Jul 20, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions