RabbitMQ startup fails due to missing wait/retry

[tharwan]

Describe the bug

The RabbitMQ container fails to start up properly.

To Reproduce

test_container = RabbitMqContainer(
        image="rabbitmq:3.9.10",
    )
test_container.start()  # the readiness_probe will fail

Runtime environment
uname -a
Darwin MABP-C02FW18CMD6V 22.4.0 Darwin Kernel Version 22.4.0: Mon Mar 6 21:00:17 PST 2023; root:xnu-8796.101.5~3/RELEASE_X86_64 x86_64

python --version
Python 3.8.11

docker info

 Server:
 Containers: 39
  Running: 0
  Paused: 0
  Stopped: 39
 Images: 112
 Server Version: 20.10.20
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
 runc version: 5fd4c4d144137e991c4acebb2146ab1483a97925
 init version: 
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.15.82-0-virt
 Operating System: Alpine Linux v3.16
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 3.845GiB
 Name: colima
 ID: 4PMF:ODB3:U555:IXBH:6ZQU:OZST:ASMT:HRMX:77AE:2ZW2:VYQS:VYPL
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

pika 1.3.1
testcontainers 3.7.1

I am happy to contribute a PR. Assuming that the current tests pass just fine in the pipeline. Are there any tests to simulate a slower container startup?

[tillahoffmann]

Thanks for reporting. Could you please provide all the information requested in the issue template?

[tharwan]

I can not share all python packages, but I think that's not necessary anyhow.

Our fix to make it work was

from pika.connection import Parameters
Parameters.DEFAULT_CONNECTION_ATTEMPTS = 10
test_container.start()

[tillahoffmann]

Two more questions: Could you please also share the traceback? Does this test work on your machine?

testcontainers-python/rabbitmq/testcontainers/rabbitmq/__init__.py

Lines 17 to 24 in 8a3a958

	.. doctest::
	>>> import pika
	>>> from testcontainers.rabbitmq import RabbitMqContainer
	>>> with RabbitMqContainer("rabbitmq:3.9.10") as rabbitmq:
	... connection = pika.BlockingConnection(rabbitmq.get_connection_params())
	... channel = connection.channel()

[jonathandannenberg]

I came across the same problem.
I am using DinD in a self hosted GitLab Instance. Gitrunner runs on a linux vm, docker version is 24.0.2.
Python venv for the job: pytest, testcontainers-rabbitmq

The Job output for running a basic publish/receive test:

$ pytest -s tests/test_debug_ci_cd/test_rabbitmq.py
============================= test session starts ==============================
platform linux -- Python 3.10.9, pytest-7.4.0, pluggy-1.2.0 -- /usr/local/bin/python
cachedir: .pytest_cache
rootdir: /builds/jda/debug_ci_cd
configfile: setup.cfg
collecting ... collected 1 item
tests/test_debug_ci_cd/test_rabbitmq.py::test_send Pulling image rabbitmq:latest
Container started: 6a67cf3f6ac0
Waiting to be ready...
Waiting to be ready...
Connection attempt '1' of '121' failed: Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/testcontainers/core/waiting_utils.py", line 49, in wrapper
    return wrapped(*args, **kwargs)
  File "/builds/jda/debug_ci_cd/tests/test_debug_ci_cd/test_rabbitmq.py", line 87, in readiness_probe
    connection = pika.BlockingConnection(self.get_connection_params())
  File "/usr/local/lib/python3.10/site-packages/pika/adapters/blocking_connection.py", line 360, in __init__
    self._impl = self._create_connection(parameters, _impl_class)
  File "/usr/local/lib/python3.10/site-packages/pika/adapters/blocking_connection.py", line 451, in _create_connection
    raise self._reap_last_connection_workflow_error(error)
pika.exceptions.AMQPConnectionError
Waiting to be ready...
...

Connection attempt '5' of '121' failed: Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/testcontainers/core/waiting_utils.py", line 49, in wrapper
    return wrapped(*args, **kwargs)
  File "/builds/jda/debug_ci_cd/tests/test_debug_ci_cd/test_rabbitmq.py", line 87, in readiness_probe
    connection = pika.BlockingConnection(self.get_connection_params())
  File "/usr/local/lib/python3.10/site-packages/pika/adapters/blocking_connection.py", line 360, in __init__
    self._impl = self._create_connection(parameters, _impl_class)
  File "/usr/local/lib/python3.10/site-packages/pika/adapters/blocking_connection.py", line 451, in _create_connection
    raise self._reap_last_connection_workflow_error(error)
pika.exceptions.AMQPConnectionError
Waiting to be ready...
Waiting to be ready...
PASSED
============================== 1 passed in 12.34s ==============================
Cleaning up project directory and file based variables
00:01
Job succeeded

Another fix is to add pika.exceptions.AMQPConnectionError to the @wait_container_is_ready decorator of the readiness_probe method.
Connection is then established after ~ 6 attempts.
Also I changed loglevel to debug for testcontainers.core.waiting_utils. (So that the failed connection attempts are shown in job output)