Celery with RabbitMQ broker ping hanging

Tue Jun 22 2021

We had an issue this morning where trying to ping (opens new window) our Celery workers just hanged. Even setting a timeout didn't stop it from hanging:

app.control.ping(timeout=0.5)

However, I was able to connect to our RabbitMQ instance like this (opens new window) so the broker URL was correct:

import socket
from kombu import Connection

celery_broker_url = "amqp://<broker-url>"

try:
    conn = Connection(celery_broker_url)
    conn.ensure_connection(max_retries=3)
except socket.error:
    raise RuntimeError("Failed to connect to RabbitMQ instance at {}".format(celery_broker_url))

I got access to our RabbitMQ Management UI (opens new window) and saw that all the connections were "blocked":

After some Googling, I found out that RabbitMQ nodes blocks (opens new window) connections that will publish messages when either the free disk space or memory use goes above their configured watermarks.

I went back to the Management UI and saw the all of our nodes have reached their disk space watermarks. After clearing some space on the nodes, we were able to ping the workers again.

We also added Datadog monitors (opens new window) to notify us when a node has reached its memory or disk usage watermark in the future.