Celery + RabbitMQ stuck at mingle: searching for neighbors

Question

I encountered strange issue while trying to get my system running on new machines.

OS:

Ubuntu 14.04.1 LTS (GNU/Linux 3.13.0-39-generic x86_64)

packages:

celery==3.1.13

django-celery==3.0.23

librabbitmq==1.5.2

broker:

RabbitMQ "3.2.4"

After restart Celery process gets stuck:

[2014-11-10 18:32:55,792: INFO/MainProcess] Connected to amqp://user:**@172.16.10.6:5672/vhost
[2014-11-10 18:32:55,804: INFO/MainProcess] mingle: searching for neighbors

Tried to find the solution elsewhere. Here I found advice to increase disk space available for RabbitMQ. I checked docs and changed in rabbitmq.config file:

{disk_free_limit, {mem_relative, 1.0}}

Now RabbitMQ should have 6GB available, but it doesn't push Celery into ready state. I also tried configuring (as someone suggested) limit of open file descriptors in

/etc/default/rabbitmq-server

without any effect. Current RabbitMQ status:

    [{pid,4131},
 {running_applications,
     [{rabbitmq_management,"RabbitMQ Management Console","3.2.4"},
      {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.2.4"},
      {webmachine,"webmachine","1.10.3-rmq3.2.4-gite9359c7"},
      {mochiweb,"MochiMedia Web Server","2.7.0-rmq3.2.4-git680dba8"},
      {rabbitmq_management_agent,"RabbitMQ Management Agent","3.2.4"},
      {rabbit,"RabbitMQ","3.2.4"},
      {os_mon,"CPO  CXC 138 46","2.2.14"},
      {inets,"INETS  CXC 138 49","5.9.7"},
      {mnesia,"MNESIA  CXC 138 12","4.11"},
      {amqp_client,"RabbitMQ AMQP Client","3.2.4"},
      {xmerl,"XML parser","1.3.5"},
      {sasl,"SASL  CXC 138 11","2.3.4"},
      {stdlib,"ERTS  CXC 138 10","1.19.4"},
      {kernel,"ERTS  CXC 138 10","2.16.4"}]},
 {os,{unix,linux}},
 {erlang_version,
     "Erlang R16B03 (erts-5.10.4) [source] [64-bit] [smp:4:4] [async-threads:30] [kernel-poll:true]\n"},
 {memory,
     [{total,46342000},
      {connection_procs,290304},
      {queue_procs,62864},
      {plugins,458136},
      {other_proc,13673776},
      {mnesia,76376},
      {mgmt_db,127808},
      {msg_index,34384},
      {other_ets,1100432},
      {binary,5282744},
      {code,19974306},
      {atom,703377},
      {other_system,4557493}]},
 {vm_memory_high_watermark,0.4},
 {vm_memory_limit,2503614464},
 {disk_free_limit,6259036160},
 {disk_free,2779140096},
 {file_descriptors,
     [{total_limit,924},{total_used,13},{sockets_limit,829},{sockets_used,9}]},
 {processes,[{limit,1048576},{used,270}]},
 {run_queue,0},
 {uptime,4561}]

Another thing: when I stop Celery processes with Supervisor no warm shutdown is logged (actually - nothing is logged).

Could you help?

Answer 1

I had the same problem it seems. Check my own answer to my own question here:

celeryd with RabbitMQ hangs on "mingle: searching for neighbors", but plain celery works

Long story short:

sudo apt-get remove librabbitmq1

worked for me.