RabbitMQ fails at startup - Erlang distribution error - Mac

Questions / Help
1

When I start RabbitMQ by typing rabbitmq-server it fails with the error message below.

When I use brew services start rabbitmq it responds with “succesfully started rabbitmq” but when I type rabbitmqctl help, after a long pause (2 min or so), it returns about the same error (less text).

Part of the response to rabbitmq-server:

2022-08-12 20:10:04.756965+02:00 [notice] <0.145.0> Protocol 'inet_tcp': register/listen error: etimedout
2022-08-12 20:10:04.756965+02:00 [notice] <0.145.0> 
2022-08-12 20:10:04.764490+02:00 [error] <0.142.0>     supervisor: {local,net_sup}
2022-08-12 20:10:04.764490+02:00 [error] <0.142.0>     errorContext: start_error
2022-08-12 20:10:04.764490+02:00 [error] <0.142.0>     reason: {'EXIT',nodistribution}
2022-08-12 20:10:04.764490+02:00 [error] <0.142.0>     offender: [{pid,undefined},
2022-08-12 20:10:04.764490+02:00 [error] <0.142.0>                {id,net_kernel},
2022-08-12 20:10:04.764490+02:00 [error] <0.142.0>                {mfargs,{net_kernel,start_link,
2022-08-12 20:10:04.764490+02:00 [error] <0.142.0>                                    [#{clean_halt => false,
2022-08-12 20:10:04.764490+02:00 [error] <0.142.0>                                       name => rabbit_prelaunch_3696@localhost,
2022-08-12 20:10:04.764490+02:00 [error] <0.142.0>                                       name_domain => shortnames,
2022-08-12 20:10:04.764490+02:00 [error] <0.142.0>                                       net_tickintensity => 4,
2022-08-12 20:10:04.764490+02:00 [error] <0.142.0>                                       net_ticktime => 60,
2022-08-12 20:10:04.764490+02:00 [error] <0.142.0>                                       supervisor => net_sup_dynamic}]}},
2022-08-12 20:10:04.764490+02:00 [error] <0.142.0>                {restart_type,permanent},
2022-08-12 20:10:04.764490+02:00 [error] <0.142.0>                {significant,false},
2022-08-12 20:10:04.764490+02:00 [error] <0.142.0>                {shutdown,2000},
2022-08-12 20:10:04.764490+02:00 [error] <0.142.0>                {child_type,worker}]
2022-08-12 20:10:04.764490+02:00 [error] <0.142.0> 

2022-08-12 20:10:04.773521+02:00 [error] <0.130.0> 
2022-08-12 20:10:04.773521+02:00 [error] <0.130.0> BOOT FAILED
2022-08-12 20:10:04.773521+02:00 [error] <0.130.0> ===========
2022-08-12 20:10:04.773521+02:00 [error] <0.130.0> Exception during startup:
2022-08-12 20:10:04.773521+02:00 [error] <0.130.0> 
2022-08-12 20:10:04.773521+02:00 [error] <0.130.0> error:{badmatch,{error,{{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}},{child,undefined,net_sup_dynamic,{erl_distribution,start_link,[#{clean_halt => false,name => rabbit_prelaunch_3696@localhost,name_domain => shortnames,net_tickintensity => 4,net_ticktime => 60,supervisor => net_sup_dynamic}]},permanent,false,1000,supervisor,[erl_distribution]}}}}
2022-08-12 20:10:04.773521+02:00 [error] <0.130.0> 
2022-08-12 20:10:04.773521+02:00 [error] <0.130.0>     rabbit_prelaunch_dist:duplicate_node_check/1, line 80
2022-08-12 20:10:04.773521+02:00 [error] <0.130.0>     rabbit_prelaunch_dist:setup/1, line 23
2022-08-12 20:10:04.773521+02:00 [error] <0.130.0>     rabbit_prelaunch:do_run/0, line 115
2022-08-12 20:10:04.773521+02:00 [error] <0.130.0>     rabbit_prelaunch:run_prelaunch_first_phase/0, line 32
2022-08-12 20:10:04.773521+02:00 [error] <0.130.0>     supervisor:do_start_child_i/3, line 414
2022-08-12 20:10:04.773521+02:00 [error] <0.130.0>     supervisor:do_start_child/2, line 400
2022-08-12 20:10:04.773521+02:00 [error] <0.130.0>     supervisor:-start_children/2-fun-0-/3, line 384
2022-08-12 20:10:04.773521+02:00 [error] <0.130.0>     supervisor:children_map/4, line 1250
2022-08-12 20:10:04.773521+02:00 [error] <0.130.0> 
BOOT FAILED

it ends with: Crash dump is being written to: erl_crash.dump...done

but I have not been able to find the file so far. A terminal find command on the whole system didn’t help.

Response I get from rabbitmqctl help (after initiating it with homebrew):

20:04:53.162 [error] rabbit_env: Failed to setup distribution (as rabbit_ctl_46@localhost) to query node rabbit@localhost: {:error,
 {{:shutdown, {:failed_to_start_child, :net_kernel, {:EXIT, :nodistribution}}},
  {:child, :undefined, :net_sup_dynamic,
   {:erl_distribution, :start_link,
    [
      %{
        clean_halt: false,
        name: :rabbit_ctl_46@localhost,
        name_domain: :shortnames,
        net_tickintensity: 4,
        net_ticktime: 60,
        supervisor: :net_sup_dynamic
      }
    ]}, :permanent, false, 1000, :supervisor, [:erl_distribution]}}}

about the setup: MacOS, Homebrew installation out of the box. RabbitMQ 3.10.7 Erlang 25.0.3

I installed RabbitMQ recently for the first time, only used for some testing, no (manual) configuration. Now, a week later it fails to start. I have reinstalled it, using homebrew, hoping that would help, but no luck.

Any idea’s?

Frankly, I don’t know anything about erlang. Based on the error and the posts I have read I am guessing it has something to do with the Erlang distribution.

I also posted this on stackoverflow:

1 Like
2

Is it possible another program is using one of these ports?

https://www.rabbitmq.com/networking.html#ports

In particular 25672 or 4369. You can check with lsof.

1 Like
3

I’d try do diagnose epmd startup problem.
E.g. see what epmd -names prints, and try epmd -daemon followed by RabbitMQ startup.

1 Like
4

Based on @domi’s suggestion I used sudo lsof -i -n -P | grep TCP to see about the used ports. What surprised me was the amount of ports being used. 4 services where most prominent, cloud storage, password manager, VPN, web-browser (Mega, 1password, Proton, Firefox respectively).
At first glance none of the used ports matched the Rabbitmq networking list. Nevertheless I decided to close Mega and 1password (both desktop app and Firefox add-on).

And surprise surprise, RabbitMQ works!

Both methods of rabbitmq-server and brew services start rabbitmq work.

Interestingly, afterwards, I shut down Rabbitmq, started Mega and 1password again, and reinitiated Rabbitmq, which started without any errors. Neither a reboot got me back into trouble. Meaning I am unable to replicate the issue I had before.
Though I wanted to follow-up on @max-au’s suggestion as well and see where it let me, for now I am unable to do so. Anyway, I am out of the woods.

Thanks a lot for your help!

1 Like
5

Update - I got back into trouble again.

So, like a trained monkey, I closed Mega and 1password, restarted rabbitmq-server, expecting to have resolved the issue. Nothing could be less true.

After more digging I found the real source of the issue. ProtonVPN gives the option to either block or allow LAN connections while maintaining the VPN connection to the outside world. I usually have this blocked. However, earlier this week I changed it (in order to establish an ssh connection with a Raspberry pi).
I’m confident this is the real issue, because unlike last time, I am able to get myself into trouble, and back out of trouble again.

What still puzzles me is why this makes any difference to rabbitmq.

1 Like
6

It could be that RabbitMQ (around here) tries to connect to epmd via non-loopback IP address, timing out with LAN blocked.

1 Like