Docker Swarm Peer Discovery Backend

[Radiergummi]

I'm operating a RabbitMQ cluster in a Docker Swarm cluster. This generally works well, but I fought bitterly to get automatic node discovery working -- in the end, I had to settle for a suboptimal workaround.
The general problem is that while Docker Swarm provides a DNS service discovery mechanism, the reverse records do not match the hostnames of the containers RabbitMQ is running in - and there is no way to achieve that. The seed host tasks.<service name> returns a list of A records as required by the DNS backend:

root@rabbitmq-foo-1:/# nslookup tasks.rabbitmq
Server:     127.0.0.11
Address:    127.0.0.11#53

Non-authoritative answer:
Name:   tasks.rabbitmq
Address: 10.100.8.158
Name:   tasks.rabbitmq
Address: 10.100.8.162
Name:   tasks.rabbitmq
Address: 10.100.8.163

The IPs resolve to the following, however:

root@rabbitmq-foo-1:/# nslookup 10.100.8.158
158.8.100.10.in-addr.arpa   name = foo_rabbitmq.tjqxe6kpe5g4mnmgwoiuink16.xhosndlwm3eaoldu36udvjfi9.rabbitmq-net.

for a container with a hostname configured to rabbitmq-foo-2.

This means that currently, the only options are to run a separate service mesh, or configure the hosts statically using the classic file backend - which works, but isn't ideal (I have to modify and re-deploy the RMQ service anytime we add or remove a Swarm node).

I would like to formally suggest adding a service discovery backend plugin for Docker Swarm; in theory, this isn't too complicated, but would make it easier to get RabbitMQ running dynamically in Swarm.
I'm out of my depth in Erlang however, so I cannot do this on my own.

---

In the mean time, if anyone is interested in my solution: We run the service in global mode, so one RabbitMQ instance is deployed to every Swarm node. This makes it possible to use a volume persistent to the host. Additionally, we set the task hostname to that of the host node, so we can configure it statically.
The compose file looks like so:

services:
  rabbitmq:
    image: "rabbitmq:3-management"

    # This ensures the node has a persistent, unique name we refer to in RABBITMQ_NODENAME.
    hostname: "rabbitmq-{{.Node.Hostname}}"  
    networks:
      - rabbitmq_net
    volumes:

      # As the task will be deployed to the same node, this works on service updates too!
      - rabbitmq-data:/var/lib/rabbitmq
    environment: 
      RABBITMQ_NODENAME: "rabbit@rabbitmq-{{.Node.Hostname}}"

    # The uid/gid/mode arguments below prevent permission errors at runtime
    configs:
      - source: rabbitmq_config
        target: /etc/rabbitmq/rabbitmq.conf
        uid: "999"
        gid: "999"
        mode: 0600
    secrets:
      - source: rabbitmq_erlang_cookie
        target: /var/lib/rabbitmq/.erlang.cookie
        uid: "999"
        gid: "999"
        mode: 0600
      - source: rabbitmq_definitions
        target: /etc/rabbitmq/definitions.json
        uid: "999"
        gid: "999"
        mode: 0600

    # See https://www.rabbitmq.com/configure.html#kernel-limits
    ulimits:
      nofile:
        soft: 64000
        hard: 64000
    deploy:

      # Setting the mode to global ensures a single replica will be started on every node
      mode: global

      # Ensure stopped instances will be restarted. RabbitMQ seems to error with an exit
      # code 0 sometimes for some reason, so `any` catches that too.
      restart_policy:
        condition: any

      # Ensure we start one node after the other, strictly sequential, and make sure it
      # is running before starting the next. This makes sure a cluster is formed instead
      # of every node bootstrapping its own.
      update_config:
        parallelism: 1
        monitor: 10s
        order: stop-first

    # This health check is used during service (re)deployments. While the timings could
    # be improved, this works well.
    healthcheck:
      test: rabbitmq-diagnostics -q ping
      interval: 10s
      timeout: 3s
      retries: 30

networks:
  rabbitmq_net:

volumes:
  rabbitmq-data:

secrets:
  rabbitmq_definitions:
    file: ./rabbitmq_definitions.json

  rabbitmq_erlang_cookie:
    file: ./rabbitmq_erlang_cookie.txt

configs:
  rabbitmq_config:
    file: ./rabbitmq.conf

The last step requires some kind of deployment script or CI pipeline. While we use Bitbucket pipelines, here's the meaty bits from our script:

# Fetch all swarm hosts from the Docker daemon. As we used the hostname for our containers,
# we can use it to infer the RabbitMQ cluster node name from it.
docker_hosts=$(docker node ls --format '{{.Hostname}}')
count=0

for host in ${docker_hosts}; do
  count=$((count+1))
  echo "cluster_formation.classic_config.nodes.${count} = rabbit@rabbitmq-${host}" >> rabbitmq.conf
done

I hope this helps someone - I spent considerable time researching.

[lukebakken]

I would like to formally suggest adding a service discovery backend plugin for Docker Swarm; in theory, this isn't too complicated, but would make it easier to get RabbitMQ running dynamically in Swarm.

You could still assist us in doing this by providing explicit API calls, code samples, doc links etc that demonstrate how to register and remove nodes from whatever registry Docker Swarm provides. Right now you're asking us to do all the work but I'm certain you could contribute a lot without writing any Erlang. This would be very helpful because I don't think Team RabbitMQ members have much experience with Swarm. Thanks.

[michaelklishin]

Consul, etcd, DNS peer discovery can be used with Docker Swarm. If there is an API that would allow for swam node discovery using a certain condition, it would be able to build a rough alternative to the Kubernetes peer discovery.

Docker Swarm has extensive documentation but I could not find anything that would cover the API bits we need.

[Radiergummi]

You could still assist us in doing this by providing explicit API calls, code samples, doc links etc that demonstrate how to register and remove nodes from whatever registry Docker Swarm provides. Right now you're asking us to do all the work but I'm certain you could contribute a lot without writing any Erlang. This would be very helpful because I don't think Team RabbitMQ members have much experience with Swarm. Thanks.

@lukebakken I'm sorry if the initial post did appear like that. I just wanted to initiate some discussion and see if the team is interested at all! If you are, I'm naturally doing everything I can to assist.

[michaelklishin]

We'd like to understand what kind of APIs are available as part of Docker Swarm, to understand if we can do roughly what the Kubernetes peer discovery implementation does. If not, then assess our other options.

[michaelklishin]

I doubt there would be any opposition to having a Docker Swarm peer discovery plugin, as long as it would end up being a reasonable amount of effort not just now but also in the future. Sadly I know little about Docker Swarm.

[Radiergummi]

This is a short introduction from 10,000 feet on how Docker Swarm works, how it enables service discovery, and which options are available for peer discovery. I tried to be as concise as possible, but can go into detail on any of these sections if desired.

How Docker Swarm works

Swarm forms a cluster from multiple Docker engines, organised as manager and worker nodes. Managers are elected via raft consensus and distribute workloads among all cluster nodes, including managers.

Swarm clusters run collections of Docker containers referred to as stacks, which can conveniently be defined using docker compose – any valid compose file can be deployed to a Docker swarm, with additional deployment settings being available. Among them is the ability to create an arbitrary number of replicas tasks of a service, that is, instances of a container.

Swarm follows a declarative approach to define desired cluster state and will attempt to reconcile any differences. Additionally, it forms a routing mesh to distribute socket connections among all members of a service; effectively this means the ingress controller is built-in.

The documentation gives a good introduction: https://docs.docker.com/engine/swarm/key-concepts/

Service Discovery in Swarm

As previously mentioned, deployments in Swarm are organised like this:

The stack groups all services an application consists of, somewhat similar to a Kubernetes deployment. A service maps closer to k8s: It represents a logical application with an arbitrary amount of replicas, defines reliance on external secrets, volumes, networks, etc. A single task (= replica) of a service is an actual Docker container running on one of the Swarm nodes. It doesn't matter which one, because Swarm forms a virtual network for each stack and transparently handles routing packets between the container instances. You can also assign additional networks to build a custom topology.

To make it easier to work with dynamically assigned IP addresses, Swarm provides a simple DNS server on each container (127.11). It resolves service names to a single, virtual IP address that will be routed to the next best task; thus, one can use e.g. amqp://my-rabbitmq-service in applications, and Swarm handles the IP routing.
For distributed software like RabbitMQ, a list of replica instances can be retrieved by querying a magical DNS record in the shape of tasks.<service name>, e.g. tasks.my-rabbitmq-service. This will return an A/AAAA record with all task IPs. When querying the reverse record of a task IP, the DNS server will return a record scoped to the container the query is issued at, shaped like so:

<Project Name>_<Task Name>.<Task ID>.< Container ID>.<Network Name>.

This name can be queried again to return the IP of the task instance.

Why the current RabbitMQ DNS backend doesn't work for Swarm

As outlined above, the service discovery DNS service provided by Swarm is insufficient for usage with the DNS peer discovery backend, because the reverse records reported by Swarm do not correspond to the hostnames reported to RabbitMQ instances themselves, that is: A replica task has an auto-generated (or manually configured) hostname, and Swarm DNS has A records for that name, but it will not report those hostnames in reverse records, nor provide a list of those hostnames.
This means that you can change the view of a container of itself, but not the view of the rest of the world; and it breaks the RabbitMQ assumption that a reverse record returns the actual hostname of a host, ruling out the DNS backend.
I also described the problem in further details on StackOverflow.

Options for RabbitMQ peer discovery

That said, I do see two alternative approaches which could work:

1 - Not relying on RR records

This Swarm backend would work similar to the existing DNS backend, but instead of relying on the hostnames obtained from reverse records, it would attempt to connect to and query the identified instances for their node name.

Open questions:

• Does this have security implications I'm not thinking of?
• Does epmd currently have the ability to report the node name? I know next to nothing about it, sorry.

This approach has the advantage of not requiring privileged access to the Docker daemon - more on that issue in the next section. I suppose a lot of code from the DNS backend could be reused if not shared, so this would probably be the least invasive option.
On the downside, a lot of the advanced functionality of Swarm could not be used; this would really "just" provide service discovery, but nothing similar to the K8s operators.

2 - Asking the Docker daemon

In this scenario, cluster operators would need to mount the host's Docker socket into the service, or expose it to the host network. Both are possible, mounting the socket is favoured by the industry. By doing so, the new RabbitMQ peer discovery backend could connect to the Docker HTTP API, which opens up interesting capabilities - such as retrieving and manipulating the full service configuration.
This has two main drawbacks.
One, this will only work on manager nodes, and thus confine RabbitMQ to those cluster nodes. Typically, a cluster has a limited number of managers, and the docs even recommend not running services on managers for critical deployments.
Two, this is all-or-nothing in terms of access control: With access to the Docker socket on a manager node, the container can issue any and all Docker management commands, such as dropping the cluster and erasing all storage. This is probably something users don't want :)

I still think this approach deserves some attention, as other major vendors use it to communicate with the Docker daemon, too; Additionally, this would allow the backend to actually perform similar work to the Kubernetes operator, even if that probably requires significant engineering work.

---

@michaelklishin @lukebakken I hope this clears some things up - let me know if there are things missing or too vague. I'm going to devote some brain cycles to the Docker daemon approach and read the RabbitMQ Kubernetes operator thoroughly in the mean time!

[Radiergummi]

While that's definitely great work, your setup evokes a tad too many feelings of "infrastructure Jenga tower" in me to be comfortable with this… I'd still hope for a proper Swarm backend to land at some point :)

[michaelklishin]

@Radiergummi hope is not how open source software is developed.

Docker Swarm users can use Consul or etcd backends today, likely with relatively little effort. If there's anything specific to Swarm truly worth simplifying and (likely falsely) assuming that RabbitMQ somehow can address that, all it takes is developing a very small plugin (a single module with a three function interface or so).

[Radiergummi]

@Radiergummi hope is not how open source software is developed.

Docker Swarm users can use Consul or etcd backends today, likely with relatively little effort. If there's anything specific to Swarm truly worth simplifying and (likely falsely) assuming that RabbitMQ somehow address that, all it takes is developing a very small plugin (a single module with a three function interface or so).

I'm afraid you might have taken my comment as snark, which it most definitely wasn't. The complexity of this specific setup is rooted in the peculiarities of Docker Swarm, not RabbitMQ, which I am very fond of.
I have contributed my share of open source software and will continue doing so, but I am not an Erlang developer as I pointed out in the original proposal, and thus cannot contribute much in terms of code.

I tried my best to outline possible solutions on a conceptual level, however, and do not expect or demand someone to take action on that; rather, I hope someone qualified with time on their hands and the same itch to scratch comes by and picks this issue up.

[dtila]

I have spent 2 days trying to make work RabbitMQ auto-discovery in Swarm to realize it's not implemented - nor documented.
I will do my best to explain what's the issue and why Swarm doesn't work and how can be done

First of all, @Radiergummi explained very well how Swarm works, but I will give more details about what is happening under Swarm governance.

I have used Docker under Windows with a single node to reproduce the following behavior with a single Swarm node.

> docker swarm init

This will create a single node but enough for debug and replicate the Swarm behavior.

To understand the problem, I will elaborate a scenario where a service name rabbitmq is replicated twice on the same node (or different nodes).

> docker ps
CONTAINER ID   IMAGE                     NAMES
81b4ddda42e7   rabbitmq:4.0-management   rabbitmq_rabbitmq.2.zdri7fa0n8gkvfp45e3192hs7
32fc128f7f23   rabbitmq:4.0-management   rabbitmq_rabbitmq.1.yc7rpeyavoiun5vmxttyxlr6c

To be even clear, this are the DNS registered for each container. The rabbitmq-{number} is an additional host added by myself in the docker-compose.yml to differentiate nodes

> docker inspect 81b4ddda42e7
"DNSNames": [
  "rabbitmq_rabbitmq.2.zdri7fa0n8gkvfp45e3192hs7",
  "81b4ddda42e7",
  "rabbitmq-2"
]

> docker inspect 81b4ddda42e7
"DNSNames": [
	"rabbitmq_rabbitmq.1.yc7rpeyavoiun5vmxttyxlr6c",
    "32fc128f7f23",
    "rabbitmq-1"
]

With this, let's see how Swarm resolves the nodes:

> docker exec -it 81b4ddda42e7 getent hosts rabbitmq
10.10.11.2      rabbitmq

This result is obtained by any container which is within the network - will get a single IP - which will be the same IP. The returned result by the Swarm is the auto balanacer which will make redirect to one of the nodes.

However, a service can also request Docker DNS to get all the nodes that are part of the rabbitmq service, By convention this can be done by tasks. format

> docker exec -it 32fc128f7f23 getent hosts tasks.rabbitmq
10.10.11.4      tasks.rabbitmq
10.10.11.3     tasks.rabbitmq

As you can see the actual nodes, are different by the one using the service name. The IP's are the actual ip's on the nodes that are part of the RabbitMQ cluster.
Also, the same results will be obtained if command is executed from any container within the network.

However when we do reverse DNS (which is what RabbitMQ does), here are the results:

(node 1)

>  docker exec -it 32fc128f7f23 getent hosts 10.10.11.4 10.10.11.3
10.10.11.4      rabbitmq-1
10.10.11.3      rabbitmq_rabbitmq.2.zdri7fa0n8gkvfp45e3192hs7.rabbitmq

(node 2)

>  docker exec -it 81b4ddda42e7 getent hosts 10.10.11.4 10.10.11.3
10.10.11.4      rabbitmq_rabbitmq.1.yc7rpeyavoiun5vmxttyxlr6c.rabbitmq
10.10.11.3      rabbitmq-2

Here is the problem since the name will be different and Rabbit considers there are not in the same cluster.

The actual error that is found in the logs is:

Peer discovery: configured backend: rabbit_peer_discovery_dns
rabbitmq_rabbitmq.1.22g75kpd9eum@docker-desktop    | 2024-12-23 12:49:56.927810+00:00 [notice] <0.271.0> Feature flags: attempt to enable `rabbit_exchange_type_local_random`...
rabbitmq_rabbitmq.1.22g75kpd9eum@docker-desktop    | 2024-12-23 12:49:56.933223+00:00 [notice] <0.271.0> Feature flags: `rabbit_exchange_type_local_random` enabled
rabbitmq_rabbitmq.1.22g75kpd9eum@docker-desktop    | 2024-12-23 12:49:56.933322+00:00 [notice] <0.271.0> Feature flags: attempt to enable `detailed_queues_endpoint`...
rabbitmq_rabbitmq.1.22g75kpd9eum@docker-desktop    | 2024-12-23 12:49:56.941889+00:00 [notice] <0.271.0> Feature flags: `detailed_queues_endpoint` enabled
rabbitmq_rabbitmq.1.22g75kpd9eum@docker-desktop    | 2024-12-23 12:49:56.941977+00:00 [notice] <0.271.0> Feature flags: attempt to enable `rabbitmq_4.0.0`...
rabbitmq_rabbitmq.1.22g75kpd9eum@docker-desktop    | 2024-12-23 12:49:56.949595+00:00 [notice] <0.271.0> Feature flags: `rabbitmq_4.0.0` enabled
rabbitmq_rabbitmq.1.22g75kpd9eum@docker-desktop    | 2024-12-23 12:49:56.949709+00:00 [notice] <0.271.0> Feature flags: attempt to enable `message_containers_deaths_v2`...
rabbitmq_rabbitmq.1.22g75kpd9eum@docker-desktop    | 2024-12-23 12:49:56.960429+00:00 [notice] <0.271.0> Feature flags: `message_containers_deaths_v2` enabled
rabbitmq_rabbitmq.1.22g75kpd9eum@docker-desktop    | 2024-12-23 12:49:56.960521+00:00 [notice] <0.271.0> Feature flags: attempt to enable `quorum_queue_non_voters`...
rabbitmq_rabbitmq.1.22g75kpd9eum@docker-desktop    | 2024-12-23 12:49:56.972041+00:00 [notice] <0.271.0> Feature flags: `quorum_queue_non_voters` enabled
rabbitmq_rabbitmq.1.22g75kpd9eum@docker-desktop    | 2024-12-23 12:49:56.972280+00:00 [info] <0.216.0> DB: virgin node -> run peer discovery
rabbitmq_rabbitmq.1.22g75kpd9eum@docker-desktop    | 2024-12-23 12:49:56.986901+00:00 [info] <0.216.0> Addresses discovered via A records of tasks.rabbitmq: 10.10.11.3, 10.10.11.4
rabbitmq_rabbitmq.1.22g75kpd9eum@docker-desktop    | 2024-12-23 12:49:56.987970+00:00 [info] <0.216.0> Addresses discovered via AAAA records of tasks.rabbitmq:
rabbitmq_rabbitmq.1.22g75kpd9eum@docker-desktop    | =PROGRESS REPORT==== 23-Dec-2024::12:49:57.228376 ===
rabbitmq_rabbitmq.1.22g75kpd9eum@docker-desktop    |     supervisor: {local,inet_gethost_native_sup}
rabbitmq_rabbitmq.1.22g75kpd9eum@docker-desktop    |     started: [{pid,<0.95.0>},{mfa,{inet_gethost_native,init,[[]]}}]
rabbitmq_rabbitmq.1.22g75kpd9eum@docker-desktop    |
rabbitmq_rabbitmq.1.22g75kpd9eum@docker-desktop    | =PROGRESS REPORT==== 23-Dec-2024::12:49:57.233692 ===
rabbitmq_rabbitmq.1.22g75kpd9eum@docker-desktop    |     supervisor: {local,kernel_safe_sup}
rabbitmq_rabbitmq.1.22g75kpd9eum@docker-desktop    |     started: [{pid,<0.94.0>},
rabbitmq_rabbitmq.1.22g75kpd9eum@docker-desktop    |               {id,inet_gethost_native_sup},
rabbitmq_rabbitmq.1.22g75kpd9eum@docker-desktop    |               {mfargs,{inet_gethost_native,start_link,[]}},
rabbitmq_rabbitmq.1.22g75kpd9eum@docker-desktop    |               {restart_type,temporary},
rabbitmq_rabbitmq.1.22g75kpd9eum@docker-desktop    |               {significant,false},
rabbitmq_rabbitmq.1.22g75kpd9eum@docker-desktop    |               {shutdown,1000},
rabbitmq_rabbitmq.1.22g75kpd9eum@docker-desktop    |               {child_type,worker}]
rabbitmq_rabbitmq.1.22g75kpd9eum@docker-desktop    |

This is the docker-compose part:

  rabbitmq:
    image: rabbitmq:4.0-management
    hostname: "rabbitmq-{{.Task.Slot}}"
    environment:
      RABBITMQ_NODENAME: rabbit@rabbitmq-{{.Task.Slot}}
      RABBITMQ_NODE_NAME: rabbit@rabbitmq-{{.Task.Slot}}

And rabbit.conf file

cluster_formation.peer_discovery_backend = dns
cluster_formation.dns.hostname = tasks.rabbitmq
cluster_partition_handling = autoheal

Per my understanding the Rabbit node discovery works, but the issue is related to DNS mismatch.

Also I have read the other suggestions that etcd or consul could be more appropriate fixes for Swarm setup. I can confirm that actually autodiscovery for etcd does not work as well, like expected, but I do not want to make the post long.

I hope I was able to explain the issue, and this will be considered as priority for next releases. I think Swarm is something very undervalued and often skipped due to confusion that Kubernetes is an alternative.

[michaelklishin]

etcd peer discovery will allow nodes to discover what their peers register but it won't magically make the nodes connect to each other. For example, the hostnames of other cluster node must be resolvable and stable.

According to Docker Swarm documentation, there are at least some ways for services to have stable hostnames but I haven't used them.

As long as nodes can reach a peer discovery mechanism and the values (node names, almost always in the format of rabbit@{hostname}) nodes register the values other nodes can resolve (as hostname) and connect to, a cluster will form.

If Swam hostnames are FQDNs, you will have to also instruct nodes to use long node names using an environment variable, RABBITMQ_USE_LONGNAME .

[lukebakken]

@dtila and others, thanks for taking the time to look into this. FYI, there is no need to @-ping people who have already commented on a discussion. They will be notified.

As a couple people have said here, the core issue is how DNS works in docker. I have run into the same problem using the DNS peer discovery plugin with docker compose.

@Radiergummi provides an idea to use the docker manager socket, would would work.

Also, I hope everyone on this discussion is aware of the classic peer discovery backend, in which you list all node names in rabbitmq.conf. This means you must use pre-defined node names, but it works great with docker (https://github.com/lukebakken/docker-rabbitmq-cluster).

Like @michaelklishin and I have said before, we'd welcome a contribution to get swarm working with RabbitMQ. Unless a customer with a support contract requests it, we probably won't work on it ourselves.