When one wants to run distributed Erlang between orchestrated Docker containers one needs to figure out how to deal with epmd functionality, i.e. the different containers need to know about the distribution ports of the other containers are reachable.
IIRC there were some recent features implemented in OTP to simplify that. Can anyone bring me up to date how the “right way”[TM] to do this with current OTP would look?
I think the simplest way would be erl -start_epmd false -erl_epmd_port 1234 so the nodes all use the same distribution port, and the epmd daemon doesn’t get started.
-dist_listen false and the node doesn’t listen for incoming connections but can still connect to other nodes (force sets -hidden though).
If you use k8s too try libcluster
Well I don’t intend to use Kubernetes (we are building our own Erlang based orchestrator) also we use Erlang and no Elixir. Was more interested in the low-level mechanisms
Then you might want to just implement your own epmd_module to integrate with your orchestrator.
For example: GitHub - tsloughter/epmdless: Distributed Erlang without the EPMD
I think i read a paper from you a while ago about an alternative Erlang distribution protocol, so i guess this would be high level 
We want to use the normal protocol but do a different way of node discovery. We ended up implementing our own Erlang -- How to Implement an Alternative Node Discovery for Erlang Distribution and have it talk to our orchestrator.
A quick look into epmdless shows it uses the same approach.
Maybe I do not understand what you are trying, but if you create a user network (non-default) you get internal DNS, so you can just do:
create the custom user network:
docker network create test
spin up a node:
docker run -it --rm --network test --name node1 --hostname node1 erlang:slim erl -setcookie test -sname test
spin up a second node:
docker run -it --rm --network test --name node2 --hostname node2 erlang:slim erl -setcookie test -sname test
After this you should be able to chat between nodes as you usually would:
(test@node2)1> net_adm:ping(test@node1).
pong
We need to orchestrate between different hosts, different data-centers and between Cloud, Edge and IoT/Fog nodes. All this in very heterogenous networks.
You’ve probably already run across this project in your research. It seems to be referenced by quite a few posts regarding Erlang+Docker.
https://github.com/oltarasenko/erlang_distribution_in_docker
Note that epmdless is no longer needed as of OTP23: Running Erlang Releases without EPMD on OTP 23.1+ · Erlware Blog
Is the orchestrator being developed in the open by any chance?
I can’t get the idea of building one out of my head ( which is how I landed on this post first place ) and joining an existing codebase would be great.
Yeah it will be open. Expect a first version in July
Great!
Really looking forward to checking it out
I use an (ipv6) overlay network - zerotier in my case, but any would do.
This takes the hard part of bridging all the different networks and firewall setups pretty easily, and adds a layer of encryption over the default without needing TLS.
This has been running fine since 2016 IIRC when I first set it up,
but I’ve not used it for very high traffic.
GitHub - zerotier/libzt: Encrypted P2P sockets over ZeroTier could very well be integrated into OTP if somebody had the enthusiasm.
I would imagine any other overlay network could work, such as wireguard or calico, however zerotier’s “flat network” topology presented back to the client, and inbuilt mesh routing, is particularly well suited to OTP.