Reading through Distributed Erlang and other resources, Erlang makes it easy to connect nodes and pass messages between them.
But taking a step back I’m still unsure what I should use to create/manage a distributed application. I would like to:
- Run a service on less-than-all nodes in a cluster while still being available to all nodes in a cluster through messages.
- Be able to call a service by name without needing to know before-hand which node/pid that service will belong to.
- Possibly being able to run multiple of the same service on a node (in-node load balancing).
- Scale in/out a service either by adding nodes or electing existing nodes as servers of that service
- Be able to distribute requests for that service across nodes (cross-node load balancing).
- By either latency, number of processes, or some combination.
Lets say user_serv is on 111.111.101,
pay_serv is on 111.111.102,
and inv_serv is on both 111.111.101 and 111.111.103.
user_serv wants to purchase an item so it needs to use the pay_serv. The pay_serv needs to fetch details for that item so it needs to use the inv_serv. These services do not know where the others are, so they have to use the reg_serv to find a node to talk to.
I think I could come up with something like the above where I could add nodes to a cluster just by connecting it to one node and letting gossip between reg_serv do the rest. But that is a lot of custom boilerplate that- while fun to think about- is probably just reinventing the wheel.
What are people using to create/manage complex distributed applications? Is everything I want already part of OTP and I missed it or are there frameworks for this sort of thing?