Partisan - high-performance, high-scalability distributed computing with Erlang and Elixir

Your Libraries & OS Mentoring Libraries
1

Hi All.

In a recent post Chris Meiklejohn (of Riak KV, Lasp-lang, Partisan, AntidoteDB fame) announced that I will be taking over the maintenance of Partisan off him. See his blog post here :slight_smile: .

I am hard at work finishing on getting 5.0.0 release ready (currently working on my fork) and merging onto the official repo.

As part of that, I would love to hear from anybody currently using Partisan in their projects so that I can include a link in the upcoming README file.

Version 5.0.0 will bring some API changes so I want to make sure those who are currently using are aware and understand the changes, so any hint on how you are using it that might help me plan a transition and or prioritise documentation topics would be awesome.

Cheers!

15 Likes
2

Excellent news Alejandro! We are currently using Partisan for our digital twin technology.
We also forked Partisan and tried to keep it up with the new OTP releases. I’ll check out your repo to see if I can propose a contribution with our code.

2 Likes
3

Fantastic, I’ll post here when finally merging onto the main repo.

3 Likes
4

@erlangMax I finally merged my fork onto the main repo. Also published a new tag in Hex incl new set of docs using ExDoc.

Check it out at partisan | Hex

7 Likes
5

Thanks a lot! I’m giving it a run and check if something breaks in our applications.

2 Likes
6

Cool! I think you will see some breaks due to API changes. In general I am moving most APIs to the partisan module, which implements an API similar (if not identical) to the erlang and net_kernel modules e.g. we now have better monitoring of nodes and remote refs (when using full mesh topology through partisan_pluggable_peer_service_manager

2 Likes
7

So, I just release Partisan v5.0.0-beta.14 with several fixes to Hyparview and Client-server topologies, plus a new feature: Parallelism per channel!

Also some more docs

4 Likes
8

Hi All,

The latest Partisan v5.0.0-beta.22 adds more support for OTP :slight_smile:

This version completes the implementation of the following OTP behaviours and supporting modules.

  • partisan_gen
  • partisan_gen_event
  • partisan_gen_server
  • partisan_gen_statem
  • partisan_gen_supervisor
  • partisan_proc_lib
  • partisan_sys

They are all tested with the original Erlang/OTP Common Test suites adapted for Partisan.

All of them use partisan_monitor that implements node and distributed process monitoring (available for the full-mesh topology only for the time being).

Each module is a fork of the corresponding OTP module. At the moment we have support for the OTP24 versions only but adding support for 25 and 26 soon. The idea is that based on the OTP version you are running Partisan will dynamically compile the corresponding versions of those modules.

Monitoring APIs are exposed via the partisan module which tries to follow the function signatures and semantics of the erlang module.

I am working on extending rpc, erpc and pg support next.

Happy to report that this version is passing Eqwalizer and Dialyzer too :muscle: .

13 Likes
9

Awesome, I’ve been looking for resource with setup and examples to bootstrap my journey but to no available. Any resources will be very much appreciated.

2 Likes
10

Hi @dowusu I’m working on completing documentation. You can find them at hex.pm. The latest release documentation is here Partisan — partisan v5.0.0-beta.24.

I’ve recently added a cheatsheet: Partisan Cheatsheet — partisan v5.0.0-beta.24.

Hope that helps.

4 Likes
11

Thanks a bunch, perusing the docs & cheatsheet.

2 Likes
12

what is it difference between libcluster?

  • current I looking to send the message across nodes and store the message in the gen server, I want to explore more about partisan seems everyone is talking about it being a very high performance :face_with_monocle:
1 Like
13

Hi @chanphiromsok . As far as I know libcluster is a library that helps you form a cluster of Erlang nodes using several discovery strategies like DNS, static, etc using distributed Erlang (aka disterl).

Partisan is a replacement for disterl that offers a number of features not currently present in disterl. Two key features are the ability to have multiple TCP connections between nodes and the ability to choose a cluster topology.

Multiple connections: this is implemented using Named Channels where each channel can have a defined “parallelism” (number of connections), compression level and wether the channel is monotonic. Monotonic channels perform message coalescing.

Topologies: partisan offers full-mesh (like disterl) but also offers partial-mesh via HyParView, client-server and static toplogies. You can also bring your own by implementing an Erlang behaviour.

In addition Partisan offers message retransmission and causal delivery.

These features currently come at a cost. Because the mapping between remote pids and references to their respective nodes is done by disterl internally, Partisan has to encode remote pids and references itself as it doesn’t use disterl.
In Partisan v4 this meant you couldn’t use OTP APIs eg gen_server:call. With v5 this is now possible as I have forked the OTP modules to enable them to work with partisan-encoded pids and references.

3 Likes
14

I was play around partisan(V5) with elixir did not find proper way to use it hope documents release soon

1 Like
15

Hi @chanphiromsok you can just add the dependency in elixir . In your mix file add the dependency

{:partisan, "~> 5.0.0-beta.24"}

Then in your config file add

config :partisan,
  name: System.get_env("PARTISAN_NODENAME", "node1@127.0.0.1") |> String.to_atom(),
  peer_port: String.to_integer(
    System.get_env("PARTISAN_PEER_PORT") || "10200"
  ),
  peer_service_manager: :partisan_pluggable_peer_service_manager,
  remote_ref_format: :improper_list,
  peer_discovery: [
    enabled: false
  ]

There is no Elixir specific API so you just use Partisan as you would with other Erlang libs.

2 Likes
16

Thanks, you so much Do you have an available example with Elixir or Erlang?
if you don’t mind I want to ask a few questions

    1. How does node1 get NodeSpec from node2 currently I use the manual to get NodeSpec from node2 I try using node name pass to join and it returns {:error, :not_yet_connected}
    1. I have follow the cheatsheet join node manually and send message from node1 to node2 I see the message log in console sent from node1. How can I handle the message something ? ex: I want to handle these in gen server etc.
    1. Do you have a plan to write documents for Elixir?
1 Like
17

Yes my idea is to add docs and examples for Elixir, Gleam and any other BEAM language people are interested to use with Partisan.

I will prepare some example projects for erlang and elixir as well. In the meantime you can see a fully developed project using Partisan in PlumDB.

  1. the way you get other nodes specs (in production) is by using peer_discovery_agent with one of the predefined strategies eg DNS or with one of your own (this I guess is similar to libcluster).

  2. there are two ways to send a message to a remote gen_server:

  1. Yo use partisan_gen_server instead of gen_server (in Elixir I guess I might need to implement a wrapper? Can you implement an Erlang behaviour in Elixir? I am not very good in Elixir yet :blush:). In this option you will use partisan_gen_server:call() and handle the message in your `handle_call’ callback.
  2. Use partisan:forward_message/4 targeting the remote gen_server process either by name (if registered remotely) or via a Partisan-encoded pid you have to obtain previously. Then you will handle the message in your handle_info callback
3 Likes
18

Thank you for your patience, I will learn more about partisan and try to implement an Erlang behavior in Elixir (I’m new to Erlang but Elixir I play around for a year)

3 Likes