Sharing a recent BEAM There, Done That episode that’s directly relevant to anyone weighing BEAM languages against Go, Rust, or Node for high-throughput systems under real production constraints.
Chris Beck of Bitcrowd (Berlin) discusses building an open-source implementation of BlueSky’s data plane, the component responsible for compiling the AT Protocol firehose into a renderable user timeline. The target load: millions of users, roughly 10,000 messages a second, with sharp spikes during real events (the team cites the November 2024 US election, where the network saw roughly six million active users).
From an Erlang/BEAM perspective, a few things worth discussing:
ETS did most of the heavy lifting here. The team describes it as already solving the raw-compute problem for storage and follow-graph lookups, since it’s a C-implemented BIF, without requiring a NIF for the bulk of the workload.
The one place pure Elixir/Erlang genuinely fell short was bitmap-heavy computation for the “social proof” feature (mutual follows at scale). The team addressed this with a Rust NIF via Rustler, implementing roaring bitmaps, accepting the tradeoff that a NIF crash can take down the whole VM rather than being isolated like a regular process failure under supervision.
The fan-out design pushes messages directly into user mailboxes on arrival, which gave them backpressure and burst absorption natively, something the team notes would have required an external queue and a serialization boundary in Go, Rust, or Node.
Chris’s closing advice for anyone picking a stack: start from a requirements list, not from what wins on paper or what a given framework already does well, and look specifically at which requirements cannot be solved any other way.
