Thank you for the interesting question, and sorry for the late reply.
The real picture is more complex than the benchmark, so let me first introduce a few base concepts and then answer your three questions directly.
**The unit of information is a tag**, much like a classic SCADA tag. A tag has a schema: which fields it has, their types, and so on. Tags are data, not processes. They live in a database engine that supports subscriptions. In the simplest case you subscribe to a tag and the engine notifies you when it changes. In the more complex case you subscribe to a query (think of a classic SQL query), and the engine notifies you when any object matching the query is created, updated or deleted. That engine is a big story of its own, and I hope to write about it soon.
**The processes are the actors**: runtime services that read and write tags. The most typical ones:
- **A device connection.** One process per connection to a field device (PLC, energy meter, any smart device). It mainly receives updates from the device and writes them into the relevant tags, so from Faceplate’s point of view it is mostly a producer. But it also writes some points back to the device, setpoints, PID settings and the like, so it is potentially a consumer as well.
- **A user script.** It subscribes to its input variables (tag fields), recalculates when they change, and writes the results to its output variables (also tag fields). Consumer and producer in equal measure.
- **An archive.** It subscribes to its source tag, or to several tags if the archived value is derived from them, and writes the result into the timeseries database. Mostly a consumer, but other services may be interested in the archived value, so it is potentially a producer too. The timeseries DB deserves a dedicated story. The short version: RocksDB underneath, through the [erlang-rocksdb](GitHub - EnkiMultimedia/erlang-rocksdb: Erlang wrapper for RocksDB. · GitHub) binding, with many thanks to its author benoitc. The store is sharded by time (we use monthly shards), with a pool of writers in front. An archive process does not write to RocksDB itself. It sends to the pool, and each worker accumulates a batch and writes it in one go. That is what lets it keep up with a really high throughput, and we have solutions there I hope to share soon.
- **A user session.** One process per browser connection. When a user opens a screen (a mimic diagram), the session subscribes to the tags on it and pushes updates to the screen elements. Mostly a consumer, occasionally a producer, when the user changes a setpoint or a setting.
- **High-level services**: the state estimation engine, prediction services, the electricity grid graph, AI agents (dispatcher assistant), and so on. Mainly consumers, with occasional writes back when they have something useful to say.
So there are no pure producers or consumers, and the overall pattern is many-to-many. A write does not stop at the node where it happened: it is shipped to the neighbouring nodes, and on every node the engine fans it out to the local subscribers.
Writes are not the only traffic between nodes. Actors have to synchronise over the tags they write to, so there are distributed locks, and every acquire and release crosses the same node-to-node channel. We have an interesting solution for that too. It is the next topic I am working on, and it will be ready soon.
Now to your questions.
**Per tag or per device?** Per device connection, per script, per archive, per session, per service. Never per tag: a tag is a record in the engine. A node runs a large population of such processes, each quiet until its device, its inputs or its user has something new. That is why the benchmark in the article is shaped the way it is.
**Many senders and few receivers, or the other way round?** At the application level it is many-to-many: many writers on one node, many subscribers on the next, and most processes are both.
**Are the problematic receivers the ones doing historical storage?** No. The timeseries archive actors are ordinary subscribers, and they are not slow, thanks to the design above: an archive actor only hands its value to the writer pool, and the pool batches its RocksDB writes. The bottleneck was not a receiver at all. It was the sending side, the distribution output queue to the neighbouring node, as described in the article.