I come from Flow Based Programming (FBP) which is centred around message passing between independent processes. Probably the best analogy for FBP would be a conveyor belt: messages are created between various stations that alter the message before the message is passed of to the next station.
Unix pipes are another example of FBP where data is passed through processes which each process applying changes/alterations to the data. Apparently Unix pipes were in fact inspired by FBP - but I’m not sure of that.
My first question: is there a Erlang framework that implements FBP ideas? Some high level component where a graph describing the interconnectedness of processes is provided and a network of gen_servers are created along which messages could be passed - that’s at least how I imagine it!
My second question relates to Node-RED which is a visual representation of a FBP system. For those who might have used Yahoo!Pipes - Node-RED is basically the same tool but more generalised. Node-RED is implemented in nodejs, hence the name not the colour.
A typical Node-RED flow is tree-like structure with a clear starting points (i.e. where messages are generated) and clear end points (i.e. where messages are discarded) and between which various actions are applied to the message. Example Node-RED flow for controlling an I2C bus (maps an HTTP endpoint to an I2C bus).
Hence my second question: is there a visual tool for designing Erlang processes and the messages passed between them? Ideally providing a certain amount of message monitoring and debugging.
Why do I ask? Because both Erlang and Node-RED are industry-related tools - Node-RED is used for IIoT, specifically collating sensor data from various IoT devices and routing that data (via MQTT) and/or providing dashboards.
I did not know that FBP existed, but it seems that I used a similar approach for my erlang applications.
In my projects I have similar scenarios and use-cases, so
I defined some building blocks for my primitives, like socket servers, protocols, databases or the like. Most of them are implemented as gen_servers.
And as each of them has a well-defined functionality, I have a limited set of messages, which they can pass to each other.
I use a structured map for the definition of the application, it is an extension to the supervisor.
The map contains each process in my application (all are named), and the names of its contacts, where it needs to send its data to.
I can read that map from a file, or generate it programmatically, but anyhow, I can have a different application in seconds.
A graphical tool would be nice, so far I only use observer in retrospective to check if the supervisor trees are built correctly. Of course it does not show the graph of the message-passing.
I hope this makes some sense to you.
PS: and to answer your questions, I obviously do not know if there is a FBP framework/library or any graphical tools. Otherwise I would gladly use them.
Thank you, that’s exactly what I mean as a backend, it does not appear to have a visual frontend. What it does have is a yaml specification for a flow.
I don’t quite understand how the whole works but I’ll continue to have a look.
Thanks for your reply! Your approach sounds a quite close to what I was thinking of …
The thing is that Node-RED consists of two parts, a jQuery/Javascript frontend that generates a JSON array structure which, in turn, defines links between individual nodes. It is an extremely simple structure, hence easily used for other purposes. This array structure is sent to the NodeJS backend that then executes/updates the flow structure on the server.
Normally the frontend is served up by the NodeJS backend and there is an intertwined relationship between the two (in fact there is also a websocket the provides realtime communication between server and frontend). However, it is possible to extract the frontend from the backend and have it run without a server - this is how the link I posted above works: a Node-RED flow in a serverless (i.e. static) Node-RED instance running (but only a few nodes are supported) in the browser.
The problem with NodeJS is that it isn’t multi-threaded, it is an event-based, single threaded runtime. It is fast because it’s event based and therefore also very well suited to message passing (since each message can be modelled as an event). What the Node-RED frontend does really well is model concurrency by simply having connections between independent processes (i.e. the lines model the flow of messages, the rectangles represent processes). State is stored in the messages and hence everything can be stateless and unrelated - although there are exceptions to this rule.
It’s a shame to be wasting all that on a single threaded runtime - tongue in cheek!
What I was thinking of was to create a simple Erlang backend for Node-RED flows by initially just interpreting the generated JSON structure with a few simple process types (i.e. nodes in Node-RED terminology). I was thinking of spinning up a gen_server for each node (nodes are uniquely identified by an id) and then have messages being passed around
But since my Erlang skills are extremely limited, I am looking for a good starting point upon which to build/experiment. (I have a basic understanding of the concepts behind Erlang and did do some coding with it a few years back. I also had a play around with Elixir but also well in the past.)
P.S. I also only came to FBP a couple of years back and was surprised not to have learnt about it back when I studied IT and learnt all about object, procedural, functional, list, rule-based languages.
I’ve now created a repo to demonstrate how I have kind of sort of imagined it to work!
It’s an extremely simple demo but the underlying workflow is this:
take a Node-RED flows.json file containing nodes and their connections
parse the JSON and create for each node a process.
each process is assigned a different function depending on the node type (Node-RED is low-code so there are nodes for doing switch (i.e. case & if), change (add/deleting entries in hashes) and debug (for printing contents of messages). So each type has a function and multiple nodes of the same time are each assigned an Erlang process.
generate a message and send to a random process. In the demo this happens to be an “inject” node that is responsible (in the Node-RED world) for generating messages. That process that alters the message before passing it on.
The demo is based on a simple Node-RED flow (see README @ repo) that has been exported as Json and then interpreted - no modification of the exported Json was made.
I hope this gives a better explanation of what I am aiming for and also provides some ideas for further discussion
