Finally done, this commit makes all nodes to gen_servers and thereby making the introduction of supervisors for flows possible.
Big thanks to all for the good advise and patiences!
The structure of individual nodes is now fairly settled, especially as I’ve implemented quite a few nodes and each had their own requirements. I hope there is now enough functionality to easily implement new nodes and add functionality to Erlang-RED.
At the moment ~20 nodes have been more or less implemented and there are ~60 test flows to ensure that functionality doesn’t break and is compatible to the original Node-RED.
Many thanks again to @mwmiller, for keeping the test version online. In the coming days, I will improve it so that it becomes more interactive and even useful
I’ve now created an architecture document but - spoiler - it has no diagrams. It is description of two workflows and how they are implemented in Erlang, i.e. deploying a flow from the flow editor and executing a flow - via the flow editor.
As I write in the readme, there is much interconnectedness within the code simply because of the nature of the implementation and types of nodes.
Quick update on the project, I bumped into milestone 3 today and it irepresents a solidification of the development process - flow tests have been outsourced, JSONata has been outsourced etc
As some of you know, I added Elixir support, basically anything that compiles to the BEAM should be supported - is my opinion!
There is now a function node that allows for direct execution of Erlang code within a flow. The node even has - rudimentary - syntax highlighting for the code. Monaco - the underlying editor - does not have an impressive Erlang mode unfortunately - but COBOL yes!
The function node is a bit deceptive since it has the incorrect default code:
fun (Msg) ->
Msg
end.
is wrong, that’s the wrapper placed around code that is put in the function node, e.g. maps:put(fubar, hello, Msg) would be valid Erlang code for the function node.
Msg is a map and contains the details of the message sent to the function node.
As always, there is a live demo available to play around with.
So having spent much time trying to get my head around supervisor behaviour and creating some kind of Erlang architecture that works, I wrote a architectural document for how one node type is implemented within that architecture.
What I describe is the function node that allows executing Erlang code within Erlang-Red - basically wrapping Erlang code around a message-passing flow-based paradigm.
Not to spoiler but in the end I came up with this process architecture:
Two features made it that complicated: dealing with supervisors and handling timeouts for a function node. Both now work and I’ll probably stick with that architecture.
I don’t use process links, instead I’m using monitor for one-way linking but that’s all described in the text above!
It describes my use of pg to map between processes and nodes, may be there is a better way of doing this. It also describes how messages are passed through the flow without need to reference the original flow - messages are passed as Erlang messages between Erlang process yet they appear to be nodes in the flow editor.
I also created a second document describing the creation of Rust-Red which describes how I went about creating Erlang-Red. The intention is to encourage others to think about flow-based programming for their favourite programming language and since flow-based programming is best done visually, how to implement a server in programming language XYZ with a browser-based Node-RED Javascript client.
I encourage anyone with questions to PM me either here or elsewhere - most happy to answer any questions of relevancy.
I’ve continued working on Erlang-Red and have actually found - for me - something useful to do with it: creating a basic MQTT broker as a visual flow. The flow also has test code (using MQTT nodes) and the state machine for storing subscriptions (that because I didn’t a gen_server node).
And actually implements MQTT v3.1.1 but for QoS 0 only. A more detailed description provides the why I did this and how I did this. There is an online version running over here.
Clicking the deploy button should start things but YMMV because it is only a small server and might well be overloaded. To interact with the flow, scroll down and follow the steps:
Conceptually I’ve called this is a breadboard because for me, it’s coding using a breadboard: I can replace and reconnect and reorganise as if I’m using a breadboard.
As far as future work goes, I’m just adding a gen_server node to Erlang-Red and then will either extend the MQTT broker idea or move to some other binary protocol.
As a further update, milestone 6 was found hiding behind a hard place.
Perhaps the most interesting “idea” is the concept of breadboard programming which basically is using a visual tool to quickly create code to test ideas but also be able to expand out and easily change that code. The MQTT example above is a case in point but also the telnet session flow.
Breadboarding with Erlang is perhaps especially useful because with the addition of a few supervisor nodes, suddenly there is stable architecture. Another good example would be creating a Kafka consumer on a breadboard - it would be very simple to test and iterate over the consumer, being able to rewind Kafkas index for the consumer. When the consumer is completed, add some supervisors and notification code and done. From breadboard to production in one final deploy!
Thank you for supporting this work, it’s a long road with many failed/abandoned attempts along the way - so why do I continue?
This isn’t a low-code visual code solution for Erlang, my aim is to construct a developer-first visual workflow/breadboard tool. What does that mean?
As a developers, its important to have existing tooling that we are used to using, available in visual form. This is often not the case for many visual/flow-based tools. It starts with a simple operation as diff’ing visual code.
Many flow tools rely on a text-based diff which makes no sense at all. Whats the point of working visually if I have to switch back to a text representation to compare versions? My solution is the flow compare plugin for Node-RED - which I ported to Erlang-Red.
What about version control? I want to be able to commit, pull and push my code visually. For that I created FlowHub.org which is fully integrated into Node-RED and also ported to Erlang-Red.
Unit testing? Assert nodes which help me to create the flow testsuite for ensuring compatibility between Erlang-Red and Node-RED.
None of these tools are perfect and there is much that I have missed. However my approach is focussed on providing the tooling that developers expect - a kind of visual coding tool aimed at developers but which can be understood (not necessarily used) by non-developers.
Thank you - great motivation for me and I hope folks have try of Erlang-Red and get a feeling for flow based programming
I did a small embedding experiment using a raspberry but that was using a docker container. I added a short description to the project. The point was to get the I2C out node working using the circuits_i2c library.
TL;DR: I’m still working on it just not full time at the moment.
What has changed?
EEF Sponsorship - Many thanks to Lee Barney and Alistair Woodman for making this happen - yippee!
Elixir support via a module node, I posted about that over at the Elixir forum. There is still much to do but its a step in the right direction!
An initial implementation of process nodes have been added and I describe their usage in a screencast to be found over at the project readme.
There is also the VDA5050 collection of screen-casts presenting a HowTo combine Node-RED and Erlang-Red to build a bot and brain simulation.
Together with @vkatsuba, I’m now involved in a project investigating the application of Erlang-Red in a Corporate Industrial Environment.
Interestingly what nobody has talked about but something that I’m now working on is getting ASTs visualised in Erlang-Red. So the background is a) understanding a large Erlang codebase and b) finding spots that many be refactored or concisified (Erlang being a concise language).
So what better way to understand code than to visualise it:
That’s a 20k line Erlang module. What is being visualised are functions connected to any case or if statements in their definitions. For the time being, I’ve ignored other statements.
And even here we gain something - bottom right there is a case statement with many branches. So ordering by size already has brought an insight. The red-dots are non-compiled code - the existing code is imported and stored as function nodes inside Erlang-Red. The code gets compiled and the process started. The reason why the code isn’t compiling is because the way the function node works not because the code is incorrect.
The flow show is executable Erlang-Red (and Node-RED) flow code. I can send the nodes messages and they will do something. VIsual meta programming based on existing Erlang code imported into Erlang-Red. My intention is to make the code executable inside Erlang-Red and then trace message flow using a plugin. Who knows what I will discover once I begin to fuzz the code with random messages
What then do is extract some flows - each function becomes a flow with case and ifs as subnodes. See on the right is the AST for a collection of random functions. Note also the function names have been minified (to protect the innocent) - I can expand them to actually see the function names. Note also, Erlang-Red now has auto-layout so that I don’t organise trees by hand
Basically the functions indentation level is been shown as a tree - the longer the branches, the deeper the indentation in the codebase.
The entire workflow to make this happen:
a Node-RED instance which accepts drag&drop github links.
upon receiving the link, Node-RED then retrieves the code from github (using API + token = private repo)
Node-RED sends the Erlang code (via MQTT) to Erlang-Red
Erlang-Red than generates the flow seen in the image inside Erlang-Red using an Erlang function code (epp and erl_pp).
This happens automagically so that I can go through github drag&dropping links to module code and having it visualised.
Going forward, I want to be able to create the codebase from the changes I make within Erlang-Red - that shouldn’t be too hard since the erl_pp and epp modules do all the heavy lifting.
