I just wanted to followup from my post of last week.
I’ve come up with Proof of Work In Progress (i.e. a quick and dirty hack) of my idea of combining Node-RED with Erlang. It’s called Erlang-RED (no prize for naming) and the code is over at GitHub. Over there is also an animated gif of what Node-RED looks like. I hope that makes it a little more clearer what I’m looking for …
For those not familiar with Node-RED, its an extremely simple why to design data flows between computational units. Each rectangle is a computational unit and each line is the flow of an data object (i.e. a message object). Each fork of a line represents a concurrent task because message objects contain state, the flows are stateless. Computational units only work on the data received in message objects. These are altered and passed on to the next computational units, i.e., Node-RED implements a type of Flow Based Programming.
These ideas lend themselves (IMHO) ideally to Erlang and hence this project. What I’ve done is created a process for each node and then just send messages to processes that represent the nodes, pretty simple really. I don’t store a graph of the flow, instead each process (i.e. node) knows to which other process it should send its altered message - using a simple naming convention for process ids.
I’ve used Cowboy for handling the html and websocket traffic - that was probably the most work! The Node-RED frontend is a scraped from a live instance and then all its API calls to the server have been simulated in Cowboy.
I would love to discuss whether others also find this useful, I got the impression from my previous post that a certain amount of curiosity.
As background, I came to Node-RED about two-and-half years ago and have been using it daily ever since. I created my blog.openmindmap.org using it, I also create a service for Node-RED also using Node-RED. So I definitely believe that Node-RED (and the ideas of FBP) can be used to created non-trivial applications in a visual manner.
Yes, as you say, “a certain amount of curiosity” certainly
At the miminum to explore a more direct integration between Erlang and Node-RED, instead of using protocols.
I think the killer app will be when I get it so far that I can do http routing and handle web traffic with it. This can easily be done with Node-RED using the http-in nodes. So once I have emulated that, I won’t need to play around with Cowboy routing syntax any longer
I guess this isn’t an integration in that sense, it’s taking the frontend replicating all nodes to be Erlang based. I aim to remain 100% compatible with existing Node-RED flows[*] so that a flow can be either executed using Erlang or NodeJS - but that’s a looooong road (with many stones).
[*]: perhaps not the ones that have Javascript code via function nodes
Btw would love to get feedback on my Erlang code, I know there are definite right ways to do things in Erlang and I’ve not worried about those things yet
Nice initiative. I used FBP before, but the context was data integration with Apache Nifi (https://nifi.apache.org/). I’ll take a look into erlang-red source code and try it.
I hope you get the time to have a look at the source code of Erlang-RED, not being a native Erlang coder, my code can definitely use some good advise
On nifi and Node-RED, if I understand correctly, nifi is designed to ingest large datasets, act and transform that data and then handle the storage of the transformed data, i.e. all the actions of a classic ETL pipeline.
Node-RED (and consequently Erlang-RED) isn’t designed for handling large datasets, rather many small packets of data. Node-RED comes from IIoT world. So many small devices generating many small data points that are directed, coordinated and managed by Node-RED.
It is possible to use Node-RED for coordinating ETL pipelines or CI pipelines or any other problem space requiring data coordination e.g. home automation, industrial settings or even websites since these are also just flows of smallish data packets.
I once tried doing ETL directly using Node-RED by using the streaming api of NodeJS - the flow for this ended up being rather large. Basically what I did was stream large datasets as individual data-points through Node-RED. It worked but wasn’t ideal. However it does show the flexibility of Node-RED to be adapted to other problem domains, not only IIoT. Node-RED has a great collection of node packages that provide all sorts of extensions.
Node-RED can be thought of visually creating Unix pipelines, i.e., one can install any command within the Unix ecosystem and have them all communicate via the pipe mechanism. That’s what Node-RED is good at: providing the mechanism for various commands to communicate with one another.
I have now posted a description of my development process, I call it Flow Driven Development - for want of a better name. The description also describes how I create flow test cases within Node-RED (the flow editor within the project) to develop the Erlang code base against.
I’ve tried to aim the description at folks that aren’t familiar with Node-RED so that it becomes a little clearer how I am using Node-RED for this.
The core aim is to create, update and run flow test cases from within the Node-RED, flow editor without touching an editor or terminal. Additionally though, all tests I do create in Node-RED can also be tested on the command line using eunit, so the rebar3 eunit also executes test flows.
Looks like building an ETL pipeline with Node-RED is possible but also it gets complicated to manage the size. I used Nifi to aggregate and transform IoT data and integrating with other services. The ability to stop, restart and redraw the data flow without stopping the system was very handy.
So, I got the Erlang-RED motivations. I’ll try to run it locally too. Looking at the code, I didn’t get if one Node is an erlang process or not. I need to run and check to get a better idea.
For source code level improvements:
use erlfmt, as rebar3 plugin
as you are using OTP 27, what about also use json module instead jiffy ?
instead eunit, did you try common_test ?
usually, in erlang, it’s common to prefix all .erl files with a prefix, like ered_ (could be any other that makes sense for the project). This is mostly to avoid module conflicts (remember that there is no namespace in erlang). It’s also helps to the overall project organization.
maybe think about switching to umbrella project
add release configuration to Erlang-RED
maybe reorganize the project in order to have supervisor trees according to a design
I can help you with these points. I’m mostly interesting to see how would be possible to design erlang process linked with Erlang-RED nodes. Because when you create a Node it should also have an erlang process, 1:1 representation.
This is kind of hidden away since nodes aren’t gen_servers - it’s basically just a spawn(?MODULE,Fub...) call. The initialisation is basically: parse json giving a list of maps, iterate through list creating named pid for each map, send message to all inject nodes to trigger the flow (inject nodes being message generators in Node-RED).
I’ve spent the last couple of days trying to scope nodes to websockets, i.e., multiple clients have different websockets to communicate with Erlang-RED but they all share the same set of node pids for executing flows - not good. So the codebase has become a bit of mess because I now pass a websocket name (WsName) through the node creation phase and then when I pass the WsName along with messages going through the nodes to execute flows.
I’m not happy with it and was thinking of whether its possible to have a container for pids, i.e., having a “process container” for each websocket and then there would be no need to scope node pids individually with a websocket name …
The browser (and hence the websocket) are essential for creating and testing flows, so this kind of important.
<off-topic>
Node-RED itself has the same issue and the community is currently gradually adding multi-user support for Node-RED. Node-RED is basically a single user development environment which is fine for most of the use cases it has been utilised for i.e. IIoT whereby one person is responsible for a setup. If Node-RED is to become more popular, then a concept for multi-user development is essential.
</off-topic>
I’m using .editorconfig[1] & erlang mode[2] for that but I’ll have a look at erlfmt
At the moment, my main concern here is whether I should move away from my 80 char max line length (exiting the world of vt100s) and come over to the widescreen world of 120+ char lines. I like 80 chars because they are sign of code smell if I have to word wrap my lines too often…
What is the consensus in the Erlang world? Is there a line length limit - implied or practiced?
I can’t remember why but something made me go to jiffy - I did use the inbuilt json module but then needed something and found jiffy … just as I’m using Cowboy for the http stuff, I thought jiffy was the go-to package for json. But I’ll reevaluate that since I’m not doing anything complicate and I’m not a fan of having dependencies for something that is builtin
I didn’t look at common_test - eunit came with rebar3 and I assumed it was the testing framework used with rebar3. I now got it to generate tests, one per test flow so I’m kind of locked in to using eunit here - unless there is a simple way to replicate this in common_test?
Since I’m creating test flows visually and storing them in the project[3] as json, I want to be able to execute them both in the node red flow editor and on the command line (for github actions or whatever) and get the same results.
The test flows will become a set of flows that can be executed on both Node-RED and Erlang-RED and will ensure compatibility between the two. Plus they provide a roadmap for the implementation - Flow Driven Development
This is my main pain point at the moment: how to organise the project. I started by creating sub-directories in the src directory and that has helped but I’m still not happy or rather something doesn’t feel right about the structure. But that could also be my minimal experience of Erlang projects - it’s just a feeling.
I’ve started using erlangred: v. nodered: especially in the http to distinguish what is original Node-RED API and what is the Erlang-RED extended APIs. That works well. In the codebase itself, it’s not that simple. I’ve used nodered:[6] for communication to the flow editor (i.e. websocket helpers). And there is nodes: and flows: but … I’m not happy with it!
What I really would like is to create a single module which consists of multiple files, something like an -import(Modle,Fun) but that automatically scopes all included functions to the module defined in the file doing the importing:
and then I could do ered:helper_one(..) - is that possible somehow? (Without defining helper_one in ered.erl and then calling helper:helper_one … )
Yes that would be very useful since, if I read it[5] correctly, I can then mix Elixir and Erlang code into the same codebase. But I do that once I’ve address the release configuration:
I first thought of something like creating a single .exe but I think you mean hot code swapping[4] (as I would call it!). This is something I really want to get happening but I didn’t know where to start - thanks for pointing me in the right direction.
This would help me to speed up my development process - I’m still stuck in a browser (edit flow) - edit code (in emacs) - restart server (make in the terminal) - browser (run flow tests) cycle. I’d like to shorten that to browser (edit flow) - edit code (emacs) - browser (reload server and run tests). So I have to gain an understanding of how to hot load code …
I would once I’m happy with the architecture. At the moment (see websocket issue above) there are too many unknowns for me to setup proper supervisors and gen_servers for the nodes. I’ve still got an issue how to start the nodes, I’m not happy with that either … (but I’m really happy with the overall project ;)).
So until I have the feeling that I have - an initial - final architecture, I don’t want to pour too much concrete around to solidify the codebase.
Thank you again for your very valuable feedback
I’ll probably remove jiffy now and then get into the release configuration stuff!
You can defined a common prefix like ered then for each subsystem you add one more prefix like http, then ered_http_empty_json, ered_server_flow_store. Name schema is important, I think.
I think it will not work as expected.
By ‘release’ I meant building a release using this approach: Basic Usage | Rebar3 and Releases | Rebar3, for installing it in a server for instance, not for development.
Sorry for the late reply, I decided I wanted to get some functionality done, so I spent the last couple of days creating test flows and some new nodes.
What I realised is that my workflow for creating and testing flows needs improving, so I spent some time creating shortcuts to make that more effective.
For the time being, I’ll keep the focus on expanding the functionality (i.e. implementing nodes and test flows) because that provides insights into what interface/behaviour is needed from the the nodes. It is also something that will always be needed for any project of this type (i.e. these test flows provide a standardisation for visual FBP applications based on Node-RED).
I did much renaming of modules and took on your ered_ prefix. Now most things (the servers are still missing) have the prefix but that should happen in the next couple of days. Adding the prefix made me also think about how divide up the functionality various modules. Still much to do in that regard.
For the time being, I’ll stick to jiffy and eunit simply because it does not seem as simple as replacing a module name, i.e., json:encode(…) and jifffy:encode(…) differ completely and I don’t really - at the moment - want to get into understanding how json:encode(…) works - it seems to have a complete different approach to jiffy.
The same goes for eunit - even though it does not support pending tests, so I had to create a hack to have pending tests. I moved to pending tests instead of failing tests for features that need implementing - makes far more sense than to have failing tests for functionality that does not exist.
Thanks for the pg kernel tip, I’ll have a look in due course. I’ve moved thinking about process isolation based on websockets to the backlog - for the time being. I have a bigger win - atm - if i focus on improving my workflow and creating as many as possible test flows to provide a solid base for replicating Node-RED functionality in Erlang.
I also fixed my code hot loading my wrapping my start script with a while [ 1 ] ; do make app-start ; done then I created a endpoint that does a halt(0), then I have a shortcut in the flow editor that calls that endpoint. I don’t mind having hacks as long they are easily replaceable!
I have the feeling that my basic Erlang approach isn’t putting up walls for future improvements, so I can focus on extending the functionality to make Erlang-RED actually do something useful even if its not conform to all best practices of Erlang. (I was very happy to have an extremely simple flow that reads a file, parses the content as json and then counts the objects - three different nodes each doing one simple task but when combined, some complex arose.)
Oh dear, to my complete humble embarrassment it was, in the end, a drop in replacement. One thing that did change and that caused some issues was that json:encode orders hash keys alphabetically (by default) while jiffy (by default) does not. I did have some code that was sensitive to this change.
I added a release stanza in rebar.config, so it should now be possible to bundle the project with all flow tests into a release.
I did a quick test and it does seem to work but I’m no expert and any feedback is welcome!
Thanks to @mwmiller there is an online version of Erlang-RED available. It kinda of gives a taste of where this is going.
It’s read-only and basically only good for going to the testing tab, pressing refresh flows and then pressing test all. Flows can be loaded into the flow editor by double clicking on them in the testing panel. Deploying them does say that they got saved but that’s a cunning lie - no changes or additions accepted at the moment.
I’ve kind of pivoted and focused on the creation of a test suite of flows for ensuring nodes work as intended rather than building an execution engine for flows - even though the tests are executed as flows (so the functionality is there but it’s not accessible via the flow editor). This test suite is intended to become a kind of repository of node behaviour which can be used to check conformity to Node-RED behaviour.
This is very much beta and buggy, it might be down or broken. I have endeavoured to use supervisors to mitigate the errors.
As part of getting this online, the release should be working and nearly everything is prefixed with ered_. So it has been a good exercise in cleaning up the project structure.
Also I started using the pg module for handling the concept of a catch node. What a catch node does is accept any and all exceptions raised by any other node within the same flow.
Since there is no direct link between nodes, there is no direct connection between any node and the defined catch nodes (multiple catches can be defined per flow). This is where a pg group does the job: using a name that corresponds to the flow id (that is common to all nodes contained within the same flow), an exception is posted to that group and catch nodes register themselves with that group.
