Markdown support - how best to do that?

[gorenje]

Hi There,

I would like to implement a markdown node for Erlang-RED and I’m looking for the best approach:

• there are a couple of libraries erlmarkdown and erlang_markdown but they seem unsupported or are they simply complete?
• then there is the Earmark parser that seems to produce an AST tree but not html or am I wrong there?

My personal level of engagement with Markdown conversion would ideally be md_to_html(Content) and then automagically everything is supported

Over at NodeJS, there is this wonderful markdown-it library - I don’t need all that but I would like to have complete support of the fireball markdown which it is seems earmark can do…

Cheers for any pointers!

[ieQu1]

Elixir has some markdown libraries. I believe ex_doc (which is currently used for building OTP documentation) uses this library GitHub - pragdave/earmark: Markdown parser for Elixir

Calling Elixir from Erlang is not super trivial and Elixir, from what I know, heavily relies on its own build system. So while it’s technically possible to run this library in the same BEAM VM with your other code, and maybe even include it in the common release, I’m not sure it will be easier than os:cmd("pandoc ...").

[rlipscombe]

@ieQu1

Elixir, from what I know, heavily relies on its own build system.

fwiw, you can build Erlang with Elixir’s build system, which makes mixing (heh, see what I did there?) easier.

[gorenje]

@rlipscombe

which makes mixing

which is something worth considering, even if I’d prefer to go the other way, i.e., compiling Elixir code to Erlang and have that executed, → gnixim then I guess

[gorenje]

So what I did was add a separate Elixir directory to my project and then reference the compiled BEAM code in my Erlang codebase.

The idea came from this post, i.e. the code:add_path(...).

What it looks like, in my code, is this line. Very simple and just what I wanted

[vkatsuba]

Of course this is your project and you can do with it all what you want but integration Elixir into Erlang project can add more complexity in future as for extending as for continue support it. Maybe better to create own library in Erlang? What is your requirements for parser?

[filmor]

You can use exerl | Hex to use Elixir libraries from Erlang (when using rebar3). Should nowadays work pretty much out-of-the-box, just add it as a plugin and add a dependency on elixir_full additionally to your Elixir dependencies.

[gorenje]

@filmor

You can use exerl | Hex to use Elixir libraries

Brilliant! I’ll definitely have a look at that.

Spent a few hours trying to get a release working which mixes the BEAMs of both the Elixir and Erlang codebases - no success. It’s much simpler in development than creating a release that combines both. So something that defines a structured pathway forward would be perfect

@vkatsuba

Elixir into Erlang project can add more complexity in future as for extending as for continue support it.

I definitely noticed that But what I’m looking for is a structured possibility to combine the two - there seem to be too many libraries that are Elixir based that could well be useful for ErlangRED.

@vkatsuba

Maybe better to create own library in Erlang? What is your requirements for parser?

The parser, in this case, is only an example of my long term aim: to create a visual flow based programming tool in Erlang.

But Erlang uses BEAM and BEAM is the basis for Elixir … I could have included the Earmark library as a dependency of the Erlang project and then used it directly but I wanted to create a pathway for adding Elixir code to the project.

I won’t go as far as to try to find a way to integrate Javascript code into the project - even though that would make a lot of sense because of the existing collection of nodes. It’s just BEAM-based languages that I’m interested in.

Why the effort? Because the original NodeRED project has some 5300+ nodes that can be imported into NodeRED. These nodes support all sorts of devices and services. It is illusionary to believe that I’m going to code up 5000 node packages in Erlang! But if others want to contribute and their code happens to be in Elixir, then why re-code that in Erlang when it’s all about the BEAM!

[vkatsuba]

Honestly, I’d strongly recommend not mixing Elixir into an Erlang project, even if there’s a tempting library available. Yes, both run on the BEAM, but the moment you start blending the two, you invite long-term complexity that’s hard to justify unless absolutely necessary.

Erlang and Elixir differ a lot in philosophy, syntax, and tooling. It’s not just about importing a function - it’s about introducing a second language into your codebase. That means future contributors will need to be fluent in both, and that’s not a small ask. It also means you’ll need to manage two ecosystems: rebar3 for Erlang and mix for Elixir. It’s doable, but it’s rarely elegant, and even more rarely robust.

Then there’s the issue of debugging and maintenance. Elixir’s macro system can make even simple code paths difficult to trace, especially from the perspective of an Erlang developer who isn’t familiar with the Elixir way of doing things. You lose the transparency that Erlang is so good at providing.

And let’s be honest: pulling in one Elixir dependency usually leads to more. Many Elixir libraries come with a trail of transitive dependencies that may bring in entire frameworks you didn’t ask for, or features you don’t need. Over time, this pollutes the project and makes things harder to reason about.

If the library is that important, I’d much rather reimplement what I need in plain Erlang. It takes more effort upfront, but the result is a coherent, consistent codebase that any Erlang dev can pick up without wondering why half the project feels like another language entirely.

Just sharing my experience - mixing languages in the same BEAM project can work, but in practice, it tends to create more problems than it solves.

[filmor]

I’m sorry, but I think you are massively overstating the problems one could have with Elixir deps in Erlang projects. I might be slightly biased as the author of the library, but Elixir is not hard to use in an Erlang system, especially since you can simply provide a wrapper module to make things more Erlang-esque.

@vkatsuba

Erlang and Elixir differ a lot in philosophy, syntax, and tooling. It’s not just about importing a function - it’s about introducing a second language into your codebase. That means future contributors will need to be fluent in both, and that’s not a small ask. It also means you’ll need to manage two ecosystems: rebar3 for Erlang and mix for Elixir. It’s doable, but it’s rarely elegant, and even more rarely robust.

Please give the linked readme a read. You don’t have to interact with mix at all. You don’t have to write Elixir at all (even though I think it simplifies things if you write a wrapper s.t. you don’t have to juggle 'Elixir.Bla' atoms everywhere).

@vkatsuba

And let’s be honest: pulling in one Elixir dependency usually leads to more. Many Elixir libraries come with a trail of transitive dependencies that may bring in entire frameworks you didn’t ask for, or features you don’t need. Over time, this pollutes the project and makes things harder to reason about.

That is exactly the same with any dependency you pull in, whether it’s written in Elixir or Erlang.

[nzok]

What is “the best approach” actually mean? What are the quality metrics you want to optimise?

• reliability => use an existing well tested and actively maintained program or library
• security => keep it OUT of your address space
• efficiency => how much you need depends on the volume of data you expect to process
• development time => use existing code, this is a common enough task.
  Nowhere in that do I see a requirement that the component be in Erlang or Elixir or LFE or anything else.

Seriously, this looks like a textbook case for using an outboard program like pandoc, and writing a small wrapper around it calling it as a separate os:cmd.

You DON’T want to have to maintain this component yourself so you don’t care what it’s written in. You will definitely want to have good support for other programs as components in Erlang-RED, so why not this one?

[gorenje]

@vkatsuba

Just sharing my experience - mixing languages in the same BEAM project can work, but in practice, it tends to create more problems than it solves.

Thank you for sharing your experiences As someone just starting out, it is great help to hear others and their experiences. Especially those areas where there might be dragons!

@vkatsuba

you invite long-term complexity that’s hard to justify unless absolutely necessary.

This is what I’m trying to avoid - exactly that complexity. On the other hand though, I don’t want to recreate all those libraries that already exist.[1]

I guess something like ErlangRED that is basically bring together a lot of utilities and isn’t therefore a single-task-done-well kind of project - it’s a collection of tools brought together under the single paradigm of flow based programming.

In that sense it is comparable to something like an operating system.[2] Following that metaphor to its logical end: the BEAM becomes the kernel and the various libraries become the thousands of Unix commands written in various languages, not just C.

That might well sound very ambitious but hey - yolo

@vkatsuba

Then there’s the issue of debugging and maintenance.

That’s also the case if I were to recreate all the various libraries in Erlang. Every bit of code has to be maintained, be it written in C, Ada, Prolog, Scheme, Lisp, BrainFuck or Awk!

Of course, it is impossible to say what is harder: maintaining libraries re-written in Erlang versus maintaining Elixir code inside an Erlang project. What is clear though, either way, it won’t be simple nor straightforward - and that’s why I’m doing this!

@vkatsuba

Erlang and Elixir differ a lot in philosophy,

Actually I see this as advantageous - having a broader experience and bring folks together with different experiences can lead to interesting results. I come from a Ruby background, learnt me some Python and ended up coding visual flows using Node-RED - for that I had to pick up Javascript ideas. I explicitly want to code in Erlang because it’s Prolog and I secretly loved Prolog during my Uni times!

For me it’s the concepts that you pick along the way that make a difference, I most certainly don’t mind mixing Javascript, Erlang, Elixir and whatever else in a project. Polyglot programmers are becoming rarer because programming languages and their ecosystems are becoming so rich and diverse but also repetitive i.e. Rakefile, Makefile, Jakefile, rebar, mix … they all do the same thing just slightly differently. To understand - broadly - what they do (e.g. build code together into a single form) is more important - IMHO - than knowing the syntax of each tool.

@vkatsuba

If the library is that important, I’d much rather reimplement what I need in plain Erlang. It takes more effort upfront,

I too see that, heck I just reimplemented JSONata because it is so central to Node-RED, hence it has to be available in ErlangRED. It’s not perfect and doesn’t cover all use cases but it’s there to be extended.

But I can’t put in that effort for everything, so I have to make a trade off. And so my trade off is to find a structured and maintainable method for integrating existing Elixir code - if that isn’t possible in a structured and maintainable way, then I would drop the idea. I certainly don’t want a hack solution that breaks by looking at it … sticky tape and sleepless nights solutions not welcome

Sorry for the long reply!

[1] = Markdown might seem to be trivially ported to Erlang but the edge cases soon also create complexity - so that’s the other rabbit hole. Markdown is fairly well defined and unlikely to change, but other libraries and protocols might not …

[2] = In fact, it has been said that the Unix pipe command | was inspired by flow based programming. So that is what I’m trying to recreate - the pipe command but visually.

[vkatsuba]

Thanks for the reply and I completely understand your perspective. But I think we’re looking at the issue from very different scales.

I’m not arguing that it’s impossible or inherently wrong to use Elixir in Erlang projects. In small tools or simple API integrations, it might work just fine - especially with wrapper modules and minimal surface area. However, I’ve worked across a wide range of domains where the stakes and complexity are far higher: from 3GPP mobile networking stacks to banking infrastructure and high-throughput web systems.

In all of these industries, I’ve seen real-world projects where Elixir was introduced as a “quick solution” to fill some gap. And in every single case, that quick fix grew into a long-term architectural burden. Eventually, you end up with a codebase that’s 70% Erlang and 30% Elixir - and now you need to maintain two languages that target the same runtime, with separate tooling, different idioms, and a mixed developer skillset. That’s not just theoretical - that’s real friction I’ve witnessed on production systems.

You mentioned that “this is true for any dependency you bring in” - but I’d argue it’s not quite the same. Erlang-to-Erlang dependencies share not just the VM, but tooling (rebar3 ), conventions, supervision design, error handling style, and debug experience. Adding Elixir changes all of that - suddenly you have mix , you have macros, you have pipelines and with , and developers unfamiliar with Elixir are slowed down, or simply block changes altogether.

Sure, Elixir can be wrapped - but wrapper modules don’t eliminate the need to understand the code you’re depending on. If something breaks, you will have to dig into Elixir code. And that’s where the divide becomes more painful - especially in fast-moving, multi-person teams.

And that’s not even getting into edge cases where you must patch Erlang/OTP internals - for example, when ASN.1 decoding behaves incorrectly, and you need a production fix now , not six months from the next release. If you rely only on Erlang, it’s trivial to create a patched build for production use. If your project has Elixir tangled in, things get murkier fast.

At the end of the day, Elixir is a great tool. But let’s not forget: it’s a parser and macro system that emits Erlang AST and runs on BEAM. Everything Elixir can do, Erlang can do - but not everything Erlang can do is straightforward in Elixir. That asymmetry matters in systems where stability, longevity, and deep customization are critical.

So no, I’m not overstating the problem. I’m speaking from firsthand experience managing projects that outgrew their original scope - and where introducing a second BEAM language caused more long-term pain than it was worth.

[gorenje]

@nzok

Seriously, this looks like a textbook case for using an outboard program like pandoc, and writing a small wrapper around it calling it as a separate os:cmd.

Indeed, that is also possible - there is even an exec node for doing that.

But the markdown node is a node in the Node-RED world and since I would like to maintain compatibility with Node-RED, it’s important -for me - to implement those nodes that are base nodes in Node-RED.

True the markdown node could actually do the command line interaction under the hood, no worries. But I assumed that there would be a Markdown library available and actually never thought of dropping to the command line.

My preference is not to drop to the shell but rather try to get some library to do the heavy lifting. That’s generally better for portability. Plus dropping to the command line also requires transforming data to whatever the shell understands. In this case, pipe would probably have done it but generalisations can be dangerous.

@nzok

You DON’T want to have to maintain this component yourself so you don’t care what it’s written in. You will definitely want to have good support for other programs as components in Erlang-RED, so why not this one?

True, absolutely true. All I want to maintain are the core functions of ErlangRED - these are providing the communication between processes, creating processes from flows and generally ensuring that things are working amongst nodes/processes.

But I also don’t want to drop into the command line to run calc 1 + 1 just to do an addition.

So where is the line in the sand?

I don’t know.

But it helps to check what the folks over at NodeRED have come up with, i.e., what they nodes do and what they don’t. So for me that is one guide. The other is of course what is available in library forms in the Erlang world. And finally, what are the use cases for something like ErlangRED?

I use it for creating webpages that are written in Markdown. Now if I dropped to the command line each time someone requested a page, then I would be using a lot of unnecessary resources. So I want the markdown node to be done as a library. Others will never touch the markdown node and won’t have the same use case …

[nzok]

Where is the line in the sand? When you are dealing with a COMPLEX structure. Have you read the CommonMark specification? It’s big. Frankly, getting my head around TeX was easier, and that wasn’t easy. Then if you want Markdown to HTML conversion, there is another large specification. A lot larger than the HTML 3.2 specification I started with. And the thing about HTML 5 is that it is not only large, it is constantly changing, or at least it was the last time I tried to catch up. And of course to get your Markdown to HTML conversion right, that’s another large and changing specification. All this without taking into account the many varieties of Markdown that are still around. These are specifications you have no influence over. They are large, complex, and written by people who seem to be terrified of precision. And people using your component will expect it to be bug-compatible with the converter they are using.

The syntax of Markdown is complex and the interactions between features are, well, let’s be polite and call them tricky. A markdown parser that gave you an AST together with an unparser from the AST back to text would be nice to have. There is a sort of community standard for Markdown ASTs, mdast (see syntax-tree/mdast at github). The tooling around that seems to be in Javascript, but it shouldn’t be too hard to provide an Erlang translation. But still, it would make sense to use existing mdast and hast providers and consumers rather than reimplementing them.

Some things, like JSON, are easy to implement. Some things, like XML, are harder. And some, like Markdown, are on the wrong side of the line.

[gorenje]

@nzok

Have you read the CommonMark specification?

I bet it was the butler who did it … No I haven’t had the pleasure and I probably won’t in this lifetime.

Funny though because Markdowns original intention can be summarised on a single webpage. What happened? Everybody added their extensions and edge cases … pity really because all it wanted to do was provide a simple method for documenting open source projects.

@nzok

expect it to be bug-compatible with the converter they are using.

nicely said “bug-compatible” - feature complete and bug-compatible software

@nzok

existing mdast and hast providers and consumers rather than reimplementing them.

Earmark does have a yecc parser - I guess from there to an Erlang package isn’t that far.

[gorenje]

Just to summaries what I did (for those that come after me):

@filmor

You can use exerl | Hex to use Elixir libraries

It worked and thank you @filmor for the help

I had to make some more changes to Erlang-RED to get it working consistently because I had a strange compile issue with the rebar path plugin that made my compiles inconsistent so I moved the Elixir code to a separate repo.

That repo is then a rebar dependency and the helpers are available in Erlang-RED.

Needed to update the release applications to include all the Elixir apps.

All of that just so that red-erik has markdown content!

Thanks to all for all the help

[ieQu1]

And some, like Markdown, are on the wrong side of the line.

If topic starter wasn’t bound by constrains imposed by the original NodeRed design, I would definitely derail this thread into “just use TexInfo!” territory.

I’ve experimented with multiple markup languages (even DocBook), and TexInfo came as the clear winner: it can output hypertext info files that don’t require browser to view, navigation in Emacs is more ergonomic than anything HTML-based, it supports indexing and offline search with very lightweight JS out of the box, it’s widely available, doesn’t need a ton of dependencies, etc., etc. It’s almost as if it was designed from the ground up for producing technical documentation… But as usual, the industry never settles on the right thing, so I can’t blame TS for having to walk on the wrong side of the line.

[nzok]

What happened to Markdown? One thing that happened to Markdown was that the original specification was rather vague and fuzzy. In particular it listed a bunch of features without saying anything about their interaction. CommonMark’s complexity is at least as much about trying to spell out how all the pieces fit together as it is about increasing the number of pieces. Markdown was never simple, just drastically underspecified.

This is an extremely common in IT. “Specifying” an interface by describing features informally and then expecting everyone to figure out the details. XML presented syntax without spelling out semantics, with much confusion resulting. JSON was supposed to be a subset of Javascript, but it wasn’t. The Ada standards have attempted to be rigorous (within the bounds of informality) but have needed a massive number of interpretations.

This is part of the reason why Butler Lampson, in his hints on system design, urged keeping interfaces small, advice I have ended up respecting every time I’ve ignored it.

[gorenje]

@nzok

advice I have ended up respecting every time I’ve ignored it.

KISS - Keep Interfaces Simple Simon

What I enjoy about the Erlang-RED project is that I don’t have to think about interfaces since all I’m doing is copying existing interfaces, everything is defined by Node-RED already.

And Node-RED has ten years of focussed development and is still going strong. So their design must have something that works

The only interface decision I make are how Erlang should emulate Node-RED behaviour.