What do we need for better BEAM interop?

BEAM Forum BEAM Interoperability
1

It’s a topic that’s come up a few times over the years - what would be needed for better interoperability between BEAM languages? For instance, to make it easier to use Phoenix in Erlang?

I touched upon this briefly with @OvermindDL1 a while ago, and he mentioned that having macros in Erlang would be a great place to start.

Personally I would love to see a BEAM world where we could use the best of each language in any other, particularly in Erlang as Erlang could serve to be an incredibly powerful gateway into the BEAM world (which of course it always has been by being the basis of all BEAM languages - but I am imagining more directly).

I appreciate it’s tricky topic though, and that we probably wouldn’t want to get to the point where any one of the BEAM languages loses what make’s it special and unique to begin with. But still, I’m curious to hear what people think :blush:

5 Likes
2

Macro’s in erlang I’m unsure would do much for interop, but it would certainly make erlang much less verbose (and it’s already pretty good with reducing verbosity, I mean more in scaffolding).

However, agreeing on a base syntax for all BEAM language, like core erlang or something that supports everything needed, with ability to write transformer for core erlang to core erlang to provide mappins and features like macro’s in languages and so forth would go a long way to providing a more common base, however it might be too far beyond this point.

3 Likes
3

I remain unconvinced. Maybe it is because I mostly use C, but I believe macros are fundamentally evil. Joe said he thought if you add something to a language, you should also remove something - for C, I’d pick macros. Oh, and typedefs.

3 Likes
4

Oh entirely agree with C-like macro’s, those are nothing but simple textual replacements. Elixir macro’s are stronger in that they are AST based but they are still not strongly typed. I prefer macro’s like how Rust does MBE’s, which are fully type safe AST replacements that, importantly, can’t run code. An example for those who may not know how they work:

// A simplified version of the `dbg` macro that comes with rust:
macro_rules! dbg {
  () => {
    $crate::eprintln!("[{}, {}]", $crate::file!(), $crate::line!())
  };
  ($val:expr, $(,)?) => {
    $crate::eprintln!("[{}:{}] {} = {:#?}", $crate::file!(), $crate::line!(), $crate::stringify!($val), ($val));
  };
  ($($val:expr),+, $(,)?) => {
    ($($crate::dbg!($val)),+,)
  };
}

It can be used like dbg!() to just print the file and line number it’s called on, or you can do something like dbg!(2+2) which will print out the file and line number, then 2+2 = 4 after that, or you can pass in many arguments like dbg!(2+2, 3*3, 4/4) to do it for each argument like calling dbg! individually on each.

A few things to note, all macro’s are called by their name with a trailing !, so nothing is hidden. The ast matchers in the macro_rules are tested in order, from top to bottom, replacing it with the specified body AST via direct replacements. There are a few special variables, like $crate is a path to the crate that the macro is defined in, rather then where it is being called at. There are a few “special” macro’s, like things like file! and line! and such are, which are supplied by the compiler, these are very few and far between and are used to be able to specify a minimal set of useful things for normal macro’s.

Stringly macro’s like in C are yes, quite evil, lol.

3 Likes
5

I don’t think Phoenix is a project to consider when talking about this subject. I’ve had limited interaction with it but as far as I know its power is from macros. Using it from Erlang just doesn’t make sense. Maybe something like its templating engine could be useful from Erlang, don’t know.

But there are small issues I’ve encountered that make general interop a pain. Most recently it has been nil and keyword lists. The keyword lists thing is just that Elixir users expect to be passing those for config or arguments instead of maps and in Erlang we often use maps now instead of lists, unless a proplist makes sense.

nil is a little worse since the expectation of using keyword lists is only when writing the code or configuration, because of the less verbose syntax available for the Elixir keyword lists.

Sadly I don’t think there is anything that can be done about these.

5 Likes
6

It is indeed very macro driven, but it gets its speed from cowboy anyway, it’s mostly just stacked middleware of a form and some macro-based DSEL’s.

The template engine is interesting, it takes the source text template files and compiles those at compile time to generate code inline so they just become normal function calls with normal compile-time-known strings, lots of speed that way, such a thing for that for erlang would be quite nice but you couldn’t really use Elixir’s EEX without having either macro’s (so some shim elixir code) or generating elixir files (so again still running some elixir code). Would absolutely be possible to generate erlang code from the EEX format though, but you’d probably have the EE(?) format use erlang syntax instead of elixir syntax unless you don’t mind including elixir anyway, which if you do you may as well just call the elixir shims then, lol.

1 Like
7

I’ve often said that Erlang needs something like eex in OTP itself. I’ve found it pretty invaluable in elixir project.

OTOH, I’d like to see OTP shrink in a lot of ways, so not on my TODO list :wink: I’m also unsure it would see a lot of use in erlang projects as things currently stand :thinking: Though, I could see that changing.

I think most of the major interop issues have been solved or are actively being solved. So, excluding macros, I can’t at least atm think of other things that are critical.

However, on the topic of macros, all things are of course possible : GitHub - lfex/nx: LFE support for the Elixir Nx machine learning library

^ It was also a great example of watching members of different communities come together to make a thing happen :slight_smile:

Now barring macros, I can think of little things that bug me atm. For example, if I want to use an elixir dep in an erlang project, the syntax is not so pleasant, but it’s also not a huge deal. That said, an ergonomic syntax and standard syntax for when you pull in an elixir, lfe, gleam, etc dep in an erlang project dep would be great. While you can focus on making say a rebar3 plugin for say aliases and wrapping elixir calls to make the experience more pleasant, trying to devise a general solution, and optimally one that doesn’t sit at the project / package manager level, that’s tricky. Of course, and I believe pointed out, using macros from Elixir is also not pleasant, such that we see wrapper libraries like propcheck vs people just using proper (though, they could simply call the functions vs the macros). Hmmm, tat’s not totally fair, propcheck does a few other things besides wrapping proper, and FWIW it’s a great lib, but it helps make my point. All that said, I’m not sure such issues are the most important things to focus on atm :thinking: I go back and forth on it :grin:

Are there other interop issues people see that I’m not thinking of right now?

4 Likes
8

Isn’t it more or less what erlydtl does? Compiles template files to .beam file. The result is not a single function but module though.

3 Likes
9

I’d have to go through and find all the differences myself, as well as look at the differing implementations. One big difference between the two on the surface is EEx is general purpose, not necessarily useful for just web page purposes, but really in general.

EEx can be found a lot in mix tasks and generators as an example.

2 Likes
10

Ah, I see. I know rebar3 uses bbmustache templates rebar3/rebar.config at 8be7c3d28355566fb5e1a9d5f929197eb6eb7941 · erlang/rebar3 · GitHub. But this one is interpreted in runtime, not compiled AFAIK.

2 Likes
11

Yes, very similar, except EEX has pluggable backends so it can actually generate a variety of things, like you can generate a function that just takes a map to fill in values in the environment to you can make a whole closure-centric thing that allows piecemeal filling in and updating for real time interfaces. A lot of it’s power isn’t just that it generates the code, but rather that you can plug in whatever generating backend you want.

4 Likes
12

For me, the number 1 thing missing is actually an elixir hex package s.t. I could seamlessly pull in Elixir dependencies into my Erlang codebase without having to install Elixir globally. As far as I can tell, this is not really a technical restriction. I had a ticket opened for this 4 years ago (Publish a hex package for Elixir · Issue #6210 · elixir-lang/elixir · GitHub), though I don’t expect that the stance of the Elixir team has changed on this subject :slight_smile:

4 Likes
13

There is 3 points in order of pain:

  1. The messy names that have to be figured out and are quite ugly when calling Elixir from Erlang
  2. Elixir compiler and libraries need to be possible to be a dependency like all other languages on the BEAM
  3. Toplevel APIs which are relying on macros

As long as these 3 points are unaddressed I consider Elixir code to be unusable from Erlang and would not use any in Erlang projects.

Regarding full Lisp like Macros for Erlang:

  • merl is part of OTP and offers that
  • Full blown Macros are a mixed blessing – wrote quite a bit of macro rich Lisp code in the past (went all the way through to macro writing macros) nice to write, super problematic to read and especially to debug.
  • If I really want Lisp macros I’d rather use LFE, Lisp syntax is optimal for macros, everything else is a crutch anyway. If one avoids them for the API interop with LFE code is seamless.
10 Likes
14

I don’t actually find that an issue, it’s just a namespaced name that uses dots instead of underscore, still very natural in erlang. Now macro’s on the other hand, those work only on elixir syntax, it has no compat with being able to consume erlang even though it definitely could be made to do so (though would take a bit work, though not any more than most things around it).

Yeah this is big, even kotlin is available as a dependency for java programs…

Yep, back up again with the macros, lol.

Parse transforms are indeed very powerful, but yeah erlang could definitely use a more simple version, perhaps something like Rust’s macro-by-example macro’s (where Erlang parse transforms are more similar to Rust’s procedural macros, which run code).

LFE is awesome though, but yeah, its macros don’t work on other languages either, lol.

2 Likes
15

It is also worth adding that, if macros are added to Erlang, then the inverse problem starts to become true: it will be harder for other beam languages to interop with Erlang and, opposite to Elixir, Erlang is the foundation we all build on. :slight_smile:

7 Likes
16

If they compile ‘to’ erlang instead of erlang core however, then they could take advantage of them? Or if the language at least allows you to embed erlang in it (like __asm for C).

1 Like
17

I definitely agree with @peerst on this one. 'Elixir.Foo':'bar'() is not ergonomic at all. It’s a small thing yes, but it’s enough of a turn off for some folks that it decreases adoption of elixir libs in erlang projects IMO. I’d much rather have a convention similar to what Elixir provides :foo:bar()

Probably the best solution is a rebar plugin, such as rebar_mix that provides the option to define wrappers for you. This would have to be done on a per language basis and might be extremely brittle as well.

Maybe an alternative is to add a callback in compilers for these purposes. I think I had rambled about it going in Erlang/OTP itself so that a standard syntax could be defined for all BEAM languages (that aren’t erlang itself). I don’t remember the details, but I remember it ending with “No, no, that’s bad idea” :rofl:

Still, this is surely a solvable problem. However, before trying to solve this problem, we need better support still for elixir deps. It’s currently a bit brittle.

Edit:

Note that :foo:bar() would not work, it’s a contrived example :slight_smile:

6 Likes
18

Yeah exactly. In general macros in APIs make interoperation very difficult so we definitely don’t want to have them pop up in Erlang APIs too. Of course the attractiveness to make a DSL like API layer with macros is too high to really work out when they are there.

Thats one of the prices of macro advantages

3 Likes
19

I think there is need for design rules and guidelines for how to make an API / library that is reusable from all or most beam languages.

In these guide lines there could be:

  • Write the library in Erlang to make it as reusable as possible
  • If written in Elixir avoid using macros in the API or deliver and make it useful both with and without the use of macros
  • beware of dependencies! Avoid dependencies but if they are necessary think carefully about how they impact the usefulness of the component.
  • There should be recommendations about data types to use (and what to avoid) in the API
  • …

This is something for the EEF WG “Language Interoperability” to work with

6 Likes
20

Did someone mention macros? :wink:

^^ This.

4 Likes