Erlang and open-source community

Something that I have reflected over for a long time is that Erlang developers usually build own libs instead of maybe using open-source or contribute back to a already existing library.

For me it feels like Elixir community is better in finding common libraries and contribute around them. This is something I feel is missing in Erlang.

And also one thing that comes around for companies discussing to use Erlang or elixir is open-source community.

So what could we do to help Erlang community to help each other more?

7 Likes

That might be because OTP does everything you ever need :wink:

Seriously though, I cannot stress how much easier life is when you have zero external dependencies.

3 Likes

I think this is true and might be something that negatively affects adoption in certain cases. It’s only anecdotal, but I know people who have shied away from the ecosystem because they were worried about overall maintenance burden. I think more open-source libraries that are well maintained would help fix the problem.

Particularly I think things like nicely formatted READMEs with getting started instructions, build status buttons passing, and release notes go a long way to building the confidence of someone looking at the ecosystem from the outside.

Maybe we need some kind of curated list to surface things?

3 Likes

That sounds good. Maybe also list of libraries of different things so people can see what’s out there.

And maybe from that help each other.

Not sure. But lately I have been in to many companies arguing for Elixir over Erlang. And that is when they already runs Erlang.

3 Likes

This really fits into my dream for the whole ecosystem where all the language communities help each other to create a larger, cohesive community around the VM/OTP/etc.

I’m not entirely surprised that Elixir is a common migration choice for companies that use Erlang. I feel like Prolog-like syntax hasn’t ever been hugely popular across my career, even though I personally enjoy it.

Perhaps this is an opportunity to come together and provide a first-class interoperability experience? Really show how all these languages can coexist so that problems can be expressed succinctly.

3 Likes

Maybe we need some kind of curated list to surface things?

:wink:

8 Likes

I think hex is great and that more should use it. Even OTP team for libraries that is own by them.

We try to use it as much as possible with Nova and plugins around it.

2 Likes

In the same line: There seems to be some tendency to create customized clones of existing things, with added or modified features that never make it back into mainstream OTP. RabbitMQ’s supervisor2 is a popular example.

4 Likes

Well, I’d say that teaching a supervisor (or whatever) some new trick that is useful for a specific use case is easy via cloning and customizing: No use/edge cases to consider beyond what you need, no pitfalls to take precautions against because your use case will not get close to any, no tests needed because (you think) you know what you’re doing, no real documentation needed (because dto)…

Making something suitable for mainstream adoption is a different matter (remember EEP 56? :sweat_smile:), going far beyond your original specific use case. Discuss the best approach, ferret out (mis-)use and edge cases, (discuss and) cover them, consider possible pitfalls, (discuss and) implement precautions, write tests, write more tests, write documentation, rewrite documentation…

Also, I guess that when you are tempted to clone and customize something, you usually need that new trick now, or at least soon. You probably have deadlines to meet, you can’t wait for the next (major) OTP release.

5 Likes

You do have a point… But I don’t think it fully applies to supervisor2. It brings mainly 2 “new tricks” (as you put it) to the table: the intrinsic restart type and delayed child restarts.

A variation of intrinsic was adopted into the mainstream with significant children, ie EEP 56.

The delayed restarts, while they serve RabbitMQ well, would have had the potential to make things very unstable/unpredictable if adopted into mainstream supervisor. I had a long dispute with @MononcQc on the topic on the EEPs ML before getting convinced, but yes. So that is mainly why supervisor2 will stay around. It’s fine for RabbitMQ, it’s dangerous for the public :wink:

5 Likes

Hex is great (especially because it has support for mix and rebar) but I’m not sure it counts as a curated list for potential new people looking at the ecosystem. Perhaps it could evolve to contain such a list of community recommendations though!

For me, all package indexes fall short of being a good ecosystem introduction because:

  1. Giving a new user a search bar and popularity numbers doesn’t help much if they don’t know the ecosystem or what to look for in packages.
  2. Popularity by download count gets skewed by CI platforms without proper caching solutions and every package index can attempt to handle this differently.
  3. There might be several libraries for a certain piece of functionality that have different trade-offs.
  4. Finally, it’s not very time friendly to someone needing to make an ecosystem decision for a project.

Basically, relying solely on a package index always struck me as a programmer’s answer to a marketing problem. I’d love it if there was a low-time investment way to show people how good the ecosystem really is that I could always point people toward.

3 Likes

I’m a total noob, so take this with a grain of salt. But when I first heard of hex.pm, I had a lot of difficulty finding anything related to Erlang. It seemed like most stuff there was for Elixir. When I asked around about it, I was told that with Erlang, most people just tend to put stuff up on Github instead.

When I was learning Python and Go, it was much easier to find relevant packages/modules. But again, that’s just as a newbie to Erlang. I’m sure once I’m more familiar with the ecosystem, it will become much easier.

3 Likes

One problem I see with dependencies and sub-dependencies, which even a curated list cannot solve, is that they may not play well with each other.

For example, say you need dependencies A and B, and they both in turn depend on, say, Ranch. But A was made for Ranch 1 (where, for example, ranch:protocol:start_link takes 4 args), and B was made for Ranch 2 (where that callback takes 3 args). But you can only have one ranch application in your project, not both (In the case of Ranch, there is a solution to provide both the /3 and /4 variants, but let’s assume neither A nor B do that, and A may really have fallen into neglect and is not likely to ever catch up). So you have a choice of keeping either A and implementing B yourself, or vice versa. Or to find an old version of B that works with the Ranch version that A requires (if that even exists, B may be very new), but then you can’t use any newer features of B.

Probably worse, a dependency may use Erlang/OTP features that have since been removed or replaced (ssl:ssl_accept vs ssl:handshake comes to mind). Now, if you need that dependency, you have no choice but to drop to (and stay with) an older OTP version even.

(I see no feasible solution for this, btw :wink: So it looks like the best way is to depend on as few other things (which you don’t have control over) as possible. Ideally 0.)

3 Likes

Yes this is a nasty problem! You’re giving me flashbacks of dealing with Python’s requirements.txt and constraints.txt files :grimacing:

I like how Go handles this at the import statement level, but not at the code organization level: versioning in the package import statement and using a package alias if necessary. How that could be automatically handled likely gets messy no matter what.

I think there’s probably a nice range here that is (0, orders of magnitude smaller than node.js). There’s probably a ton of projects out there which could benefit from Erlang and OTP, but can’t necessarily afford to have 0 dependencies (i.e. smaller teams, start-ups).

2 Likes

Great topic @Taure :023:

There was a discussion on the Elixir Forum some time ago about wanting to try and get an official library from a payment provider, and a few of us (including @lpil) felt it would be beneficial having these types of libraries in Erlang - because all BEAM languages could easily make use of them.

I think this makes a lot of sense because it means there would potentially be more people who could help maintain them (potentially anyone from any BEAM lang) and because these libraries could be used by all BEAM lang users it would mean that the potential userbase is greater, making it more attractive for large companies to provide official libraries. I think it’s also a nice way to support Erlang.

With regards to the more general Erlang/OS situation, I think a good and meaningful interop story would help Erlang (as well as all BEAM langs). We discussed some of this in What is Erlang's gateway application? - but in order to grow a language userbase I think you need a gateway into it - i.e. something that gives people a good reason to use it, and that is usually via a good framework or some other killer application. In Erlang’s case, a better BEAM interop story could mean it could benefit from all of the gateways of each BEAM language. And for BEAM lang users it could mean they could make use of the best that each BEAM lang has to offer.

5 Likes

Well, it’s usually possible to adopt the code to support both versions (of OTP or Ranch or whatever).
For example, in gen_smtp we were able to support both Ranch 1.x and Ranch 2.x. In epgsql and jesse we suport OTP versions from 19 to 24 with really low effort (a couple of macros here and there).

2 Likes

Yes, I know :grinning_face_with_smiling_eyes:

… and yes, I know. I’ve been working for MySQL-OTP, which claims support all the way back to R16B even :wink:

So yes, it is possible, but that’s not my point. This isn’t in your hands as a user of dependencies (A and B, I called them before). It is in the hands of the maintainers of A and B. If one or both of them are neglected (but well, would still be oh-so-useful for you), it’s not going to happen unless you do it yourself, via a fork or clone or something.

2 Likes

There’s two problems.

One is that Erlang doesn’t have a package manager/index as has been mentioned here. It is difficult to find Erlang projects. And no, Hex.pm doesn’t count because you cannot filter out non-Erlang projects, which is a big deal, and you cannot really idle browse projects either anyway (for when you don’t know what you’re looking for exactly).

The other problem is that the number of developers that can dedicate time to work on open source projects is small. So even if someone does create and publish something, by the time you try to use it that developer might have already moved on and cannot update/support that project anymore. If there was strong interest and many users before that happens, the project will most likely continue in some form. But otherwise it’s the end of the line. For low activity projects it can also be difficult to determine if it is actively supported or dead.

By the way, there is no such thing as an “open-source community” or “Erlang open-source community”.

3 Likes

Read carefully, the thread title is “Erlang and open source community”, the two are set apart :grimacing:

3 Likes

And… if this is no such thing as an Erlang open-source community. How do we support each other to create one?

2 Likes