Based on the discussion in Heroku replaces its Erlang-based router with Go , I wanted to start a new thread and have a focused discussion around libraries in the ecosystem (both non-existent and existent) :
What do you think our ecosystem is missing in regard to libraries ?
What functionality do you think is missing in existing libraries?
What libraries do you think seem like they need help? I think sometimes libraries either in need of help or they just work and never see “updates”. In the latter case, people unfortunately may think they are “dead” or “no good” (people problems).
To keep things on topic, I’d like us to refrain from the following :
Heroku, I’d just refer to the the aforementioned thread if you’re interested in beating that dead horse
Getting things into OTP. While there are certainly some things we can say would be great to have in OTP or removed for that matter, I’d encourage participants to refrain from those type of suggestions.
I’m terribly interested in listening vs being chatty on this one. Please share your thoughts. Proposals for action are ok, but I think it’s important gather a list of deficits (so to speak) to begin with.
I can’t say, up to now I found something to answer for any need I had. But read on… →
This is actually a tricky question. In the projects I encountered over the years, it wasn’t so much that some functionality could not be found in any library, but that no single library had all the functionality I wanted. Some functionality A could be found in a library but then it missed B, another had B but missed A, etc.
MySQL/OTP is such a case. It is alive and kicking, but nowadays rarely sees updates.
I think MySQL/OTP grew out of a work-related project of @zuiderkwast. In my last job, I needed a MySQL driver for Erlang. This one looked promising, so I started using it. I also needed some features that it didn’t provide back then, so tl;dr I started contributing to it. @Maria-12648430 also made contributions, out of idle curiosity I guess (or maybe because MySQL/OTP works equally well as a MariaDB driver )
In my current job, I don’t use MySQL any more. I think it is the same for @zuiderkwast. So there is no incentive on our side to add new features. Also, there is little to no demand for changes or new features from users, so I would say it is near feature-complete, at least for the features that users require. Bug reports are also scarce. In a nutshell, updates are rare because all we could reasonably update is the copyright years. It is not abandoned - it just works.
That all said, what we could use help with is setting up automated testing via GitHub Actions. We had that on Travis CI back in the Old DaysTM, but ever since that went all testing is done locally.
The only thing that I really pine for is something like the Python package XlsxWriter.
I’d like some more industrial communications protocols but I don’t imagine that is a common need and have no problem implementing them myself or waiting for Peer et al to.
The main thing we keep hitting up against in the Gleam community is a way to reliably install Erlang.
macOS is OK thanks to brew, but the package is currently out-of-date.
Debian and Ubuntu Linux have an Erlang package but it’s very out-of-date and doesn’t contain all the Erlang applications so projects will fail in cryptic ways.
Windows has a bunch of confusing hoops to jump through to get rebar3 installed and working.
Compared to other ecosystems going from zero-to-development can be challenging and off-putting for newcomers.
I think what we need is not necessarily libraries and frameworks, but rather to work around building and/or strengthening community and ecosystem around them. The question we need to answer is:
How easy can we make it for a new person in a team to join an ongoing/existing Erlang project?
Below is what I believe, potential answers to the question above:
Installation of Erlang/OTP
For one, we need to make it easy to download/install Erlang/OTP itself across all supported platforms. We kind of already have that, but maybe only 80% of it and it’s quite fragmented, especially on Linux and Windows. Between the Debian official repos, the Erlang Solutions repos, the official Docker images, it’s difficult for newcomers to know which one to use. The Erlang.org website had a download page, mentioning OTP 27.0.1 but then goes on to say apt-get install erlang for Debian/Ubuntu which will most probably install 25.2.
Package Discovery
The second point is, is that we should do something similar to this: https://packages.gleam.run/ It’s effectively a mirror to hex.pm, but filtering to only show gleam packages. Discovering available Erlang libraries is not always easy on hex.pm. I personally have to resort to GitHub search to find useful libraries.
Erlang website
This leads me to the next point, which is to improve/modernize the erlang.org website, both visually but also content wise. Visually, it’s very information dense at times, and there is a certain lack of information hierarchy that if addresses, would help a lot in finding the necessary information quickly.
Focused work groups
Finally, I think, and I might be completely wrong here, that it would be great if the EEF organized focused work groups that are time bound and have a narrow mission of addressing a specific need:
For example we could have a “Database connectivity” work group, whose purpose is to implement and document a set of libraries to simplify database access. Once the goal is reached the work group would then be dissolved or reduced to a basic core team, at that point maybe a new focus group such as “Document processing” or “K8S support” would be started.
Having well-maintained generic components you might use as dependencies in your projects is a massive win for the ecosystem and its perception. Coming up with a list of those is the easy part. It will certainly contain libs from at least the following domains:
Database drivers/clients (PostgreSQL, etc.)
Streaming log clients (Kafka, etc.)
Process pooling
HTTP/TCP stuff
Maybe distributed protocols (Raft)
Telemetry
Tooling
The hard part is how to support and pay maintainers. I don’t see any other options besides some sort of commercial entity paying developers, with support from the entire ecosystem, of course.
I would like to second your ask for easier package discovery.
Here is a recently maintained awsome-erlang list. It is one of the only sources of Erlang libraries. Unfortunately, the maintainer has stated he doesn’t have too much time nowadays to maintain it.
I wonder how difficult it would be to support a project like this in a more official capacity.
A big thing on my wish list for Erlang ecosystem would be http clients for common APIs. One I would like to point it is the Stripe API. The lack of a good Stripe client library adds just one more reason to not use Erlang backend for your startup when it could be faster to build with Elixir, Javascript, Python, etc.
On package discovery, how hard would it be to add such filtering to hex.pm directly? It is something I’ve myself missed often, it doesn’t seem like something specially complicated, and all BEAM languages could benefit at once
OTEL, test frameworks (property testing, etc), LSP, rebar3, relx…
To be clear, I don’t mean to list deficient or unmaintained projects, but identify projects of high value to the ecosystem and Erlang projects in general.
Also, thank you for your work on those!
I realized that there is filtering on hex, it’s just completely undocumented, and seemingly collectively forgotten about by everyone. It’s not the most pleasant to use, but it is usable. I just opened a PR to add information on how to use the filters to the search page.
The basic idea is description:foo description:bar will find packages with both “foo” and “bar” in their description. You can also search for something like name:foo* to find all packages that have a name that starts with “foo”, or even find packages that depend on “ecto”, for example, with depends:ecto.
I figured I’d share this here just in case the PR takes a bit to go live, and maybe it’ll help some people find some tricky packages. As a fun example, packages that mention elixir and gleam.
)