Heroku replaces its Erlang-based router with Go

Chat / Discussions
, ,
1

I recently stumbled upon this blog post: So Heroku is on the way to replace its Erlang-based router with a Go-based one.

This would not be all that noteworthy, happens every day, and I could just say “Have fun with your new GF then and good luck”.

What is interesting is the reason they give for taking this step, which is for once not the shortage or costs of Erlang developers (they are backed by Salesforce, what do they care about money?), but that tl;dr (internal and) open source projects they depended upon declined, decayed, or were altogether abandoned, at some point forcing them to pin down on an Erlang version, in turn preventing updating other components and impeding development and progress in general.

Now, one might say that they could have contributed back to the projects they were depending and making a profit on. Then again, from a companys perspective, this is not what you bought into back when you decided on a technology. It is not even in your best interest to pay people to further projects from which your competitors can later benefit for free. And last but not least, some projects are really stone-dead, contributions ignored, nothing happening or likely to ever happen. There may be alternatives, but switching to them requires effort, which again may have to be repeated when that project dies.

I am worried about the example this sets. Erlang itself may be top-notch and at the bleeding edge of things. But to build something meaningful at production scale, you will always need additional libraries and projects to pull in. Erlang projects often seem to be one- or few-man shows, more than what is the usual case with open source projects, and thus disproportionally at the risk of dying with the interests of their maintainers, or their maintainers, period. And then, along comes something like Go…

In a nutshell, I can understand what induced Heroku to take this step. But it makes me worried. I think this is something that the EEF should be discussing seriously.

10 Likes
2

Erlang is not prepared for migrating to k8s, while Golang is native there.

  1. I don’t know if there is any other implementation of schema validator except our GitHub - flussonic/openapi_handler: OpenAPI server handler for Cowboy: erlang http server (maybe it exists, point me there please)
  2. Erlang has ancient grandfather’s ASN.1 first class, but it is a real pain to work with gRPC
  3. comparing to Golang, Erlang has a very poor support for code generation. Compare our pains with oapi-codegen/examples/client/cfg.yaml at main · oapi-codegen/oapi-codegen · GitHub
  4. Erlang with its wonderful code reload doesn’t fit into Pod management system. Why? Well, because neither me, neither you haven’t invested into it.

So, world is moving from Linux operation system to cluster Kubernetes operation system and our favourite tool is not moving there.

Meanwhile erlang remains absolutely mind-blowing tool for Linux OS. Our video streaming server remains technology leader among on-premises solutions. I have golang developers in my company and can tell for sure, that it is much, much harder to do it in Golang.

6 Likes
3

While it’s always sad to see people go, it’s worth keeping in mind we also have a steady influx of people coming into the community as well :blush: though admittedly we usually see this in the other BEAM languages like Elixir - tho they often trickle through to Erlang when they start making libraries that they would like to be Erlang and BEAM compatible as well.

And on that note (of libraries) I have in the past thought how Ericsson could have an impact - wouldn’t it be cool if they allowed some of the Erlang Core Team to dedicate a number of days per month or week to help mentor people creating Erlang libraries? Or helping others work on wider ecosystem libraries (i.e those not directly used by Ericsson) or even working on those libraries themselves to help bolster Erlang and make it more generally appealing? I’m sure lots of people would jump at the chance of working with a member of the mighty Erlang Core Team :star_struck:

I know they already sponsor the EEF (and obviously fund the Erlang Core Team) but I think Ericsson, could, if they wanted to, play an even bigger role in Erlang (it is their baby after all). I’m sure I could rustle up loads of ideas here and so if anyone from Ericsson is interested I’d be happy to explore this.

I also think that when it comes to large tech giants Ericsson is definitely one of the more respected ones (so increased involvement from them would, imo, certainly be welcomed). In fact, and I don’t think I have mentioned this on the forums yet, but many years ago I met the (then) CEO of Ericsson on a flight to Sweden from the UK - but he wasn’t in the front of the plane in first/business class, but right at the back with us commoners :lol: I believe he said this was because he doesn’t like wasting shareholder’s money, and that always stayed with me - as it obviously reflected the (more humble/ethical?) culture of the company. Incidentally, there’s another reason why many people prefer the back of certain planes - because in the event of a crash they are considered the safest - so he was smart too!

1 Like
4

Maybe @MononcQc can explain better the situation at Heroku.
He worked there and implemented most of their libraries (LogPlex, etc.).

1 Like
5

I don’t have insider information anymore and I haven’t worked at Heroku since early 2017. I had assumed they had already replaced the router.

There was two halves to hermes. The first was vegur, the proxying layer we had written and open-sourced. The other was hermes, the parent app that managed all state, certificates, auth, domains, and hooked it into vegur.

The things I imagine didn’t age well are probably:

  • vegur
    • a dependency on a fork of cowboy required for proxying needs
    • a bunch of internal but public libs nobody in oss would maintain
  • hermes:
    • lager (outdated but shouldn’t break)
    • aws libs (easy to update)
    • bitcask (was used for per-host TLS cert stores)
    • a bunch of internal but public libs nobody in oss would maintain

Interestingly enough, I recall a coworker and I delivered a functional HTTP/2 prototype version of the proxy back in 2016-2017 so there is definitely a way to do it. It used Joe Devivo’s libs that are loosely maintained and that I think Tristan since took over but don’t move fast. (Incidentally Joe and Tristan both were on our team there for a good while…)

So while I would expect the dependencies on a very old cowboy/ranch fork to be trouble, the main blockers in my mind would be the inertia of having lost most devs, and possibly bitcask.

Bitcask had a bunch of C code, it has seen a few updates in the last few years and maybe that was a blocker.

But unless things changed drastically since my time there, people internally were just not excited for Erlang anymore, and rewriting everything to Go (including Ruby and Erlang) was the objective starting somewhere in early 2016. All the people most excited about Erlang left before me, and a few stayed after my departure who were okay with it but that wasn’t their main thing.

Salesforce kept me on retainer for almost 2-3 years to help with rare logplex incidents before they fully replaced it (it took two if not three rewrite attempts to do it).

But like at that point I suspect any excuse is right to replace the old Erlang stuff, and the cost of updating OSS deps wouldn’t need to be high to tip the scales, even if it’s just forking to fix deprecation errors.

Either way I imagine that it’s not a question of “would it be too hard?” But one of how much work is it compared to the alternative they picked.

My assumption is they’re actively maintaining the Go stuff and they long ago decided to let the Erlang stuff rot. They decided that before I left and we had discussions about that.

I think they’d easily favor migrating rather than going back on the rot and wanting to re-staff teams to clean up and reappropriate our old libs that seemingly did the job running in prod mostly unmaintained for 7 years.

10 Likes
6

So, do I understand you right in that you think that this is some kind of excuse, or a secondary reason?

2 Likes
7

Curious about your pain with grpc since I maintain grpcbox. Plenty I want to improve with it, but if you happen to be using it and not an alternative (tho I don’t think there is an alternative that is actively maintained?) I’m always interested in hearing them in case they differ from my own.

I completely disagree that Eralng isn’t prepared for migrating to k8s. I’ve run Erlang on k8s for what will soon be a deacde I bet, it has greatly improved in that time with hardly any nudging required on the OTP team to improve support for container environments (nothing k8s specific, but things that help in k8s and apply to all linux container based enironments like how many active schedulers to start, the default for sbwt, signal handling).

Something lacking in Erlang that Go certainly has great support for, and I think Elixir likely has better, is k8s API tooling. I have my own I used swagger for but have always really wished k8s supported GRPC for their API because that works much easier IMO from Erlang than openapi/swagger. But that isn’t needed if you just want to run your application on kubernetes.

There is swagger for Erlang, but I haven’t touched it in many years, so can’t speak on it.

Hot code loading just doesn’t really fit the k8s model.

Again, not saying its perfect, but I don’t think it is at all a dire situation like you make it sound but actually pretty good these days. But also, esp when it comes to grpc, very interested in feedback on so I can try to improve the situation.

6 Likes
8

I think it’s true that the Erlang ecosystem moves slower and likely requires more self-participation into libraries than a language like Go.

Like even in 2016 if we wanted to rapidly adopt new Erlang versions, we’d need to fork and send PRs to some of our deps ourselves. It likely hasn’t gotten much better with the last few years.

However I think a lot of these things in a moderately small codebase like Heroku’s Erlang stack was easily manageable, and would have remained manageable had they not frozen versions for something like 5+ years.

I think most reasons people choose languages are excuses: “it’s what $company uses” or “right tool for the right job” or “it is more intuitive” are rationalizations of “I like it more”, and concepts like “it’s harder to hire for” are often largely “we like turning training and support into an externality”

So if they are excuses, they still tend to be attached to real ideas: a popular language means you have to pay less (and think less) in aggregate about training. Your engineers might be more easily replaceable for that reason and commoditized entities can be managed more straightforwardly. A more vivid ecosystem means you can generally use more open source libraries without concerns to their maintenance cost because the pool of people doing it for free is much bigger and shields you from considering the brittleness of your own current stack.

So my read is that when all of this stuff is already in motion and you’re 5-6 years down the road, small hurdles for teams who are bought into a tech become huge perceived blockers for teams who may consider that commitment to be a regression.

It’s possible that in isolation it would be technically quicker and simpler to update the Erlang stack than to migrate the rest. In practice it isn’t because it doesn’t align with where people are invested within that org.

It’s understandable and I can’t fault them for it. Part of my reason leaving was in being tired of fighting that fight and needing to go to a different environment.

10 Likes
9

so this is because we haven’t made it fit into k8s =)

10

But do we really need k8s, extreme high availability, grpc (complexity), etc.?

you_are_not_google
you_are_not_google943×1608 162 KB

5 Likes
11

More that k8s had to be made to fit it. But others wanting stateful applications have been working on making it fit k8s. But the original and main way of deploying applications on k8s it doesn’t fit and I don’t think there is any “making it fit”, nor should there be.

What is “needed” is a tool that utilizes things like the fact, I believe, stateful sets don’t restart when you set a new image value, along with either a sidecar, a helm hook or controller that injects the relup through a volume and runs the upgrade.

So sure, in that sense it hasn’t been invested in. But its still not a matter of investing in making reload fit the Pod management system.

2 Likes
12

I don’t think there is a shortage of open source projects around Erlang.

The problem as I perceive it is that many fall into disrepair and get abandoned after a while for one reason or another. Maintainers may lose interest or not have the time any more, for example. This is putting projects that depend on them in trouble. They either have to dive in them themselves, or change their code to accommodate for an alternative. In any case, this is extra work, which they would rather want to spend on their actual product.

What I would therefore ask the EEF to think about is identifying popular and important open source projects and watch their health, get and stay in contact with the respective maintainers, detect early when one of those projects is at risk of dying, if possible offering some form of help (which doesn’t necessarily have to be funding, it could also be assistance in finding good co-maintainers, etcetc).

13 Likes
13

Maybe commercial companies who use free software should get used to the idea of PAYING a reasonable amount of money to the free projects they use, even if they are free. Especially if those things are essential to their business.

Just an idea.

2 Likes
14

I don’t think that’s too surprising given Erlang’s age - which is also a positive as there are so many libraries around, and so many that were written a long time ago - so probably lots of things we can learn from them :003:

I think the EEF already have initiatives in place where people can seek funding for working on certain things (though please check their website to make sure).

I still think people might jump at the chance of being mentored by the Erlang Core Team mind, especially if they’re already thinking about working on a library or package :blush:

In an ideal world they would, and we know many do - tho I’d guess at a similar split that we see for many languages.

Btw please avoid hard wrapping when posting to the forum (it makes things harder to read on phones/tablets etc) thanks :blush:

1 Like
15

I like Maria’s suggestion. If there is a list of projects that are good quality and essential to the community, I am quite willing to invest time in them to keep them working. There is a difference, though, in keeping them working and developing them further. The latter often requires more than just knowledge of Erlang. For me it is a way to keep my Erlang skills up-to-date and learn new things. The EEF can play a key role in this process and also further shape its visibility and relevance.

5 Likes
16

erlang from version 23 is ready for running in cloud environments, and it has flags to aware of k8s or docker

17

I agree that statefulset/deployments/whatever they’re called this week can not be easily combined with hot code reloading. It can be done (I’ve done a demo of this at work), but it’s pretty awkward.
The solution, I believe, is to replace StatefulSet with an operator that doesn’t insist on rolling restarts being the One True Way.

4 Likes
18

I have pondered on this quite a bit, I’d love to have more focused discussions on the matter. I think there’s one other thing to think about : What libraries or functionality in existing libraries are we missing?

3 Likes
19

When EEF thinks about things like that it usually means some volunteer wants to pick up the task. We don’t have paid employees to put on such a task, for such a recurring cost we have much too little sponsorship money and also paid memberships. So the board of EEF can discuss this and we can state that we would like such volunteers to exist but I fear it doesn’t get them into existence.

What we are doing already and specifically I myself is having open ears to the people in the community asking for some funding to get an important package into a maintainable state again. So the impulse needs to come from the community, then we can point our limited funding towards it. But in general it seems we lack on people wanting to work on the side on projects like this.

Having said that: no amount of this would have kept the internal Heroku stuff alive because no-one outside of Heroic cared about it anyway and Heroic itself didn’t care either. Also they never funded EEF. What they should have done is not write the use-less blog post but I guess someone wanted some final attention of our community, while they have not been part of the community since Fred left there.

4 Likes
20

Sure. But I think that when it comes to find volunteers for such a task, I think it is important to specify clearly what is expected. As you said, this can’t be a paid position, so a prospective volunteer would have to do it alongside whatever pays his bills. So in order to decide if he wants to shoulder the task or not, he needs to have a rough idea regarding how much time and resources he would have to invest, what skills would be required, what connections would be helpful, etc.

IMO, what the EEF should do is decide if they want a this “job” to exist, and if yes, think about what kind of tasks and responsibilities it would involve. Think of it as a job advertisement. Finding somebody to actually fill this position is subsequent.

This is all well as far as it goes, but funding is not always (in my experience, even rarely) the issue. The circumstances of peoples (maintainers) lifes change, they take new jobs involving different technology, become parents and have other priorities, get new and lose old interests, etc. Funding doesn’t solve any of those things and doesn’t keep a project from becoming gradually neglected and abandoned. It may help to revive an already dead project, but few people want to do that: making new things is always more fun than worming through other people’s old code.

What I think this “job” should be about is getting and keeping in touch with maintainers of existing projects while they are still healthy, detect early when they are about to fall into neglect for whatever reason (ie, before they need revival), at least assess together what problems they face and what they need to keep going. In essence, keep up community spirit.

Why is this a task for the EEF? IMO, the EEF is closest to being a central authority or “hub” in the Erlang world. It’s not all about funding, but about keeping up the community and supporting individuals making up that community.

Hm. I do agree on the first part about Heroku getting out of Erlang anyway, no matter what we did now or would/could have done earlier. It may have been for the reason they name in their blog post, or it may have just been a made-up reason to justify an already preconceived decision. However that may be for Heroku, I think it still highlights a real problem we have in general :man_shrugging:

3 Likes