Can you share any cool Erlang stories for evangelism purposes?

Chat / Discussions
,
1

I’m doing an intro talk about Erlang at work - just like an overview of what Erlang is, and what sets it apart from other languages. The audience is other consultants who mostly know more “traditional” languages. I’m cribbing a bunch from Joe’s excellent Stanford Seminar, but I would like to put some numbers to things. Joe mentions that an Erlang process is about 350 bytes at about 1:10:20, but I’m getting lost trying to find out what the size of a Posix thread is.

If anyone knows of any sexy resources; graphs, animations and the like, I would very much appreciate that as well. I remember listening to some talk on Youtube where someone was talking about how when the occurrence of a fault falls below a certain threshold and it can be “solved” by just restarting the process, that they don’t even attempt to figure out what the problem is. Tidbits like that make for great storytelling.

Also if anyone has any stories about live code upgrades. It’s a cool feature, but only once have I heard of anyone actually doing it.

3 Likes
2

This one is a very good example of showing what sets the ecosystem apart from other languages. It likely has a lot of useful insights to pull out as well.

8 Likes
3

My first Erlang application was deployed and working and the next day I discovered the logs showed that every process was crashing! However it wash crashing during the end of processing so supervision took over and things just kept working and no one noticed anything wrong!

We use live code upgrade all the time as we test new code. Restarting the whole application, and setting up all the use case conditions just takes too long. Updating code in runtime gives incremental changes high velocity.

6 Likes
4

This one, exactly! I was about to recommend precisely that talk, too.

1 Like
5

Also if anyone has any stories about live code upgrades. It’s a cool feature, but only once have I heard of anyone actually doing it.

We do hot-code upgrades every day at Klarna. It works really well, but you do need to be careful so that the code is upgrade-safe (like removing functions which existing processes might call).

I managed to bring down the single node where all the transactions goes through by calling supervisor:which_supervisor(foo) where foo was not a supervisor, but a regular gen_server. foo did not have a clause to handle the which_supervisor call, so foo died with a function_clause error, and since the application was permanent, the crash propagated all the way up and eventually took the entire node down for a restart. Fortunately, the system has enough redundancy that it could do a failover and recover all on its own, but there were lots of alarms which rang that day. :slight_smile:

Richard Carlsson held a talk at CodeBeam SF 2019 about Kred and how we do live upgrades:

2 Likes
6

The question “What is the size of a POSIX thread” has no answer.
(By the way, Joe was talking about 32-bit systems. A hibernating
Erlang thread will be somewhat bigger on a 64-bit machine.)
There are several reasons that POSIX threads have no definite size.
(1) When you create a POSIX thread, you can control the size of the
stack. There is a default size.
According to ‘man pthread_create’ on Ubuntu 22.04, the default size
ranges from 2MB to 32MB.
The default size in Mac OS X was 0.5 MB and the minimum was 16 kB. I
have no idea what the numbers are now.
Now Erlang processes can be small because their stacks can grow as
needed, just like the hardware-managed stacks on
the computer I used as an undergraduate back in the 1970s. But modern
POSIX systems cannot do that. The stak they
are given at startup is all the stack they will ever have.
(2) the amount of “meta-data” about a thread held in some control
block somewhere varies from implementation to
implementation of the POSIX thread interface, and indeed may vary from
release to release. It was 1 kB in Mac OS X.
(3) Whether there are “red pages” above and below a stack and how big
they are also varies.

3 Likes
7

here is says 338 words (including heap)

1 Like
8

I’m running RabbitMQ in production for hundreds days without any downtime.
Last year the only restart I did was when changing certificates that was expiring.
Really amazing.

2 Likes
9

Eugh, please do not focus on this in a talk unless you plan to convince to the audience this is somehow different and ‘better’ than machinery that has also existed elsewhere for decades such as co-routines, ‘green’ threads, etc. Spoiler, it does not…

I love hearing about what makes people excited about a language they have discovered, but there is a lot of churn as things come into and fall out of fashion.

What I find interesting to separate the chafe is “what super powers do you gain?”

By this I mean mostly what in this language do you find easier to express that was difficult in other languages you have used.

You may notice, ‘350 bytes per process’ does nothing to actually help people communicate their ideas into code. Message passing similarly, other languages already have this and a wildly known example is channels in Go. What Erlang does have going for it though is an aggressive “in your face” stance of “state machines talking to one another”.

Only last week I was at at conference and had a conversation with someone filled up with beans on Flutter. The only thing they could describe as ‘life changing’ was named arguments to functions. This definitely has its advantages and to someone who has not seen or had access to this it must be great, it is though not unique to the language or reason alone to learn it. In practice, many languages do not use this as their design decision was to instead pass an ‘object’ or in the case of Erlang a record or map.

When ‘lobbying’ you really need to lean on uniqueness and things that people may not even be aware of that are problems. @vances is 100% on the mark with his story about supervisors and similarly my story is live tracing, hot patching, and distribution RPC direct from my workstation into production…these are things difficult in other languages.

Personally I think having a wide base of different languages under your wings is useful as it helps your brain discover several ways to solve something and then pick the most appropriate way and tooling for it. Lua, kdb+/q, Perl, Python, C, Shell, … as well as my former life as a {sys,network}admin have all shaped how I perceive problems.

Erlang too has enriched me and it is your job to tell others how it will enrich theirs! Do not let us down…no pressure… :slight_smile:

4 Likes
10

Somewhat related, have you seen this blog post about doing hot code updates under Common test?

2 Likes
11

Ah yes, totally fair! But don’t worry, I will try not to be one of those people. I just want to have actual facts in my back pocket, in case someone asks. And I guess evangelism isn’t quite the right word, it’s more like here’s a rundown of the least bad programming language (that you probably haven’t even heard of) I’ve used in my career. My current plan is to come at it from the angle Joe was, in the talk I originally linked - what is the problem it’s trying to solve.

3 Likes
12

My personal favourite is My favorite Erlang Container[1] which is a wonderful love letter to the tire fire that is now considered doctrine…

[1] here is an IA link as ferd’s site died in the past day or two

2 Likes
13

At my previous company, I introduced a few Erlang components to support the PHP monster that was their flagship product. The first of those I did in 2013 IIRC, my first Erlang project that made it to production ever. It has been running for years straight, ie it never went down unless you wanted it to go down for things like planned downtimes. I hear that stuff is still running to this very day. Mind you, not because I’m a brilliant developer, but because Erlang is a brilliant language/system :smiling_face_with_three_hearts:

3 Likes
14

The stories @vances and @juhlig told highlight what I think is one of the main reasons that there are only few full-time Erlang jobs out there, at least outside of companies where it is the language of the flagship product, ie only in a supplementary role.

An Erlang component in such a system, even if somewhat buggy, even on flaky hardware, doesn’t require much attention or maintenance. While all the other stuff around it keeps crashing on any tiny bug, or in the face of adverse conditions, or just because the moon is not right, ie needs constant pampering, the Erlang stuff just keeps chugging along, day after day for years on end. Taken for granted, easily forgotten that it even exists.

But what company pays full-time for a developer who does nothing but bathe in the glory of nigh-indestructable software all day? :sweat_smile: So Erlang is likely only a part-time pastime, the major part of the job is done in other languages.

(Disclaimer: At least this is the case at the company (-ies) I work for. The language stack is, well, just short of this. And everything crashes. All the time. Except for what is written in Erlang :grin:)

(And yes, I’m aware that “only few full-time jobs available” is not supposed to be on the top of the list that you bring to the table when evangelizing for Erlang. Or anything else for that matter :smile_cat:)

2 Likes
15

My experience entirely. I spent the vast majority of my days doing PHP stuff :cry:

Now that I’m gone, after one of the two guys which I gave a crash course in Erlang also left soon after, and after being swallowed in a recent merger, they are considering replacing the Erlang components with a reimplementation in Go or PHP+Swoole :roll_eyes: I question the wisdom of replacing the proverbial sturdy steel bolts with frail wooden pegs for the sole reason that they know how to repair wooden pegs while not knowing anything about steel bolts which don’t need any repairing, vs just learning about steel. But this is not my problem any longer :man_shrugging:

1 Like
16

I disagree, they already do. Have you seen how much it costs to run k8s? :slight_smile:

The industry now perceives this type of reliability as a credit card scaling problem solved through outsourcing to a fleet of backends, a load balancer and a health check. The skill sets to maintain this are substantially different and tend to be a lot more accessible. This is not a statement of whether it is technically “better” or “worse” but as it solves 95%+ of those problems, why use Erlang?

By example, back in the day you had to use kdb+/q to solve some problems, as disks were spinning blades of rust and resources were tight. Now a days, you can solve 95%+ of those problems by cheaply scaling both horizontally and vertically, run MariaDB/MongoDB/whatever and call it a day.

kdb+/q devs, like their Erlang counterparts, do not come cheap. For many other languages, they come “a dime a dozen”.

It is easy to attribute “low demand” to the affects of “low maintenance” but really I think we all like to say that out loud as it makes us feel better.

Really, the problem is we are all just too bloody expensive and hard to source.

Are you expensive only because you know Erlang, or because of all the other stuff you know and can do?

We all help our clients develop solutions that are a good fit for them, and selecting Erlang would harm them more than it helps them for those all the non-technical reasons.

This is only as we were too lazy to do that monthly one page ‘tl;dr’ executive summary to remind the company why the services we were responsible for are important (“…this is why the CTO gets to work on their golf swing”).

Well, good to know I am not the only in the ‘acceptance’ stage :slight_smile:

How to solve the world domination problem?

Maybe the evangelism teams of those groups behind Python, cloud vendors and various proprietary software vendors are on to something when they focus on the people entering the market not the ones already opinionated in there.

17

I think we are disgressing a bit from the actual topic, but here goes :wink:

Just for the record, I tl;dr said that I think that “low maintenance” was one of the reasons for “low demand”. And admittely, yes, whether that is really true or not, it gives me a certain feeling of pride :smile_cat:

I’m a high-maintenance girl, I’m expensive by definition :stuck_out_tongue_winking_eye:

Seriously though, I think I’m reasonably expensive because I deliver solid work, and over time my employers realized that it is still cheaper to hire one comparatively expensive person to do a job properly and be done with it than hiring five cheap people to do it five flavors of wrong. And yes, I’m aware that I may be flattering myself here just a little bit :wink:

That being said… While OTOH I agree with most of what you said, OTOH I for myself can’t just let it stand at this. Like, when you say…

… Have you seen how much it costs to run k8s?

… or…

Now a days, you can solve 95%+ of those problems by cheaply scaling both horizontally and vertically, run MariaDB/MongoDB/whatever and call it a day

… IMO that is the same argument as “… have you seen what shirts mass-produced in Bangladesh cost?” or “… you could buy and then throw away 10 or more of those for the price you’d pay for a single quality-made one” when discussing the price and quality of clothing: While the argument by itself is valid, personally I don’t want to encourage that way of doing things :frowning_woman:

Hold my beer coffee :sunglasses: I do accept this as the current, but not the final, state of affairs. I’m just 31, so 30+ good years to go and make a change :tipping_hand_woman:

1 Like
18

I didn’t dare say it out loud, but since you said it yourself: I agree :grin:

2 Likes
19

Why you…! Don’t agree :stuck_out_tongue_closed_eyes:

1 Like
20

This is a nice recap of the WhatsApp success story with Erlang:

Was also reviewed here:

2 Likes