Some Thoughts Regarding Recent EEPs - should Erlang be kept minimal/small?

[schnef]

Perhaps entirely unfairly, I feel that lately more attention has been paid to enhancements to core Erlang than before. The feeling creeps up on me, perhaps completely unfairly, that for people who work on core Erlang or those with a strong interest in computer languages, it is more challenging to work on new features.

Erlang has always been a small language that could be fully explained in a small number of chapters. Look at the Erlang documentation, books like Programming Erlang by Cesarini, and Thompson, Joe’s Programming Erlang or Erlang in Action by Logan er al, and the entire core Erlang language can be covered in detail in just a few pages. Compare this to Elixir, Hasskell, Java, C++ etc, where it is much harder to (really) master the language.

I have always seen Erlang as a very effective tool to build things with. It is so far the most effective tool I have ever used. And like other tools, you have to learn to use it and deal with any minor limitations. Erlang may have some minor limitations or quirks such as no pin operator, zippers in generators or strict comprehensions etc, but it has never bothered me in the engineering tasks I perform.

It is quite often said that Erlang is quite a difficult language for newcomers to learn, partly because of its different syntax. Many newcomers will (have to) learn Erlang on-the-job for maintaining existing code bases. As an engineering tool, core Erlang should instead remain consistent over long periods of time and syntactically simple because it is the underlying concepts, such as concurrency, that demand vole attention.
Obviously I do not have a computer science background but rather an (electtical) engineering background. I want tools like a hammer and chisel that I can work with, even if you sometimes hit your thumb with that not quite perfect hammer.

So I advocate modifying core Erlang as little as possible and being particularly picky with modifications. Maps were a good idea and the performance improvements totally super. There is plenty to do in terms of OTP it seems to me (gen_statem is awesome).
(I am not a native English speaker and hope I can still convey my ideas in a constructive way. Erlang is very close to my heart and I appreciate the efforts of the community and the Erlang team).

[rvirding]

I totally agree. I am a strong KICASS person. (Keep It Consistent And Simple Stupid)

[maxlapshin]

Yes. It is very easy to add one syntax sugar item per month and get a complete mess in 2 years.

[AstonJ]

I’ve seen similar arguments against Ruby, where you often find more than one way of doing the same thing or tools to simplify using the language. However a lot of people (including me) feel this helps make the language more intuitive and natural - there’s more than one way to say something so why can’t there be more than one way to express something in a programming language? Sometimes the context may require one over the other or sometimes even one way might come more naturally than another to some people. Similarly, if you can make something easier with syntactic sugar or other constructs then there is a decent argument for it as it can make the language more enjoyable (/less mundane) to use and therefore arguably more accessible and attractive.

Having said that there is no right or wrong answer as everyone will feel differently about it and there are languages that fall into one category or the other - so it essentially boils down to what the core team feels is the right direction for their language after considering of course what they feel might make the language better, what the community might like, what might help with adoption, what might help make using the language more effortless, etc - ultimately what they feel would be best for the language with their goals in mind.

While I personally feel making a language easier to use and more intuitive and natural is a big advantage I support the Erlang Core Team in whatever they decide is best

[igorclark]

Strong agree especially re simple and general tools. Erlang’s simplicity is its strength and a lot of power comes from the standard lib and OTP. The simpler the language stays the less confusing and the stronger it stays as a foundation for the rest.

[kuna.prime]

I have found that although Erlang for newcomers usually seems very alien, it is very fast to learn (in order of couple of weeks). That is great thing, as core language is cognitive addition because of it’s unique design and concepts (cheap processes, messaging, parse transforms … ). In contrast learning C# when you know Java is of very low cognitive benefit as main differences are expressed in syntax, and not in concepts (of course i’m omitting lot of detail here just to convey a point).

For standard library and OTP-design it takes more time to learn and understand, but library is comprehensive and well designed so for ones that intend to stick with the language/technology it is worth it and it is beneficial for this part to be vast.

My view is that as community we should be very careful and conservative when adding to core language and more liberal when it comes to libraries and tools.

Also maybe there should be guidelines how do we consider modifications to core language, so i would suggest several questions/criteria:

• if we are adding are we adding cognitive value
• if we are modifying are we achieving more symmetry
• how much of use cases/languages feature touches (measure of impact)
• can it be achieved with library addition

notes on symmetry: symmetry is reducing cognitive load and enables better composition. Basically it reduces number of surprises (i don’t like to be surprised)

notes on impact: is it really worth adding something to the language that only have narrow impact? Narrow impact introduces edge cases, which in turn increases cognitive load but not value.

I believe that both EEP-70, EEP-73 fail on all accounts if evaluated on above criteria.

• there is no new cognitive value
• what symmetry are we achieving? we don’t have non skipping receive can we also zip receive blocks?
• this feature is only limited to comprehensions
• both are achievable with qlc and other facilities already present

Both proposals have merits and there is opportunity there to make language and/or libraries more symmetrical: maybe introduce binary:zipwith/binary:map, and maps:zipwith. Or introduce Collection:/table function to make qlc generator? Or we can have have discussion for iterators for all collections and lazy iterations for all of them.

[scherrey]

@schnef and @kuna.prime make compelling and wise arguments. This is my strong voice of support for the priorities they lay out in terms of guiding when and how the language should be extended. The criteria @kuna.prime lays out I would suggest should, at least, be a first burden to overcome before even considering a language extension.

That said - and here I go offering the possibility of a language extension - it seems to me that a macro capability akin to what Elixir has would go a long way towards allowing people to have their own local extensions and could allow them to be tried out for a good period of time if there were ever a just cause to extend them into official language standards.Giving Erlang homoiconic properties would make it extremely attractive to other language designers and go a long way towards making interoperability between BEAM languages potentially simpler and more expressive. EEP-70 and EEP-73 both seem like good candidates for being addressed as such and would give them an opportunity to demonstrate wide spread adoption (or otherwise) without impacting other Erlang dev orgs. Of course @kuna.prime’s alternatives beyond the EEP proposals are promising options as well.

Just an idea. The main point being adding new keywords or symbols (even in limited scope) to the core language should be a Really Big Deal. I don’t think these two items, as interesting as they are personally to me, have met the bar necessary to be declared as such.

[bjorng]

I’m probably not going to convince anyone, but here are some of my thoughts.

As I wrote in another thread, when deciding to accept a new feature, we weigh the pros against the cons. That is obviously subjective.

The two EEPs in question enhance a single language feature: comprehensions. I know of professional Erlang programmers that never use comprehensions; they find the basic language and/or the library functions in STDLIB sufficient. They can continue not to use comprehensions.

Fans of comprehensions can continue to use what they know or adopt the new features.

Newcomers to the language should not learn comprehensions the first thing they do. They should probably master most of the other language features before learning comprehensions.

This is all to say that I don’t find the negative consequences very serious. Your milage may vary.

As I wrote in another thread, the strict generators can make the intention clear whether all elements in a generator must match the pattern or that they can be other elements that will be filtered out. There are other ways to express that intention, but they require more code and/or have a performance penalty. Because there is more work to use them, those alternatively ways to protect against unexpected elements will in practice not be used, and the resulting bugs are likely to be detected much later.

I consider features that help me to find bugs quicker a big deal.

Zip generators will allow me to write natural, concise code that will not build an extra temporary list. That is important in at least the compiler and Dialyzer. As far as I know, no one has ever complained that the compiler or Dialyzer are too fast. It has been pointed out to us that the compiler has became slower in recent releases. I have refrained from using lists:zip/2 in time-critical parts of the compiler to keep the compiler as fast as it can be.

Therefore, I also consider zip generators a big deal.

[schnef]

I, and allegedly most programmers, spend most of my time maintaining code that is mostly written by others. This is about understanding what the code does rather than how the language works. The more syntax a language contains, the more ways to package the same problem. I personally prefer to see some more code that I can read without much effort than fancy language constructs that (theoretically) describe the problem better.

The problem with advanced macros is that while the language may not become more complex, the code I have to read does. Macros and parse transforms are usually even trickier than new syntax because they are not documented and you have to figure out their source code first to understand them.

I have the impression, perhaps unfairly, that many of the proposed extensions are more or less domain specific. I think this was also the case with string sigils, for example. I myself have never had any need for them despite the rather wide range of topics I have worked on.
I have always wanted to learn and be able to use Haskell but every time I encountered real world code the fun was over. And, I hardly dare admit it, when I have to write front-end code, I secretly use Elm: boring, repetitive, but oh so much more productive and easier than anything else. (Who knows someday Gleam / Purerl?)

[dischoen]

Hi,

I would support a conservative approach to new features. I think all has already been said about that.
On supporting more or new macro features, I would strongly oppose against that:
In the current state, you exactly know what a piece of code in erlang does and where the function calls are defined (as long as you do not use -import).
This feature really breaks my performance when I use other languages which heavily rely on macros/pragmas. In the worst case you have to consider the whole toolchain to reason where a piece of code really is defined.

[kuna.prime]

i’m not that familiar with Elixir’s macros, but i would argue that parse transforms are extremely powerful so i don’t see any place for macros other than building on parse transforms usability more.

As @schnef said macros and parse transforms really hider understanding what is going on in code however the same argument could be said for BIFs and NIFs. Here we should be careful about what we expect of users of our code, parse transforms are amazing tools for library authors but they should be used with care in terms of “surprises”. If API is logical at least within it’s targeted domain then i don’t see any problems with using such features. It is unfortunately common (in all languages) that advanced and powerful features are abused and then wrapped in poorly designed interfaces which then confuses users.

Here i must commend Erlang Team for producing very fluid and logical stdlib so far, which i believe is part of success of the language. (qlc as mentioned before is a good example)

[jhogberg]

Pop quiz! Without checking with the shell, what do these expressions yield?

[X || <<1, X>> <= <<0, 0, 1, 4, 0, 0, 1, 7>>].

[X || <<X/float>> <= <<0.0/float, 2.0/float, 3.0/float, -1:64, 4.0/float>>, (trunc(X) rem 2) =:= 0]

[X || <<X, "åäö"/utf8>> <= <<0, "åäö"/utf8, 1, "åäö", 2, "åäö"/utf8, 3, "åäö">>]

[Y || <<X, Y:X/bits>> <= <<8, 1, 167, 1, 3, "abc">>]

In three of these cases we get a surprising result that is overwhelmingly likely to be incorrect, but is nevertheless likely to silently “work,” discarding data. This is very much against the spirit of the language: as Erlang programmers, we write our code offensively under the assumption that it’s not worth continuing once we encounter something unexpected, leaving error handling to the layers above. Limping onward with bogus data is anathema to us, but here we have a language feature that puts it front and center.

Comprehensions are under-documented and full of surprising corner cases, and we have long thought them to be lacking but have not found a good way out. EEP 70 provided the missing piece of the puzzle, which prompted us to update EEP 19 in EEP 73. These changes have been 16 years in the making and if that doesn’t sound like a “Really Big Deal” then I don’t know what to say anymore.

We are by and large the same people who have worked on Erlang since the 90s. I’ve been working on Erlang for a mere seven years which makes me one of the new kids. The average tenure is about 20 years, and the person who has been championing this has worked here for much longer than that. We are very, very hesitant to change the language and only do so after great deliberation. To suggest that we’re eager to extend the language for personal reasons (“challenging”) is not just unfair, it’s dismissive of the 16 years we’ve been mulling this.

EEPs 70 and 73 address footguns in the design of comprehensions, and we want them to supplant rather than extend their respective corners. Furthermore, as the problems are in a language feature, we cannot fix them by changing the standard library.

The vast majority of times that <- is used, <:- is the actual intent, and I cannot think of a single time that <= was not intended as <:=. Similarly, cartesian generators are extremely rare and newcomers often wonder what the heck we were smoking when they were added; everyone assumes that they would zip.

Once the documentation and literature has been updated to prefer <:- and friends, this will be no different from is_atom/1 supplanting atom/1. The language will become simpler in practice for those starting to learn it, not more complex. EEP 73 hasn’t been accepted yet either, and we are considering further simplifications of it.

[hanssv]

This is all great insight, and a couple of arguments for the flip side of adding syntax. I wonder if perhaps there is a communication problem then? My main gripe in EEP70 was the lack of positive support, no one could be arsed to write a single supportive thing (apart from the EEP author, who unsurprisingly was in favor) - apparently such opinions exist and there are some (semi-)valid arguments for why adding crashing generators to the language. At least this was very non-visible to me.

@jhogberg

The vast majority of times that <- is used, <:- is the actual intent, and I cannot think of a single time that <= was not intended as <:=

That is, for sure, not true in the code bases I’ve worked with, admittedly it is very hard to quantify these things, but I can’t really think of many times when I’d whished for a generator to crash. It happens that you change a data-structure and accidentally filter out everything, but you normally notice quickly regardless

[schnef]

I started this topic not because of just EEP70 and 73 but more in response to some (proposed) modifications to Erlang in recent months, maybe years. I also tried to be careful in my statement because I may be completely wrong. I sincerely hope no one takes my statement personally.

And yes, I did indeed have to look very carefully at the various comprehensions in the quiz to get them right. The point is, that precisely in this kind of complex situation it is better not to use comprehensions but rather to use lists functions to write out explicitly what is meant. Precisely this kind of code is difficult to maintain because the complexity of the syntax now overshadows the complexity of the problem to be solved. Whether it says ← or <:- no longer matters.

[bjorng]

@hanssv

admittedly it is very hard to quantify these things

I’ve counted the number of strict and relaxed generator operators for this branch that modifies most of the modules in the compiler application:

GitHub - bjorng/otp at bjorn/use-strict-generators

There are 105 strict generator operators and 73 relaxed generator operators. (Counting only generators with patterns that can fail.)

There is exactly one strict binary generator and no relaxed binary generators.

(This is 10 more strict operators that I found while counting.)

[Maria-12648430]

I’m pretty sure that this was not meant to be the way it ultimately came across. As @hanssv suspects, there is likely to be a communication problem here.

I guess nobody outside of OTP knew this was being thought of, certainly not for 16 years, so nobody tried to be dismissive. You OTOH know, so it is totally understandable that it looks like that to you. But I don’t think it was intended.

(Adding to that, many older EEPs look totally abandoned, left to gather dust. Some are obsoleted by others even. Nobody knows if they are still considered or not. Maybe some cleaning up is in order?)

Regarding supplanting vs extending, well. I looked, and in the thread announcing EEP 70 here on the forum it was in fact worded as an extension to the language

The way the overall process went here, especially in the case of EEP 70, was not at all ideal as well: The idea was proposed, the feedback was generally critical (or so it feels to me). Then it was decided to implement it, to the surprise of everybody who voiced concerns. Turns out there were good reasons for that, but they only surfaced later, in threads like this one.

Like, from the outside, it looked like bloating the language for little obvious benefit. On the inside, the decision to implement it was made for the benefits you guys saw in it, accepting language bloat as the price to pay for it.

---

So… where does this leave us?

First, lets bury the proverbial hatchet, shall we?

Next… Well, I’m not saying that you guys from OTP should tell everybody everything all the time what goes on in your heads or team. In fact, I’m happy to think that you guys know pretty well what you’re doing
OTOH, not being in the picture easily gives raise to misunderstandings and talking at cross purposes, like in this case So, please, explain, and early, don’t wait for the storm to break

And on the other side… let’s give the guys from OTP some slack and put some trust in them

So, I hope we can get along better in the future

[hanssv]

@bjorng

There are 105 strict generator operators and 73 relaxed generator operators. (Counting only generators with patterns that can fail.)

I’d perhaps not call that a vast majority. Also changes in cases like

    Fnames1 = lists:keysort(2, maps:to_list(Fnames0)),
    Fnames = [Name || {Name,_} <:- Fnames1],

is not a huge improvement. But again - it is really nice to (finally) see some arguments why this isn’t just an ad-hoc extension!

[rvirding]

As you have probably guessed I am definitely on the side that Erlang should be kept minimal and small. There will ALWAYS be requests for new features that some feel are really important. As for the comprehensions, they are what they are, just get over it. If/when you need more complex handling then just write a recursive function which does exactly what you want.

[dszoboszlay]

I understand the desire to keep Erlang a simple language, but I believe focusing on the “language” part without the wilder ecosystem is missing a point. It’s true that Erlang, the language is small, especially compared to monstrosities like C++, and it makes learning it easier. However, a newcomer, especially one getting onboarded to an existing project, will not only have to learn the language, but also all the libraries and tooling around it, plus the actual project built on top of these components. Looking at the complexity or simplicity of the language alone is pointless, and I believe the “keep the language stable, develop the libraries” philosophy is missing the point.

I would even argue that in many cases bringing a feature from a third party library to the standard OTP libraries, or even from a library to the language can bring the net complexity of the ecosystem down. Just think about Dialyzer: because static type checking is not part of the Erlang language, we have Dialyzer as a library. But Dialyzer is quite opinionated about what kind of things it wants to check, and not everybody is happy with these choices. So we now also have Gradualizer and Eqwalizer and many more. And these tools all build on the type specifications that are part of the Erlang language, but neither of them has a perfect match with the language part (sometimes Erlang type specifications are too generic, other times are not expressive enough for a particular tool). So while the type language in Erlang remains simple, the entire problem space of static type checking in Erlang is way more complex than in other BEAM languages that decided to add static types to the language.

That said, I’d still want to avoid adding bloat to either the Erlang language or to the OTP libraries. But my criteria are very different from the points suggested above (cognitive value, symmetry, measure of impact, could be a library or not). For example if adding cognitive value (as defined here) would be desirable, we should strive for adding at least monads to Erlang, if not some real bleeding edge concept from the frontiers of functional programming theory. But please don’t, I prefer Erlang being a pragmatic language for engineers!

The new features and changes I’d like to see in Erlang and OTP should instead support goals such as:

• Writing maintainable code.
• Eliminating possible bugs (including various foot guns and surprising edge cases).
• Increased productivity.
• Keeping up with the changing world.

Based on these measures I find for example:

• The pinning operator a good idea, since it could have eliminated an entire class of bugs.
• Allowing underscores in numbers a good idea, since it can lead to more readable code.
• Sigils a mixed bag, to be honest. They give a more convenient syntax than binary strings and verbatim string values can be more readable too. But my feeling is that triple quoted strings for example offer too much flexibility that can lead to confusion.
• Adding JSON support to OTP is a nice example of keeping up with the changing world.
• Significant children for supervisors is missing the mark in my opinion. I understand the need for more supervision strategies than the four built-in, but I’m not convinced significant children are the best way deliver those.
• Also a bit divided on the maybe expression. Cutting nested cases and focusing on the happy path gives more readable and maintainable code, but I’m worried about unexpected right hand side values and else clauses becoming the perfect foot gun.

Anyway, sorry for the long aside! I think I basically want to say that we should indeed be careful about what we add to the language, but also to the broader ecosystem, because they are tied together. And that I value maintainability and correctness more than simplicity, which means I’d rather go and try to fix foot guns in existing language elements than to say “you should know not to use this feature beyond this point”, because people in practice won’t know that.

[Maria-12648430]

@dszoboszlay

Significant children for supervisors is missing the mark in my opinion.

Because you are aiming at the wrong target

@dszoboszlay

I understand the need for more supervision strategies than the four built-in, but I’m not convinced significant children are the best way deliver those.

They don’t, and they are not meant to. They address an entirely different topic, please read the respective EEP (I don’t know about the impression the documentation creates on the topic TBH. I had a pretty hard time fitting this into the existing docs back then )