EEP 70: Non-filtering generators

[josevalim]

There has been some discussion about this feature in a separate thread, with some rationale from the Erlang/OTP team. I am moving them back to this thread because I am mostly interested on the discussion around this feature.

Pop quiz! Without checking with the shell, what do these expressions yield?

[X || <<1, X>> <= <<0, 0, 1, 4, 0, 0, 1, 7>>].

…

In three of these cases we get a surprising result that is overwhelmingly likely to be incorrect, but is nevertheless likely to silently “work,” discarding data.

…

EEPs 70 and 73 address footguns in the design of comprehensions, and we want them to supplant rather than extend their respective corners.

There are two arguments here:

1. Binary comprehensions are confusing and prone to mistakes (which I agree and I will add that using <= in general for comprehensions is confusing)

2. The footguns in comprehensions are meant to supplant rather than extend

To the second point, @bjorng said “there are 105 strict generator operators and 73 relaxed generator operators”. That tells me that, what we will see in practice, is that at least three operators will be used: <-, <:-, and <:=. So if the goal is to supplant, I don’t believe the current proposal has enough tools to make that a reality (please correct me if I am wrong). To do so, we need to either add a “strict” match to comprehensions or a “relaxed” match. I will outline both options below.

Option 1: Add strict operators and relaxed match

This option suggests adding <:- and <:=, as in EEP 70, and then adding ?= (which is used in maybe) for relaxed matching in comprehensions. This way, we can argue that all generators should be strict by default, <- and <= can be deprecated in long term, and effectively supplanted. For example, if I have a list with ok/error tuples and I want to filter the ok ones, I could write:

Oks = [Value || Pair <:- List, {ok, Value} ?= Pair].

But if I am expecting all of them to be ok, I would write:

Oks = [Value || {ok, Value} <:- List].

Option 2: Add a new operator for binary generators and a strict match

This option suggests adding a new operator for binary comprehensions only (either <:- or something else), which is relaxed but raises on trailing entries. And then allowing = in comprehensions. This way, <= can be supplanted.

If I have a list with ok/error tuples and I want to filter the ok ones, I could write (as of today):

Oks = [Value || {ok, Value} <- List].

But if I am expecting all of them to be ok, I would write:

Oks = [Value || Pair <- List, {ok, Value} = Pair].

Summary

While I would prefer Option 2, because that would require fewer changes, I also understand the goal of making generators strict by default, so I am perfect fine with any of the options above. My main goal is to point out we might not have enough to replace the existing generators. If that’s one of the rationales behind accepting this EEP, then I’d suggest to:

1. Update the EEP to mention the goal of replacement and include examples of how existing relaxed generators can be written from strict ones

2. Include a rough estimate of when the current generators will be deprecated (we can keep them around forever but we could emit a warning - which one could turn off if desired).

Thank you for your time.