Support adding multiple elements in list comprehensions

Questions / Help
, ,
1

It is possible to prepend multiple elements to a list:

1> [a, b | []].
[a,b]

But it is not possible to prepend multiple elements to a list using list comprehensions:

2> [K, V || K := V <- #{a => b}].
* 1:7: syntax error before: '||'
3> lists:flatten([[K, V] || K := V <- #{a => b}]).
[a,b]

Is there any good reason for this not being possible? It would be neat and more efficient to be able to append multiple elements in a list comprehension in only one pass :grin:

1 Like
2

Not an answer to your question, per se, since it only covers the efficiency side but:

lists:append([[K, V] || K := V <- #{a => b}]).

would be more efficient. Else, a hand-rolled fold if you can stomach that.

3

In that particular example - you can use second generator:

3> [ X || K := V <- #{a => b}, X <- [K, V] ].
[a,b]
8 Likes
4

Just to complement the solutions above…
All of them produce the same result:

1> lists:flatten([[K, V] || K := V <- #{a => b, c => d, e => f, g => h, i => j, l => m}]).
[c,d,i,j,a,b,l,m,e,f,g,h]

2> lists:append([[K, V] || K := V <- #{a => b, c => d, e => f, g => h, i => j, l => m}]).
[c,d,i,j,a,b,l,m,e,f,g,h]

3> [ X || K := V <- #{a => b, c => d, e => f, g => h, i => j, l => m}, X <- [K, V] ].
[c,d,i,j,a,b,l,m,e,f,g,h]

But as expected, using list comprehension is much faster:

./erlperf \
'lists:flatten([[K, V] || K := V <- #{a => b, c => d, e => f, g => h, i => j, l => m}]).' \
'lists:append([[K, V] || K := V <- #{a => b, c => d, e => f, g => h, i => j, l => m}]).' \
'[ X || K := V <- #{a => b, c => d, e => f, g => h, i => j, l => m}, X <- [K, V] ].'
Code                                                                                            ||        QPS       Time   Rel
[ X || K := V <- #{a => b, c => d, e => f, g => h, i => j, l => m}, X <- [K, V] ].               1    9910 Ki     100 ns  100%
lists:append([[K, V] || K := V <- #{a => b, c => d, e => f, g => h, i => j, l => m}]).           1    3501 Ki     285 ns   35%
lists:flatten([[K, V] || K := V <- #{a => b, c => d, e => f, g => h, i => j, l => m}]).          1    3010 Ki     332 ns   30%

Anyway, I don’t know if there is any limitation, but IMO support for adding multiple elements should be a nice addition to list comprehension.

2 Likes
5

I was curious if a recursive function would be faster, but apparently not:

$ erlperf \
      '[ X || K := V <- #{a => b, c => d, e => f, g => h, i => j, l => m}, X <- [K, V] ].' \
      '(fun(M) -> (fun L(none, A) -> A; L({K,V,I}, A) -> L(maps:next(I), [K,V|A]) end)(maps:next(maps:iterator(M)), []) end)(#{a => b, c => d, e => f, g => h, i => j, l => m}).'
Code                                              ||        QPS       Time   Rel
[ X || K := V <- #{a => b, c => d, e => f,         1   21138 Ki      47 ns  100%
(fun(M) -> (fun L(none, A) -> A; L({K,V,I}         1   15375 Ki      65 ns   73%

(I tried putting the same recursive function in a proper compiled module, but that didn’t make any difference).

6

Flatten would not be good here as it flattens out nested lists so if your K or V were lists they would be flattened as well.

5 Likes
7

It all depends on what you are willing to accept as a good reason.
List comprehensions in most functional programming languages (and in Python)
share this restriction. Once you start thinking of it as a restriction,
you wonder, “if one result per iteration, or two results per iteration, why
not (one or two results) per iteration?” Three languages I know of that do
allow multiple AND variable numbers of results per iteration are

  • F#, where [ for … do yield! xs ] adds the elements of xs to the list
    being built
  • Lisp, where (for … collect x) adds x to the list being built,
    (for … join xs) adds the elements of xs, collect and join may appear
    any number of times, and may be conditional, e.g.,
    (for x in l collect x unless (minusp x) collect (minus 0 x) when (minusp x))
    That’s Interlisp. Common Lisp uses (loop for …) instead of (for …),
    collect is collect, join is append, and conditionals are
    … when/unless P collect/join X … rather than the other way around, but
    it’s quite similar to Common Lisp
  • Pop-2, where [% %] gathers up whatever dropped onto the stack
    as a list, so [% forall n 1 1 3 do n, -n enddo %] => [1 -1 2 -2 3 -3].

So there is definitely prior art.

The problem is that the first generalisation (allowing any FIXED number of
results per iteration) fits into Erlang syntax well enough, the second
generalisation (allowing a VARIABLE number of results per iteration) doesn’t.
Both generalisations can be done using
lists:append([ whatever || … ])
where whatever evaluates to a list of result elements, and if this proved to
be so popular that the overhead of creating two list cells per final result
was worth bothering about, the compiler could recognise this construction and
do something special.
2 Likes
8

PS, I benchmarked the following four functions using erlperf,
with the argument being lists:seq(1, 10000).

f1(Xs) → [T || X ← Xs, T ← [X,X,X]].

f2(Xs) → lists:append([[X,X,X] || X ← Xs]).

f3([X|Xs]) → [X,X,X|f3(Xs)];
f3( ) → .

f4(Xs) → lists:reverse(f4(Xs, )).

f4([X|Xs], R) → f4(Xs, [X,X,X|R]);
f4( , R) → R.

Scaled so that the fastest is 1.00, we have
11.16 f1 nested list comprehension
11.44 f2 list comprehension + lists:append/1
1.00 f3 body recursion
4.72 f4 tail recursion + lists:reverse/1.

The difference between f1 and f2 is not worth worrying about.

4 Likes
9

I don’t know why I presumed using the second generator was slower, but I am glad to be shown the way with hard facts! :smiley:

2 Likes