Is partial function application in Erlang a good idea?

Gwaeron · November 5, 2021, 12:53pm

Hi all,
I’ve been thinking quite a lot about partial function application and the benefits it might bring to the language the last few years. I know others have had similar thoughts since I’ve found several different implementations or proofs of concept online, but I never really saw it discussed (I’m sure it has been, somewhere, somewhen).

I am often struck by how nice Haskell’s higher-order functions like map or fold can look with some partial function application magic. For those unfamiliar, what is written in Erlang as fun(X) -> X < 0 end would be written as (< 0) in Haskell, which simply returns a new “arity 1” function that expects the missing operand.

Erlang isn’t curried, like Haskell is. Furthermore, the arity is an integral part of the function signature, so defining an add(A, B) -> A + B. and then calling it as Increment = add(1) would make no sense. The best idea I’ve had is to use “argument placeholders” to preserve arity.

The add/2 example could be written as Increment = add(1, _) which would return an arity 1 fun such that 101 = Increment(100).

Why bother?

There are a few reasons I think it’d make a great addition to the language. First of all, it would reduce the verbosity of trivial lambda functions often encountered in lists:filter/2 and its higher-order function friends map, foldl, and so on. Compare the two ways of filtering all strings beginning with “Hello” from a list:

get_hellos(Greetings) ->
  lists:filter(fun(E) -> lists:prefix("Hello", E) end, Greetings).

get_hellos_pfa(Greetings) ->
  lists:filter(lists:prefix("Hello", _), Greetings).

It would also provide a convenient alternative syntax to fun something/1, as in:

["HELLO", "WORLD"] = lists:map(string:uppercase(_), ["hello", "world"]).

Lastly, there has been a lot of talk about bringing over the pipe operator from Elixir. I think it has two big flaws: It obfuscates which function is being called by making an arity N function look like an arity N-1 function. And it assumes the argument needs to be piped as the first argument (or last, if the <| reverse pipeline is added).

add(A, B, C) ->
  A + B + C.

add(A, B) ->
  A + B.

do_something() ->
  100 |> add(50, 2). % This looks like add/2, but it's add/3.

Allowing the piping operator |> to be used only with arity 1 funs as its second operand would eliminate both of these problems, and would even look quite elegant with partial function application. It would be possible to pipe the argument to any position, and it would be explicit which function/arity is being called.

do_something() ->
  20 |> add(10, _, 30). % Very clearly arity 3 and piping the second argument.

Useful?

I’d love to hear everyone’s thoughts. Does this make absolutely no sense? Did I miss something obvious that makes this unrealistic? Does anyone have any cooler or nicer ideas to share on the topic? Any other adjacent ideas? Has it already been discussed to death?

essen · November 5, 2021, 2:35pm

I think it would be good to have. Not vital, just nice. I wonder how it would look like in lists or binary comprehensions, which are likely more common than lists:filter.

As for the pipe operator, I think you have the right idea. We definitely shouldn’t add an operator that hides arguments.

elbrujohalcon · November 7, 2021, 12:39pm

Hernán Rivas Acosta wrote an article related to that a long time ago…

https://web.archive.org/web/20160623210114/http://inaka.net/blog/2014/10/14/metaprogramming-in-erlang-partial-application/

Gwaeron · November 7, 2021, 1:25pm

Thanks for the link, that’s definitely one approach, and arguably nicer than the one I tried to write myself a while back, which uses a parse transform (and that always feels a bit hacky to me). But also, I think my version produces more elegant code in the end!

github.com

gwaerondor/partial_application/blob/master/test/partial_application_tests.erl

-module(partial_application_tests).
-compile({parse_transform, partial}).
-include_lib("eunit/include/eunit.hrl").

core_functionality_test_() ->
    [fun arity_two_function_with_second_argument_missing/0,
     fun arity_two_function_with_first_argument_missing/0,
     fun arity_three_function_with_two_arguments_missing/0,
     fun wrapped_in_parentheses/0,
     fun apply_argument_to_partially_applied_function_in_parentheses/0,
     fun apply_argument_to_partially_applied_function/0,
     fun return_partially_applied_fun_from_code_block/0,
     fun apply_argument_to_nested_variable_function/0,
     fun apply_argument_to_partially_applied_fun_in_code_block/0,
     fun missing_argument_in_nested_function/0,
     fun missing_arguments_in_nesting_and_nested/0,
     fun partial_application_of_already_partially_applied_function/0
    ].

arity_two_function_with_second_argument_missing() ->

This file has been truncated. show original

cmo · November 7, 2021, 9:44pm

I would have to disagree with the idea that the pipe in elixir obfuscates which function is being called. F# does it too, but into the last position. However, there is Magritte which does it the way you suggest, which is certainly clearer and more flexible.

Gwaeron · November 8, 2021, 3:10am

F# is a curried language though, just like Haskell which I mentioned, so there’s really no such thing as an “arity 3 function”, just a single-argument function that returns a new single-argument function that returns a single argument function that returns an integer. See e.g. Currying | F# for fun and profit

This form of piping makes perfect sense in curried languages, where the arity is not (and indeed cannot be) part of the function signature.
The add example from above in Haskell (and probably identical in F#) would be

add a b c = a + b + c

Calling add 1 2 would return a new function that expects a single argument and returns an integer. This is not the case in Erlang (or Elixir for that matter, but that’s beside the point). The pipe operator in F# has no special status, it is simply the function

(|>) x f = f x

which is precisely what I’m suggesting is the best idea in my original post.

cmo · November 8, 2021, 4:19am

Yeah for sure. I was writing F# before I got the sh*ts with it and discovered the BEAM. I don’t miss it, though I love discriminated unions

The point I was trying to make was that it is convention in Elixir to pipe into the first argument and so commonplace and useful that you’re not getting confused about arities.

Erlang doesn’t have this convention , e.g. queue, so your idea of using a symbol to specify the position ala Magritte is bang on in my opinion.

Maria-12648430 · November 9, 2021, 3:30pm

I was with you until you said…

… and went away when you said…

Ok, jokes aside.

Now I may have missed the real point here, but we already have partial application, just more wordy than (< 0), namely fun(X) -> X < 0 end. So what your suggestion essentially boils down to is more or less a shorthand form, right?

As suggested, the placeholder shorthand form has a few shortcomings when compared to the verbose form which I’d like to point out:

Argument order is fixed. You can only leave some out, but the overall order is the same. For example, using the verbose form, this is possible: fun(X, Y) -> otherfun(foo, Y, X) end, but not with the shorthand form.
Arguments can not be repeated in the call to the full form. For example, using the verbose form, this is possible: Double=fun(X) -> add(X, X) end, but not with the shorthand form.
The shorthand form can only reduce (or keep) the arity of the resulting function, not increase it. For example, using the verbose form, this is possible: fun(X, Y, _Z) -> otherfun(X, Y) end, but not with the shorthand form.

I don’t think that those are unlikely edge cases, in fact I guess they are quite common ones. If so, with the placeholder shorthand notation falling short () in a significant number of what would otherwise be its primary use cases, I’m not sure if Yet Another Way of writing something is warranted.
What do others think?

Finally, I guess having this feature might lead to a lot of really ugly code. I myself might be tempted to use something like lists:filter(erlang:'<'(_, 0), SomeList). And not too much later, people reading it would probably hate me, including myself

Gwaeron · November 10, 2021, 2:21am

Not quite - that’s just a lambda function. Maybe using an operator instead of a function in the example was a bad idea. The point of partial function application is to apply a subset of the arguments and get a new function in return, so you’re absolutely correct that the end result would be the same.

All things you list as shortcomings while technically true, are not in the scope of partial function application. I wouldn’t call them shortcomings of PFA since they’re unrelated concepts. Reducing the arity is the entire point (we are after all doing partial application, not inflation). Your examples would be normal lambda functions instead (also in languages with PFA), or use of higher-order functions like flip.

I agree that there might be limited value in implementing “yet another way of doing it,” the reason I’ve been interested in it is due to how elegant it can make code in languages that have it and that maybe my toy examples haven’t expressed in a very good way.

I don’t think that just because something can be used in an ugly way, it shouldn’t be implemented. You can write erlang:'<'(A, B) in your code already today, but hopefully you aren’t. If PFA was also implemented for operators your example could look like lists:filter(_ < 0, SomeList) which in my eyes looks much neater than lists:filter(fun(X) -> X < 0 end, SomeList).

Maria-12648430 · November 10, 2021, 11:48am

You’re right, I was sloppy

And you are right here, too. Maybe I should have called it (implicit? inherent?) limitations, not shortcomings.

In retrospect, I think my argument was badly structured. Let me try again To be clear, I’m not against the feature as such. I’m putting things up for debate, things to think about if you will. Convince me and/or a significant number of others and I might even be interested in taking part in making it become a reality

(In what I’m about to say, excuse me if I’m sloppy again and mix or misname concepts I think you’ll get the drift, anyway.)

With lambdas, we can do anything that we can do via partial application, plus all sorts of things that partial application can’t do (see my previous list of “shortcomings”). For sake of brevity, let’s say that lambdas are the more powerful concept.

As some kind of analogy, with explicit recursion we can do everything that, say, lists:foldl/3 can, plus all sorts of things that a fold can’t do. Anyway, if your use case looks like a fold, you can usually do it with a fold, and don’t have to resort to explicit recursion. If that was different, ie if many/most fold-like things couldn’t be done with a fold and you had to regularly resort to explicit recursion, the usefulness of folds would diminish, and beyond some point it just wouldn’t be worth to have it at all.

So, the question I’m asking is, in how many places where we would like to use a partially applied function will we be actually able to use it, and in how many cases will we have to resort to lambdas just because arguments are delivered in the wrong order or there are too many of them etc?

Take maps:filter/2 for example, assume you want to filter out all entries with negative values from a map. The arguments to the predicate function are the key and value, that is, there are supposed to be 2. You can’t do maps:filter(erlang:'>='(_, 0), Map) (or use the _ >= 0 you suggested), even though that would be an almost perfect use case. Instead, you have to use a lambda that ignores the key.
Obviously, you don’t have to use partial application just everywhere. But having such cases, which IMO are common everyday ones, significantly diminishes its usefulness. Know what I mean?

Not habitually, no, and I don’t see it often in other peoples code, but it happens.
But I mostly put that down to two facts, namely that it is not widely known that you can do this, and that cases where using erlang:'<'/2 instead of the < operator would make even the least bit of sense are extremely rare.
This could change with partial application however, and erlang:'<' would start to appear in code more often, so more people would get to know about it and think it’s fine because everybody else is doing it, and so in turn… etcetc.

I question that, because of what I just said above. And yes, practically anything can be used in an ugly way or other and is still implemented, but at the same time the advantages that come with the (intended) use vastly outweigh that.
To be blatant, I don’t see that here (yet), I just see syntactic sugar (don’t take me too serious, I tend to exaggerate )

Maria-12648430 · November 10, 2021, 12:27pm

Oh, something else that came to mind yesterday. There is an ambiguity here.

Res=add(1, 2, _).

This is clear, it will give you an partially applied function of arity 1.

Res=add(1, 2, 3).

This is ambiguous, the result could be whatever add/3 returns, or a 0-arity function. Now in Haskell, the latter would not make sense (I think) or be very useful, so it can be nailed to be a call to add/3 (or whatever the Haskell way to say that is). In Erlang, however, it could be a sideeffect-ish function, and calling Res() could make sense.

For example:

IncCounter=ets:update_counter(Tab, Key, 1).

This would, with the suggested notation, result in a call to ets:update_counter/3 and store the new counter value in IncCounter.
But instead it might have been intended to be used as a 0-arity function that updates the counter each time it is called, ie like:

> IncCounter().
1
> IncCounter().
2
...

For that reason, I would suggest adding the fun keyword in front (always, even when there are placeholders that would it make clear to the compiler what is to be done, for consistency), such that…

ets:update_counter(Tab, Key, 1).

… would result in a call, and…

fun ets:update_counter(Tab, Key, 1).

… would result in a 0-arity function, and…

ets:update_counter(Tab, Key, _)

… would result in an error (missing fun keyword in front of partial application).

It would also make casual reading of code easier. Without fun, you will have to look at the argument list to see what will happen, a call or a partial application, while the fun keyword would even be syntax-highlighted by simple IDEs/editors, making it even easier to see.

On the downside, this makes it harder to apply the partial application notation to operators, as was suggested by @Gwaeron in an earlier post. Maybe something like fun (_ < 0)? (Looks ugly I guess).

zuiderkwast · November 10, 2021, 1:27pm

I like the idea and think it fits nicely into the language. Perhaps even more so with fun in front as in Maria’s last suggestion.

If the ugliness of erlang:'<'(_, 0) is a concern, then perhaps we can add yet another syntactic sugar for binops: (_ < 0)? (with or without fun in front)

I seems strait-forward to implement it as a parse transform. Why are parse-transforms bad again? (AFAIK the main trouble is the compile time dependency between modules, but if it’s in OTP, that not an issue. There are parse transforms shipped with OTP already, e.g. ms_transform.) The other option is to have it as a real feature (but it would probably work as a parse transform behind the scenes anyway).

Apparently, the OTP team has plans for a way to enable experimental features per module, which could make it easier for experimental features to get accepted. (“Since the OTP team is now planning to introduce a new mechanism for activation of new language features per module the potential risk for incompatibility with introducing a new keyword is reduced. Only modules explicitly using the new language feature will be impacted.”, EEP-49)

OvermindDL1 · November 10, 2021, 5:23pm

I do indeed very much like the _ style of putting a hole. Like perhaps you can allow _ for a single hole, or perhaps number it to say the argument position for multiple holes like _1 and _2 and so forth. You still need some way to say what scope to enclose in the closure though, you can’t just say “the tightest” as that would just be the _ and nothing else turning into the identity function or whatever, so some kind of ‘lambda’ symbol to say what scope to enclose, or fun for erlang, which it already is the thing that encloses the scope, which gets to the issue that you can’t really get reliably more succinct without making certain tradeoffs (enclosing parenthesis for example, but what about function calls, etc…) or you just have single-arg functions like most ML languages (which makes currying very trivial with extremely simple syntax).

If only there were types, then it could be inferred based on the type, like being passed to a function, lol.

Yep, that’s what I was getting it, you need to be able to define the scope.

juhlig · November 10, 2021, 8:24pm

No, that’s not possible or feasible, at the very least it would create another ambiguity. _1, _2 etc are regular variables like X1, X2 etc are. Only _ alone is special, any other underscore-prefixed variables are regular variables with the peculiarity that they don’t cause warnings when they are unused. So…

_1=foo,
fun erlang:'<'(_1, 0).

… would not create a 1-arity function (equivalent to fun(_1) -> erlang:'<'(_1, 0) end) but a 0-arity one (equivalent to fun() -> erlang:'<'(_1, 0) end, which means fun() -> erlang:'<'(foo, 0) end here).

OvermindDL1 · November 10, 2021, 9:08pm

Just an example from an elixir library that did that, could use '1, and '2 or `1 or whatever, lots of options. Issue is still what scope to bind it to, that’s the big issue.

juhlig · November 10, 2021, 9:27pm

Sorry, I don’t get that… I don’t know what @Gwaeron originally imagined, but I think of it like (I guess) @Maria-12648430 does, and @zuiderkwast seems to think something similar as he mentioned parse transforms: That something of the form fun somefun(A, _, B, _, C) would/should be transformed into fun(X, Y) -> somefun(A, X, B, Y, C) end, simple as that. The scope would be the same as if you had written the lambda there instead of the partial application construct.

OvermindDL1 · November 10, 2021, 9:30pm

That doesn’t really save enough of anything to make it worth it though was what I was trying to imply, which is probably why nothing like it has ended up in Erlang to date.

Gwaeron · November 11, 2021, 5:13am

Wow! Lots of new comments here, giving me exactly the kind of criticism I was hoping for, with all sorts of things I hadn’t considered.

These are all excellent points! fun does seem like it’d be more in line with the rest of the language. That’s much better than what I was thinking. But yeah, the operators look pretty awkward and un-Erlangy this way, so I could see your erlang:'<'/2 fears coming true. I’ll have to get back to the drawing table for that one… Same goes with list/binary comprehensions. fun [ f(X) || X <- _ ] looks very wrong.

I think they are fine as a concept, and perfectly fine if part of the standard library. I think this could absolutely be implemented as a parse transform (I have made a prototype myself). I think it breaks the principle of least surprise in a monumental way to use a home-brewed parse transform that changes the syntax of the language. But if it shipped officially that’d not be an issue.

Thanks, I picked _ because I couldn’t think of any situation where it would be ambiguous with _ on the right-hand-side of a pattern match, and conceptually it feels consistent with “ignore this (for now)”. But I can buy that it introduces confusion to use the _ for two very different things and maybe some completely different symbol would fit better. _1 and so on wouldn’t work for the reasons mentioned by others, but with a different unused symbol, maybe @, it could be done. fun subtract(@2, @1) == fun(X, Y) -> subtract(Y, X) end. But with that I think we’re dangerously close to it being as long as a lambda function would be, without adding clarity. I think this addresses one of @Maria-12648430’s previous comments.

Yep, absolutely correct. A shorthand form of writing funs.

I think it’d be a nice thing to have, but not necessary since it’s just a disguised lambda. Pretty much an extended form of fun f/2 which we already have, and I think everyone agrees is very useful. But it happens often enough to me, that I only need one of those arguments in f/2 and have to wrap a single function call inside a fun and it makes me wish there was a neater way of writing it. List comprehensions often do the trick, but they can’t simplify things like this foldl in beam_ssa_pre_codegen, dict:map/2, orddict:fold/3, maps:filter/2 and so on. Potentially allowing a concise and positional-agnostic pipe operator in the future is a nice side effect.

Maria-12648430 · November 11, 2021, 11:24am

Well, I think I have nothing more to add right now…

It would be nice if we could have more opinions on the subject. As of now, (only) 8 different people have posted something here if my count is correct, including the OP and myself.
@Gwaeron, maybe post a link to this thread with a summary so far on the erlang-questions ML to get some more? That carries the risk of getting more than you wish for in terms of feature wishes, though

OvermindDL1 · November 12, 2021, 4:22pm

I think it would be good to show actual use-cases where it shows a measurable benefit, that would be most convincing one way or another (and the lack of such things are showing that adding partial application is not worth it).