At Code BEAM Europe, I had several discussions which all pointed out towards a need for protocols, or a similar solution, in Erlang. This post is an attempt to summarize them and start a discussion on the topic.
What are protocols?
In Erlang, we are familiar with polymorphic code:
negate(Int) when is_integer(Int) -> -Int;
negate(Bool) when is_boolean(Bool) -> not Bool.
The code above is poly (many) morphic (shapes) because it can handle different types of arguments (integers and booleans).
The limitation in Erlangâs polymorphism is that it is âclosed to extensionâ. It is not possible to add additional clauses to negate/1 unless we change its original source.
Many programming languages provide a mechanism to have âopen polymorphismâ. Typeclasses in Haskell (with a paper dated back to 89), Protocols in Clojure, Interfaces in Go, Protocols in Elixir, etc. I am using the name âProtocolsâ for the rest of the post, as that is what it is called in Elixir, which Elixir inherited from Clojure, all of them being dynamic languages.
Example #1: pretty printing
One canonical example of protocols is pretty printing data structures. Many data structures in Erlang have complex implementation, which is private to the data structure, but ends up leaking in the shell:
1> gb_sets:from_list([foo, bar, 123, {tuple}]).
{4,{foo,{bar,{123,nil,nil},nil},{{tuple},nil,nil}}}
2> re:compile("(a|b)", []).
{ok,{re_pattern,1,0,0,
<<69,82,67,80,86,0,0,0,0,0,0,0,1,0,0,0,255,255,255,255,
255,255,...>>}}
While Erlang could potentially provide mechanisms to customize the shell, it goes beyond the shell. For example, any logging mechanism may print the internals of data structure.
3rd party libraries may also have the same constraints: they want to create new data types and abstractions without leaking their implementation details.
This is also problematic for user data types. With you have a user record, with email and password fields, you want to make sure those are not written in logs for data privacy reasons. Therefore, we need a mechanism where the data type extends Erlang over how it should be pretty printed. And protocols are a possible mechanism for such.
Example #2: JSON
There is a discussion for adding JSON support to Erlang/OTP. Serialization is another area where protocols shine, as they provide an extensible mechanism for data types to express they can be converted to JSON (or any other serialization format).
This way, a JSON implementation in Erlang could support the native data types out of the box. Custom types, such as gb_trees, library types, and user-defined types can all opt-in to encoding by âimplementing a protocolâ.
Of course, it is possible to solve this problem by providing a ârecursive custom encoderâ function to the JSON encoding mechanism, but it forces developers to be responsible for stitching together how all the different data types should be encoded.
Example #3: String interpolation
There has been on-going discussions about adding string interpolation to Erlang. Without going into the merits of the feature itself, they can be another use case of protocols. For example, imagine that we want to allow interpolation in Erlang:
"My name is ~{Name}"
Then you need to decide what values can be interpolated. If you say only strings, then thatâs fine, but if you want to allow integers or floats to work naturally, then the same question arises: how to allow custom data types? Such as a library that implements arbitrary decimal precision?
Other examples
There are many other examples of where protocols can be handy. For example, Elixir uses protocols to provide a module, similar to the lists module, but which works with a huge variety of data structure, including maps, file streams, etc. It doesnât mean all of these features need to be added to Erlang, but protocols would enable doing so, if desired.
Different serialization formats may define their own protocols and libraries may define custom protocols for specific needs.
Protocol dispatching
There are many design decisions to be taken around implementing protocols (or similar) in Erlang. The main goal of this thread is to not explore solutions, but bring attention to problems and situations where protocols may be a good fit.
I will also be glad to document how the Elixir implementation works but I assume an Erlang solution could (should?) work differently. I would like to use this final section to discuss the main challenges here.
The core of a protocol is a dispatch mechanism. When you implement a protocol, the implementation is not part of the protocol nor of the data structure. For example, take a possible âjsonâ protocol. If that protocol is defined as part of Erlang/OTP, then Erlang/OTP could implement the âjsonâ protocol for all of its built-in data structures: maps, integers, floats, gb_trees, etc. However, a library author may also want to implement the âjsonâ protocol for their own data structures as well.
I can also be a library author that defines a new âcsvâ protocol. In this case, I also want to define and implement the protocol for all built-in types (which I donât know, they come as part of OTP). I also want to allow library and application authors to implement the âcsvâ protocol for their own data structures.
In other words, you may implement any protocol for any data structure at any time. And because code in Erlang always exists inside modules, the protocol dispatching mechanism most likely needs to be based around modules.
We can do so by breaking the protocol dispatching in two parts: a naming schema and a cached module lookup.
The naming schema
The naming schema defines how we are going to find the module that implements the protocol. For example, when giving a map to the json protocol, we could dictate the protocol implementation must be placed at the json_map module. For integers, in json_integer. For a custom type foo_bar_baz, at json_foo_bar_baz.
Therefore, when calling json:encode(SomeValue), we need to get the type of SomeValue, concatenate with the protocol name, and then invoke that module. Once more, if SomeValue is an integer, then json_integer. If SomeValue is a dict record, then json_dict.
The need for caching
The tricky part is that, the json_integer module may be in three different states:
- loaded in memory
- not loaded but available in disk
- not loaded and not available in disk
If json_integer is loaded in memory then it will be fast. If it is not loaded, then we need to go through the code server and traverse code paths. If none is available, we will traverse all code paths, which may be slow.
The trouble is that some protocols may also define âdefault implementationsâ. For example, a pretty printing protocol must be able to fallback to a default implementation if none is provided, so we always pretty print something. In other words, every time the protocol is invoked, we would traverse all code paths, find nothing, and then fallback to the default one. This will be extremely slow.
Therefore, we need to cache the result of the module lookup: if the module cannot be loaded, we shouldnât try to load it next time around. This cache is tied to the module the protocol is defined (think about it as a âmodule dictionaryâ). Furthermore, we would ideally cache both the computation of the naming schema and the module lookup.
Letâs imagine we want to dispatch a pretty_printing protocol to custom records, such as re_pattern or dict. The pretty_printing protocol would need to have code that looks like this:
-module(pretty_printing).
to_string(Value) when is_tuple(Value), is_atom(element(1, Value)) ->
Record = element(1, Value),
case erlang:module_get(Record, undefined) of
undefined ->
Module = list_to_atom("pretty_printing_" ++ atom_to_list(Record)),
case code:ensure_loaded(Module) of
{ok, Module} ->
erlang:module_put(Record, Module);
Module:to_string(Value);
{error, _} ->
erlang:module_put(Record, pretty_printing_fallback);
pretty_printing_fallback:to_string(Value)
end;
Module ->
Module:to_string(Value)
end.
There are many design decisions to be made around the code above. Ideally we would want to optimize it such that, at runtime, it is literally equivalent to either pretty_printing_re_pattern:to_string(Value) or pretty_printing_fallback:to_string(Value) value, but the Erlang/OTP compiler/runtime team will be the most capable of making these decisions.
Not only that, we also need to talk about cache expiration. After all, if pretty_printing_some_module was not defined at some point, but it is loaded in the future, it must expire its dispatch cache. This could be done by annotating the implementation modules:
-module(pretty_printing_re_pattern).
-expire(pretty_printing, re_pattern).
Of course, I am not advocating for the user to write these, but thatâs the rough lines of a low-level mechanism required to implement protocols.
Thanks for reading and hopefully it gives some ideas, even if I am barking at the wrong tree. 
or a âWTF my girl, are you crazy? 
â. Even the dog being filled with coffee 
could be an expected outcome.
), just because somebody who happened to be around knows how to fill dogs with joy.
, where I ran into some strange problem and protocols where the issue behind it all or made anything hard to debug for that matter. When something tries utilize the string protocol and itâs invalid, itâs quite clear why as an example.