The need for "protocols" in Erlang

Erlang/OTP Proposals Proposals: Ideas
,
21

yep. my problem with attaching a data structure to a single implementation is that you don’t have the option to specify something different for the same data structure when you need such a thing.

for those i think they have the same usage, if it’s something handled internally, i can pass a function to handle just one level of the data structure and the library/internal code take care of handle the nesting of it as it prepare for encoding/interpolate. thinking for a record or a map that i’m either encoding to json or interpolating, for botth cases i’d just need to take care of how i want the data shaped.

I’m not the OP you quoted but afaik, protocols in clojure are not “in the same place” as protocols in elixir. mostly because of the runtime they interact with. there, protocols are intended to be a way to define interfaces on the clojure side, so in case you’re using some java interop, you could have “interfaces” that are better in clojure and can work with java stuff.
clojure community in general afaik goes against using deftype and defrecord(that would be ways of creating “classes” in clojure) that are the things supposed to use protocols. they work in the direction of using plain maps with qualified keywords(it would something like atoms with namespaces, beam doesn’t support such stuff afaik).

1 Like
22

as far as you don’t have any use case that goes against the constraints of 1 struct have 1 unique implementation of the protocol.
edge cases to that i’ve dealt with:

  1. json encoding different payloads for the same struct. since all good libraries for json ecoding in elixir relies on protocols, it’s always tricky to go around that. more difficult when people start using @derive in large codebase, it’s a little bit tricky to track all places that the thing is encoded.

  2. when you have a protocol and a struct that you didn’t define but you need that thing implemented. it becomes troublesome because the right way of doing it is actually have your own struct that works as middleman for that, but again, really difficult to make sure that the transformation of one struct to another happens in all places.

those two were the biggest painpoints i had with protocols… a very long time ago i had an issue with implementing the same protocol twice for the same struct, but that was just a messy codebase and i don’t even know if such a thing is still possible.

23

The difference is that JSON is recursive, so you need the dependency injection for the recursion. I am assuming interpolation isn’t recursive, so there is no need to inject anything, you do the dispatching at the call site.

It is completely fine for you to implement a protocol for a struct you don’t own, especially on your own application, given you won’t push this decision to anyone downstream.

24

As for the fsck fairy, I heard a great polite version (Cadogan car expert):
“laying the foundation for a building full of fail”.
(I’ve probably misremembered it.)

25

I think related:

A few times I’ve done a pattern of making a data structure be {module(), data()} (most recent case being things like a Tracer in OpenTelemetry, https://github.com/open-telemetry/opentelemetry-erlang/blob/main/apps/opentelemetry_api/src/otel_tracer.erl#L51-L54) that the module providing the behaviour will know to call

Something that’d be nice as an alternative would be the ability to define a map extending another map. That way the “interface” could include a type like:

-type t() :: #{__module__ := module()}.

And the implementation in otel_tracer_default adds the data part:

-type t() :: #{otel_tracer:t()  + 
               sampler := otel_sampler:t(),
               ...}.

Now otel_tracer could pull out the module by matching on __module__ instead of {Module, Data} making matching on the structure of the data less awkward since it never has to be pulled out of a 2-tuple.

To even better define the structure would be if the “interface” could be part of the module type:

-type t() :: #{__module__ :: module(otel_tracer)}.

Then either some sugar to make it part of the language and not have to manually setup the __module__ map or just do this as convention.

26

This whole thing reminds me of OOP. As far as I know, this use
of the term “protocol” comes from Objective-C (see Apple’s 2014

documentation of protocols in Objective-C
https://developer.apple.com/library/archive/documentation/Cocoa/Conceptual/ProgrammingWithObjectiveC/WorkingwithProtocols/WorkingwithProtocols.html
) – while the term/concept is a core aspect of the 1998 ANSI Smalltalk
standard, it’s part of the standard, not part of the language, and in
any case, any influence this may have had was probably filtered through
Objective-C.

So, hands up, how many people here are familiar with what Joe Armstrong
thought about OOP? I saved a copy for a Software Engineering class:
https://www.cs.otago.ac.nz/staffpriv/ok/Joe-Hates-OO.htm

I’m going to argue, by thinking about (communication) protocols (real
distributed programming protocols), and using the example of JSON
(but XML or YAML would do just as well), that “protocols” are the
wrong answer. They put the wrong code in the wrong place. I like to
think that Joe Armstrong would agree with this. Since this is just me
ranting at a keyboard at midnight, I’m quite sure he would say it
better.

OK. Let’s consider generating JSON.
The whole point of JSON is to be almost as simple as S-expressions,
at any rate, simple enough NOT to need magic machinery in your
programming language.

I’ve implemented JSON in several programming languages, and the
hardest thing to deal with was string escaping. (The second hardest
is dealing with the JSON5 extensions. Why do people find extending
communication protocols such a compelling itch to scratch? Oh right,
because they’re done wrong the first time.) (The third hardest will
be STON, and it’s so, so, well, what’s really hard is that I really
don’t WANT to implement it. Why DO people find extending data
transport protocols such a compelling itch to scratch? Oh right,
because they don’t do the job you want them to do.)

And this brings me to the issue of non-JSON data structures.
The absolutely central problem here is that there is no right way
to do it. Let’s take a nice little example, one I’ve actually
had to deal with. A circle. Which do you use?
{“r” : R, “x” : X, “y” : Y}
{“type” : “circle”, “centre” : [X,Y], “radius” : R}
[X,Y,R]
amongst others. Pick one, be happy, then find you need to
exchange data with someone who wants something else.

What we actually have is
Protocol layer 0 : byte transport
Protocol layer 1 : pure JSON transport
Protocol layer 2 : JSON with encoded geometric objects
Protocol layer N+1 : who knows?

Note that each layer is more restrictive than the lower-numbered layers.
Protocol layer 2 almost certainly should not be able to generate
arbitrary JSON.

How a data structure should be written out as JSON,
and how it should be read back, does NOT depend on the data type!
It depends on Protocol layer 2!
Something that can only understand layers 0 and 1 is fine for
relaying layer 2 data. Something that processes layer 2 data
has to know what to expect.

Oh, and writing is easy. My Smalltalk JSON implementation has
a JsonVisitor class you can extend, and a JsonWriter class derived
from that, and also a JsonBuilder class, so by providing the
interface JsonVisitor expects, you can easily choose between
converting a data structure to JSON internally, or streaming it
out without building a temporary data structure. Nice.
(A) It doesn’t work in general because the right output depends
on the (COMMUNICATIONS) protocol, not the data type.
This is why I know that for example being able to extend
visit_json(Datum, Visitor) is not the way to go.

(B) There is no nice stream-in-without-intermediate-data-structure
equivalent because the information you need isn’t available at the
beginning of a JSON element. (With well-designed XML the information
you need IS there at the beginning. Not all XML is well-designed.)
You have to buffer it somehow.

It was an education.

So here it is. You have Protocol layer 2. Joe Armstrong was
RIGHT. It’s protocol layer 2 that should be in complete charge
of how to convert a data structure to JSON form, NOT the data
structure. And if the data structure is inside a module and
doesn’t give you the information you need for Protocol layer 2,
the interface of that module is defective.

The other problem with
w := JsonWriter on: aStream.
w visit: someDataStructure
=> someDataStructure visitJson: w
is that someDataStructure visitJson: w could do ANYTHING.
It could be as harmless as

visitJson: visitor
visitor beginObject: 3;
visitKey: ‘x’ andValue: x;
visitKey: ‘y’ andValue: y;
visitKey: ‘r’ andValue: r;
endObject.
It could be as dangerous as
visitJson: visitor
OS exec ‘rm’ withArguments: {‘-rf’. Filename home}.

No. What you do is have a JSON streaming library that
provides operations for your Protocol layer 2 output code to
call, so you don’t repeat yourself. Or rather, for each
Protocol layer 2 to call differently as appropriate.

As for input, I still don’t have a cleaner way to do it than
to have the library code construct an intermediate data structure
that represents exactly the information in the Protocol layer 1
data, and then walks over it generating Protocol layer 2 data.
Which again is the business of Protocol layer 2, because making
the data know about the representation
(A) does not WORK in general for reasons already mentioned and
(B) COUPLES (remember that term from your Software Engineering
classes? It’s something to avoid) your data structure to JSON
(or XML, or ASN.1, or whatever you’re using.)

For JSON (or XML or whatever),

  • input is as important as output, and when you’re doing
    input you don’t have an object to delegate the task to
  • multiple mappings may need to be supported in the same node
  • it’s the JSON (or XML or whatever) value that is specified
    in the higher level protocol(s); what kind of (Smalltalk,
    Erlang, Lisp, Java) data structure you map that to is an
    implementation detail you want to be easy to change without
    having to rewrite a lot of code.

That’s my 29c US (50c NZ) on protocols-and-JSON.

2 Likes
27

I feel there are two discussions here which are getting mixed.

The name “protocols” is not important. We can call the feature anything.

Is it OOP? Philip Wadler and co. discussed this in the linked paper:

Type classes extend the Hindley/Milner polymorphic type system, and provide a new approach to issues that arise in object-oriented programming, bounded type quantification, and abstract data types.

Type classes appear to be closely related to issues that arise in object-oriented programming, bounded quantification of types, and abstract data types [CWSS, MP85, Key85]. Some of the connections are outlined below, but more work is required to understand these relations fully.

In particular, there is a similarity on how closed classes make it hard to add new functions to an existing type. While Erlang’s (and functional languages at the time) make it hard to add new types to an existing function. Type classes (or interfaces/protocols as in this proposal) are meant to solve both problems. The paper does a better job explaining these than I could ever do.

Still, I don’t think “reminding of OOP” is a concern. The fact something is OOP does not necessarily make it good or bad, and I sure hope we can argue past that. After all, many will gladly point out that Erlang processes are objects per Alan Kay’s description, and it is assumed to be a good thing.

As an aside, I did discuss protocols with Joe. Despite not liking the name, he seemed to appreciate them as a solution to the pretty printing problem. I did not discuss the JSON problem. Even if we did, I’d rather not speculate since he could have changed his mind at any point. :slight_smile:

Having typeclasses/interfaces/protocols does not mean they belong to layer 2.

What dictates what are valid data structures for the JSON Protocol layer 1? We can say lists, maps, strings, integers, and floats. Let’s also say Erlang/OTP includes dict (dual of maps) and array, because an array is a low-level data structure that maps to a JavaScript array.

Now let’s imagine that at my company I created a fast_array module, that improves Erlang’s array. Its API is the same as array and it should be a drop-in replacement. Except that I cannot encode it to JSON without forking Erlang/OTP.

The fact I cannot extend Layer 1, even for mechanical data structures that have no semantic meaning in my application, means that I either need to fork the JSON code, or handle fast_array as a Layer 2 concern, while for all intents and purposes, it should be synonyms to array.

4 Likes
28

Other than that, you are obviously correct that protocols/typeclasses/interfaces do not solve the input problem, since there is no data type to dispatch to. And yes, “multiple dispatch” may be a concern, as brought up by @Maria-12648430. Although I would argue that “mechanical data structures” (i.e. with no semantic meaning like my fast_array example), the odds of having different representations at Layer 1 is very unlikely (and if it does, it should arguably live at Layer 2).

29

There seems to be some miscommunication happening.
“Having typeclasses/interfaces/protocols does not mean they belong to layer 2.”
That’s not just not what I said, it’s a category error.

Protocol layer 2 is about abstract information.
typeclasses/interfaces/protocols are an implementation issue.
There are three things:

PROTOCOL LAYER TWO (this is conceptual)

DATA STRUCTURE PROTOCOL LAYER ONE (UBF, BSON, JSON, ASN.1, XML, &c)

IN CODE |
lower level transport system

When I say that protocol layer 2 is conceptual, I mean that it deals with concepts like “user credentials” or “geographic location”.
There are TWO mappings to/from Protocol layer 2, and they have to be
able to vary INDEPENDENTLY:

  • the mapping between data structures in some programming language
    and the concepts of protocol layer 2, these might be utterly different
    in different programming languages and would almost surely need to
    change with scale in a single programming language
  • the mapping between concepts in protocol layer 2 and forms in
    protocol layer 2. Indeed, you are quite likely to find yourself
    needing more than one version of protocol layer 1; JSON and XML
    for example. This further stresses that NO HINT OF PROTOCOL LAYER
    ONE SHOULD BE VISIBLE in the mapping between your data structures
    and protocol layer 2.

Let me stress this again: we do not generate JSON just for the fun of it;
we generate JSON in order to communicate something else, and letting
JSON be visible in our own data-structure/layer two mapping just makes it
HARDER to adapt to other “wire protocols” (XDR, Protobufs, whatever).

And if you have to talk to two clients with different layer two protocols,
you need in general DIFFERENT data structure<->wire protocol translations.
This is why protocol level 2 needs to be reified in your program and IT
needs to manage the translations, not your data structures. Your data
structures shouldn’t “know” that JSON even EXISTS let alone be tied to it.

"What dictates what are valid data structures for the JSON Protocol layer 1? We can say lists, maps, strings, integers, and floats. Let’s also say Erlang/OTP includes dict (dual of maps) and array, because an array is a low-level data structure that maps to a JavaScript array.

Now let’s imagine that at my company I created a fast_array module, that improves Erlang’s array. Its API is the same as array and it should be a drop-in replacement. Except that I cannot encode it to JSON without forking Erlang/OTP."

This couldn’t be wronger. The point of JSON is that it is very simple. Seriously. The last JSON writer I wrote (in a Lispy language) was 42 raw lines + 16 lines to write strings. To generate output in a way that accepts your new type, you have to clone at most a page of code. The corresponding JSON reader is 124 raw lines. 35 deal with JSON strings. 24 deal with numbers. 17 deal with JSON5 comments. 48 do everything else and looking at them, only 44 are really needed. Making it generate a different kind of sequence or a different kind of dictionary would be one line changes.

(“Raw” lines means with blank lines and comments removed, but counting end brackets like “end” as full line.)

If you want to encode/decode JSON differently, you need to clone and modify a JSON writer/reader. AND THIS IS TRIVIAL. Which is the point of JSON. The claim that you would have to fork the whole of Erlang/OTP is absurdly exaggerated.

But remember, each layer two protocol will want its own way of decoding JSON. One application will want numbers in certain contexts to represent timestamps and never change strings. Another application will want strings in certain contexts to represent timestamps and never change numbers.

The whole idea that there is, or should be, or even could be a “one size fits all” (extensible) mapping between data structures in any programming language and JSON is wrong. Even in Javascript, you sometimes have to convert something else to JSON-data before generating JSON-text from it.

In any case, it’s up to the Protocol layer 2 code to decide what it is going to accept or generate. Lying to it, pretending to offer it what it wants but actually giving it something else, is “laying the foundations for a building made of fail.” I know about the Liskov Substitution Principle. I’ve never yet met one data type that’s an exact drop-in replacement for another, and I’ve struggled very hard to produce some. In order to get close to that, you hnave to use very narrow interfaces, which means not being able to exploit composite operations that are especially efficient.

Here is an example I had recently. Suppose you have a Point.
{“x”:X,“y”:Y} is the obvious representation. Suppose, as many Smalltalk systems do, you have a PointArray. The default output for such a thing will be [{“x”:X1,“y”,Y1},…,{“x”:Xn,“y”:Yn}]. But suppose you have a lot of these to exchange. You can save a factor of 2 by using[[“x”,“y”],[X1,Y1],…,[Xn,Yn]]. But you DON’T want to tie this to PointArray because some receivers are not expecting this optimisation. A revised Protocol layer 2 where the sender and receiver agree on this optimisation, that’s where the code goes, and it doesn’t matter whether the datum is an ordinary Array of Points or a specialised PointArray or even a DoublyLinkedList of Points, at this point in the higher level protocol the sender demands and the receiver expects some sort of sequence of Points. It’s the (communication) protocol that decides, NOT the data type.

What we really want for JSON is some sort of declarative notation that says

  • THIS concept

  • is represented THIS way in JSON

  • and implemented THIS way in Erlang

  • and THIS way in Common Lisp

  • and THIS way in Smalltalk

  • and so on.

I’m afraid I have a bad habit of whipping up little tabular DSLs with AWK scripts to generate the code, instead of implementing something I could report at a conference.

3 Likes
30

This is why introduce namespaces in Erlang need to be done first .
Erlang should be able to handle two modules with same basename and different namespaces . Something like an optional *prefixe::*module:function .

-module(blah).
-namespace(“www.acme.com”).
-namespaces([{‘a’,“www.other.com”}]).

Then blah:f() is local call in www.acme.com namespace and a::blah:f() a call in www.other.com namespace.

1 Like
31

We can already give modules names like
nz.ac.otago.cs.ok.unrolled_lists
What, exactly, do you mean by ‘namespaces’?

If we had applications that lived in their own virtual nodes,
which has long been wanted on other grounds,

what would namespaces add?

There’s a 700 kSLOC C program I work on from time to time
and a 1 MSLOC Fortran program I thankfully had no hand in
developing but do have on my machines because I needed to
know what OS facilities it depended on. C of course has no
namespaces, and the Fortran program doesn’t even use F90 MODULEs.

I note that when Haskell went from simple module names to
hierarchical names, modules went from things like
import qualified List
… List.scanl f a xs
to stuff like
import qualified Data.List as L
… L.scanl f a xs
so that modules became littered with single-letter module aliases
because the full names were intolerable to repeat.

32

In an Erlang project using third party code, what if a module came with same name than yours ?
I rarely see nz.ac.otago.cs.ok or such in module names, especially in widely scattered opensource applications or modules. BTW Erlang module itself does not come with a com.ericsson… prefix and AFAIK no recommendations are done in official documentation regarding this.
With lot of dependencies, risk is not null.
‘namespaces’ are like XML namespaces allowing several same semantic to be used at same time.

This can also be used for other goals, like packager tools (hex …) to find modules or application from a specific source, and also to denied use from some unsecure source. Many usages.

33

i’ve mentioned qualified keywords in clojure in other context here but i’d love to see something like that in erlang. if erlang had namespaced atoms and since module names are just atoms, they’d be naturally namespaced this way.

34

the way i see it, interpolation is recursive. let’s say you have a map and one of it’s values is a datetime tuple, defining how to interpolate the map shoudn’t include how you interpolate the datetime tuple.

but since protocol consolidation isn’t scoped, downstream changes could conflict with my implementation of the protocol. let’s say the application owner of that struct now implements the protocol i was implementing in my application. implementations would conflict and i wouldn’t be able to chose which implementation should be consolidated. so to properly “own” that implementation i need to have the middleman struct.

35

There was a mechanism called packages that did exactly the same thing, but is abandoned/not supported for some reason (packages). Maybe we can learn something from it.

Also, if third party app have modules that have the same name, what’s preventing them to have namespaces with the same name?

1 Like
36

The old bugaboo: “what if library X and library Y both have a module Z?”

I use several different Smalltalk systems. I just checked one, which turns out to have 1.4MSLOC of sources preloaded, and has hundreds of third-part packages available. Does that Smalltalk dialect have namespaces? Nope.
Is it a problem in practice? Nope. Common practice is to include a
package prefix if necessary, so a Chess game might have ChessBoard, ChessPiece, ChessPlayer, etc, but that improves readability anyway.

I did some work on this for SERC back in the 1990s.
(1) Namespaces just add structure to names. We want a way of
ENCAPSULATING the names of modules that are just a private
part of an application. And of my gosh, it’s not just modules.
The thing I used to really worry about was registered names
for processes.

So we want some sort of “box” that can contain modules and
pid names and possibly other things, and hides them from everything
else in the node, exposing just a small controlled subset of them.

(2) We HAVE a kind of box that does this. It’s called a node.

(3) Ah HAH! We’re right back at Lawrie Brown’s “Safer Erlang”.

If library X provides module Z as part of its published interface,
and library Y provides module Z as part of its published interface,
then I know not to put them in the same (virtual) node.

XML namespaces are a horror. They have the same defect as Haskell
module aliases. Consider
<m:text> … </m:text>
Without looking thousands of lines away, what does m stand for?

Without an alias facility as in XML or Haskell, nontrivial namespace
hierarchies are unreadable to use.
With an alias facility, we have BOTH the same name may stand for
different spaces AND different names may stand for the same space.

Virtual nodes, however, let you have multiple modules with the same
name and multiple versions of the same name and multiple process
registries AND control resource (file descriptor &c) management AND
offer protection against untrustworthy code without the overhead of
operating system process switching or operating system IPC.

We’ve known that this was worth doing for 25+ years.
We’ve known it was doable for 25+ years.

1 Like
37

Even otp team reused the pg module name with same function names in both modules without guarantee that they are doing the same.
Namespaces like “ericsson.com/v2” could implement module instances (different from version here)

38

Maybe instead of introducing a completely new concept the focus should be on the dispatch mechanism and definition of some new predefined behaviours (which could be shipped with e.g. kernel or stdlib)? IMO protocols and behaviours do seem slightly redundant in Elixir and they would be so in Erlang, though I’m not very familiar with the Elixir implementation details.

The three examples from the top post could all be expressed similarly to this, by predefining such behaviours in OTP:

%% Example #1: pretty printing

-module(display). %% or gen_display

-callback print(any()) -> iolist().

This addresses the question in the last bullet point:

A module encapsulates processing of data of some type (sometimes it’s expressed by using an opaque type), so IMO 1 and 3 fold into the behaviour concept. What do you think?

1 Like
39

When you think about a behaviour, such as Module:foobar(Arg1, Arg2, ...), Module tells where the implementation will be found. When you do Pid ! message, Pid says where the message is received (which is a process running some code). So a similar mechanism for data would require somefun(Data, Arg1, Arg2, ...) to dispatch somewhere based on the Data. Therefore a dispatching mechanism is crucial for the analogy.

A gen_display would solve the problem at hand but it doesn’t dispatch based on the data. In this case, you would need to implement your own dispatching mechanism. For example, you could do this:

-module(gen_display).

-callback print(any()) -> {ok, iolist()} | next.

Now you can allow multiple gen_display implementations to be installed. For a given data type, each installed handler is invoked in order, stopping if it returns {ok, iolist()} or continuing to the next handler if it returns next. It is definitely a valid answer to the problem, it is similar to how logger filters work, but I don’t think it answers “what is the interface of data?” (but perhaps it is not a question worth answering anyway).

1 Like
40

Hello!

I worry that this would add a great deal of complexity to Erlang and make it potentially a good deal harder to read, understand, and debug Erlang code. Today if you wish to understand what a function does you go to the single place where it is defined. With this feature there are a potentially many implementations, and they could be anywhere in the codebase.

The part I think I am most interested in here is how values are determined as belonging to various types. My first thought is that we would need namespaced records so that the implementation can do (get_record_module(Record)):to_json(Record) or such.

Although, if we did have namespaced records would we need this feature? Couldn’t the behaviour module be used as it is today, as the module could be easily identified? This would seem less “magical” and closer to Erlang as it is today.

nit: I agree that a different name would be needed as the name protocol already has a meaning in Erlang. I’d probably go for “trait” as that seems to be the popular name for this feature with new languages, even if this proposal isn’t capable of supporting generic data structures like traits/type-classes tend to be able to.

6 Likes