HTTP Content Coding: Erlang External Term Format

Questions / Help
1

I have long thought about using HTTP content negotiation to skip expensive CODECs (i.e. JSON) and just exchange Erlang terms within our Erlang based cloud.

This weekend I finally got around to doing something about it. I created a project (github.com/vances/erlang-content-coding) with an example implementation and test suites.

There is definitely an improvement when you skip the CODEC and just use Erlang external term format (binary_to_term/1,2). The test suite for raw CODEC performance (ecc_demo_codec_SUITE) consistently supports that:

encode_json Encoded 240184 bytes in 2141us.
decode_json Decoded 240184 bytes in 270us.
encode_external Encoded 240625 bytes in 68us.
decode_external Decoded 240625 bytes in 121us.

The idea with content negotiation is to use the Accept-Encoding: header to advertise availability and Content-Encoding: to declare it’s use. This allows us to support native Erlang term exchange on any client or server, without impacting any other use cases. I implemented a test suite (ecc_demo_http_SUITE) with inets httpd/httpc which works but I’m still thinking about how to get meaningful performance metrics.

I’m considering authoring an IETF draft next to move towards getting an IANA registry assignment (erlang?).

2

+1 for IANA registration! It would probably be good to land this first: Add non_executable option to binary_to_term/2 by josevalim · Pull Request #11094 · erlang/otp · GitHub .

For some prior art, on hex.pm API we use:

json by default:

1> inets:start(), ssl: start().
2> {ok, {{_, 200, _}, _, Body}} = httpc:request(get, {"https://hex.pm/api/packages", [{"user-agent", "httpc"}]}, [], [{body_format, binary}]).
3> Body.
<<"[{\"meta\":{\"links\":{},\"description\":\"Test package\",\"licenses\":
[\"MIT\"],\"maintainers\":[]},\"name\":\"a_\",\"url\":\"https://he"...>>

but also allow: accept: application/vnd.hex+erlang:

1> inets:start(), ssl: start().
2> {ok, {{_, 200, _}, _, Body}} = httpc:request(get, {"https://hex.pm/api/packages", [{"user-agent", "httpc"}, {"accept", "application/vnd.hex+erlang"}]}, [], [{body_format, binary}]).
3> erlang:binary_to_term(Body).
[#{<<"configs">> =>#{<<"erlang.mk">> => <<"dep_a = hex 0.22.0">>,<<"mix.exs">> => <<"{:a, "~> 0.22.0"}">>,<<"rebar.config">> => <<"{a, "0.22.0"}">>},
3

Are you sure you want Accept-Encoding/Content-Encoding as opposed to Accept/Content-Type?

application/erlang or application/etf would be similar to application/json

4

Yes, for me what I am wanting is an encoding choice (only).

There’s room for debate here, however we can start with the premise that media types are a high level concern. The Content-Type header value helps us choose a software application to handle a payload: text/html, image/svg+xml, application/atom+xml, application/vnd.oasis.opendocument.spreadsheet, text/csv, text/calendar, application/vcard+xml. Those values tells us whether we should use a browser, spreadsheet or calendar application respectively.

The same RSS reader would handle application/atom+xml or application/atom+json as those are exactly the same information model, just different data coding formats.

I just added XML handling to ecc_demo to help generalize the content negotiation concept. The CODEC test suite now compares external term format with text encoded by xmerl and json. I added a module for generating Atom (RFC4287) sample data as xmerl:xmlElement() terms and use that in the httpd callback. I could use the media type “structured suffix”, +xml or +json, to select a CODEC, xmerl or json.

An optimization choice to use Erlang’s external term format is not at the same level as the media type declared with the Content-Type header. The ecc_demo project demonstrates this with one clause selecting a CODEC (json:encode|decode or erlang:term_to_iovec|term_to_binary) based on Content-Encoding, and all the application handling stays the same.

Where there’s room for debate is with the selection of CODEC implementation. For my main use case of REST APIs we mostly use Content-Type: application/json where JSON (RFC8259) is clearly defined as a textual encoding. When we include Content-Encoding: erlang we are abstracting the media type to a logical information model and choosing an Erlang term format for the encoding. Here we may assume //stdlib/json:encode_value() however that’s only because that is part of stdlib now. For XML we may assume //xmerl/xmerl:xmlElement() because that’s also part of OTP, however if the protocol is not supported by OTP we are left to a priori understanding between consumer and producer. A mechanism to explicitly indicate the CODEC implementation would be more satisfying.

An argument could be made for a structured suffix of +erlang in the media type however in the Atom example how would we also communicate +json/+xml?

5

It would be good to have it standardized. Some years ago I used the external term format in backwater, an RPC library for the BEAM. I picked Content-Type: application/x-erlang-etf, paired with optional gzip encoding. It worked very well.