Hex and/or base64 sigils for string literals

Erlang/OTP Proposals Proposals: Ideas
1

I have a proposal for string literals; I’ve found myself needing to embed some binary data into the code occasionally (e.g. a cryptographic key into the config), plus in want of a decent way to serialize binaries that’s relatively human-readable (or at least: copyable into other tools).

The “usual” way to do this is to make it <<249,151,173,246,...>> — which isn’t ideal, as binary data is commonly represented via hex or base64, decimal is more or less an Erlang-specific rarity; or a string that’s then decoded at runtime (I guess on launch), e.g. "f997adf6" (hex) or "+Zet9g==" (base64).

There is sort-of a way, in doing <<"\xf9\x97...">>, but that’s awkward for pasting from other tools, plus it adds syntactic noise, making it harder to read.

Thus, I have two proposals, ostensibly ortogonal; the former of which I’m more certain about:

1. Add a ~x"..." sigil for hex strings

I’d imagine this would mostly be used for binaries, something like ~bx, unless we make ~x itself give us a binary. My examples will assume the latter (~x is binary), for brevity.

Above example would then become ~x"f997adf6", which would be exactly equivalent to <<"\xf9\x97\xad\xf6">> or <<249,151,173,246>>.

I think it makes sense for such strings to ignore whitespace (including newlines), but enforce an even number of hex digits. And for it to be case-insensitive.

So the following examples are valid (and equivalent):

~x"f997adf6"
~x"F997ADF6"
~x"f 99 7aD F6"
~x"""
F99 7
adf
6
"""

… and the following are invalid: (N.B.: I could see an argument for allowing ‘-’ in these for the sake of UUIDs, but I think it’s better off if it’s not allowed; YMMV).

~x"f99"      % ERROR: un-even number of hex digits
~x"f997-ad"  % ERROR: stray '-' in hex string/binary literal
~x"\xf9"     % same as above, but with a stray '\' (there is no escaping of these strings)

A big part of the motivation for me is being able to serialize (non-text) binaries in a way that’s relatively human-readable and also a valid Erlang term. E.g. UUIDs as ~x"26b85f7a ea9c 4a2f a495 1f79c6e92ff5" (basically the ISO format with spaces instead of dashes); or hashed passwords as (say) {argon2,<...params...>,~x"<...salt...>",~x"<...hash...>"}.

2. Add a ~b64 sigil for base64 data

Same idea as above, except base64. I think it makes even more sense to support whitespace/newlines here (think pasting a .pem’s multi-line certificate contents or similar, directly).

I’d allow, but not require, padding characters (=); though if present, I’d enforce the correct number of them (so: either no padding or the correct padding for the length).

In terms of dialect, it’d be pretty easy to support both the “classic” Base64 as well as the URL variant (via aliasing of the symbols for digits 62 & 63), though I’d disallow “mixing” of them in the same string in this case. If we’re only doing one, I think the classic one makes more sense.

A few examples (the valid ones are equivalent):

~b64"L6/va06PT1Srjg1lu+nXWg=="  % OK, have padding, and it's valid
~b64"L6/va06PT1Srjg1lu+nXWg"    % OK, no padding
~b64"L6_va06PT1Srjg1lu-nXWg=="  % OK, URL dialect, have padding
~b64"L6_va06PT1Srjg1lu-nXWg"    % OK, URL dialect, no padding
~b64"""
L6/v a06P
T1Sr jg1l
u+nX Wg
"""  % OK, multi-line + whitesppace

~b64"L6/va06PT1Srjg1lu+nXWg="   % ERROR, invalid padding
~b64"L6/va06PT1Srjg1lu-nXWg"    % ERROR, mixing non-URL and URL dialects
~b64"L6/?a06PT1Srjg1lu-nXWg"    % ERROR, stray '?' in base64 value

Admittedly, I need this one less, but I figured I’d bring it up since it’s closely related to the hex proposal, and it makes a lot of sense for some use-cases.

1 Like
2

It’s an interesting proposal, but I’m not sure cryptographic keys are the best example. Private and symmetric keys normally wouldn’t be compiled in anyway. You’d load them from secrets at runtime, where binary:decode_hex/1 works just fine.

As for compile time, it’s not necessarily as ugly as <<"\xf9\x97\xad\xf6">>. It can be a bit easier to read as <<16#f997adf6:32>>, but yeah, I know, you have to “count” the digits to make sure :32 is what you want :grinning_face:

3

That’s a fair point on the example, though I mostly had public keys in mind anyway.

But the entirety of Base64 is (IMO) less interesting & more niche than the hex one. I just figured I might as well bring it up as another option.

On your <<16#f997adf6:32>> example, I know that it can be done that way, but the whole :32 feels like an error waiting to happen.

Plus, in the case of (for whatever reason) parsing a term at runtime, I’m not sure if this will do the “right thing” of producing the binary directly, or if it’ll build a N-bit (for a potentially huge N!) integer and then convert/serialize that to a binary. Unlike string examples, where I’m more confident that it’ll be done directly. I know it usually doesn’t matter, but it just feels wrong to create a (say) 8192-bit integer just to handle a kB of data.

Either way, I do feel like it would cover some legitimate use-cases, especially considering that ~x"..." covers my niche of “Erlang-parseable, but relatively human-friendly output of binary data”.

1 Like
4

Yes, I agree.

5

The only reason for making these things literals is if you want to pattern match against them. The set of binary encodings is not closed. It keeps expanding. Do we really want a crowd of delirious sigils? How many is enough?

A much simpler approach is to have a decode: module with a function for each of the dozens of encodings around and specify that calls with arguments known at compile time must be evaluated. Adding yet another encoding is just adding another function to the module. No change to lexing or parsing tools or editor support.