# Adding atoms to the definition of IO data?

**URL:** <https://erlangforums.com/t/adding-atoms-to-the-definition-of-io-data/3982>\
**Category:** Chat / Discussions\
**Tags:** atoms, iolist, iodata\
**Created:** [September 5, 2024, 10:24am UTC](https://erlangforums.com/t/adding-atoms-to-the-definition-of-io-data/3982 "2024-09-05T10:24:40Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![eproxus](https://erlangforums.com/user_avatar/erlangforums.com/eproxus/32/452_2.png) [@eproxus](https://erlangforums.com/u/eproxus)\
**Post date:** [September 5, 2024, 10:24am UTC](https://erlangforums.com/t/adding-atoms-to-the-definition-of-io-data/3982/1 "2024-09-05T10:24:40Z")

</div>

I often want to render IO data where the source data contains atoms. It would be super nice if atoms were automatically converted to binaries when used in e.g. `iolist_to_binary/1` and other IO functions.

I’m thinking of cases like these:

```erlang
callback_url(Type) when is_atom(Type) ->
    iolist_to_binary([config(callback_root), $/, atom_to_binary(Type)]).
    % Which could become:
    iolist_to_binary([config(callback_root), $/, Type]).

```

(Ignore the fact that this might not be the cleanest way to join URI components, that is besides the point).

I don’t immediately see a problem with with it since it would only be at “encoding” time (and not “decoding” time which is another can of worms). It would just be a convenience for things that already looks like text to developers, e.g. `iolist_to_binary([foo, $., bar, $. baz])` would generate `<<"foo.bar.baz">>`. Then one could use `iodata()` structures as a poor mans templating system as far as atoms are concerned.

What are your opinions? Any reasons why this would be a bad idea? Are there cases where it would cause problems or not be feasible?

---

<div class="post-metadata">

**Author:** ![williamthome](https://erlangforums.com/user_avatar/erlangforums.com/williamthome/32/471_2.png) [@williamthome](https://erlangforums.com/u/williamthome)\
**Post date:** [September 5, 2024, 10:53am UTC](https://erlangforums.com/t/adding-atoms-to-the-definition-of-io-data/3982/2 "2024-09-05T10:53:36Z")

</div>

I’m not sure why, but [lists:concat/1](https://www.erlang.org/doc/apps/stdlib/lists.html#concat/1) seems to have a similar behavior:

```erlang
1> lists:concat([foo, ".", bar, ".", baz]).
"foo.bar.baz"

```

But by using char it does not return the expected result:

```erlang
2> lists:concat([foo, $., bar, $., baz]).
"foo46bar46baz"

```

---

<div class="post-metadata">

**Author:** ![eproxus](https://erlangforums.com/user_avatar/erlangforums.com/eproxus/32/452_2.png) [@eproxus](https://erlangforums.com/u/eproxus)\
**Post date:** [September 5, 2024, 10:58am UTC](https://erlangforums.com/t/adding-atoms-to-the-definition-of-io-data/3982/3 "2024-09-05T10:58:15Z")

</div>

Interesting find. It even supports integers and floats too. 🤔

IO data is kind of ideal, because in the end one does not have to concatenate or flatten anything, so supporting atoms there would be the most performant choice.

It would be impossible to support integers since those are already used for code points, but personally I don’t want that anyway because there usually isn’t a good 1:1 mapping between an integer/float and the desired string representation since it usually varies (precision, thousand separator etc.). But for atoms if you really want them as a string by default it is easy, since they are their own string representation (there’s just no automatic conversion for it, which is what I wish for with this post 😄).

---

<div class="post-metadata">

**Author:** ![LeonardB](https://erlangforums.com/user_avatar/erlangforums.com/leonardb/32/1856_2.png) [@LeonardB](https://erlangforums.com/u/LeonardB)\
**Post date:** [September 5, 2024, 1:06pm UTC](https://erlangforums.com/t/adding-atoms-to-the-definition-of-io-data/3982/4 "2024-09-05T13:06:49Z")

</div>

This is because `$.` is a representation of 46, not a `list`, which  
would be `[$.]` or `"."`.

```erlang
1> lists:concat([foo, [$.], bar, [$.], baz]).
"foo.bar.baz"
2> 46 =:= $..
true

```

The implementation is:

```erlang
integer() -> list().
float() -> list().
atom() -> list().
list() -> list().

```

Think there’s there’s a typo/bug in the documentation.

```erlang
%% concat(L) concatenate the list representation of the elements
%% in L - the elements in L can be atoms, numbers of strings.
%% Returns a list of characters.

```

It should probably be `numbers or strings`

---

<div class="post-metadata">

**Author:** ![eproxus](https://erlangforums.com/user_avatar/erlangforums.com/eproxus/32/452_2.png) [@eproxus](https://erlangforums.com/u/eproxus)\
**Post date:** [September 5, 2024, 1:49pm UTC](https://erlangforums.com/t/adding-atoms-to-the-definition-of-io-data/3982/5 "2024-09-05T13:49:36Z")

</div>

From the documentation it also looks like `lists:concat/1` only handles flat lists so it’s not a replacement for deeply nested IO list creation, which is very efficient.

---

<div class="post-metadata">

**Author:** ![garazdawi](https://erlangforums.com/user_avatar/erlangforums.com/garazdawi/32/175_2.png) [@garazdawi](https://erlangforums.com/u/garazdawi)\
**Post date:** [September 6, 2024, 7:25am UTC](https://erlangforums.com/t/adding-atoms-to-the-definition-of-io-data/3982/6 "2024-09-06T07:25:26Z")

</div>

An `iolist` is a collection of bytes, not a collection of characters, so I don’t think that it would make sense to include something that would need to be converted to characters in there.

If we were to add it anywhere, then I would then add it to `unicode:chardata/0`, as that is a collection of characters where we have a known encoding to convert to. You could then to `unicode:characters_to_list(["hello ", atom])` and get what you expect. I don’t think that we should do this, as dealing with chardata is complex enough in places like the `string` module.

As a small aside, the [`file:name/0`](https://www.erlang.org/doc/apps/kernel/file.html#t:name/0) type is already almost what you describe that you want 🙂

---

<div class="post-metadata">

**Author:** ![eproxus](https://erlangforums.com/user_avatar/erlangforums.com/eproxus/32/452_2.png) [@eproxus](https://erlangforums.com/u/eproxus)\
**Post date:** [September 7, 2024, 1:09pm UTC](https://erlangforums.com/t/adding-atoms-to-the-definition-of-io-data/3982/7 "2024-09-07T13:09:52Z")

</div>

> [@garazdawi](#):
>
> An `iolist` is a collection of bytes, not a collection of characters, so I don’t think that it would make sense to include something that would need to be converted to characters in there.

Right, so with that explanation, I realize that what I’m really asking for is a _built-in, singe-pass, recursive way to encode (some) terms to UTF-8_… I suppose.

That is, something like `foobar_to_binary/1,2` where `..._to_binary` has the same meaning as in `atom_to_binary/1,2` (since it _is_ encoding aware) and `foobar_to_...` has the same recursive meaning and flexibility as `iolist_to_...`.

> [@garazdawi](#):
>
> If we were to add it anywhere, then I would then add it to `unicode:chardata/0`, as that is a collection of characters where we have a known encoding to convert to. You could then to `unicode:characters_to_list(["hello ", atom])` and get what you expect. I don’t think that we should do this, as dealing with chardata is complex enough in places like the `string` module.
> 
> As a small aside, the [`file:name/0`](https://www.erlang.org/doc/apps/kernel/file.html#t:name/0) type is already almost what you describe that you want 🙂

If I understand you correctly what you are saying is that `iolist_to_binary` is purely a way to efficiently traverse BEAM memory (from various sources) and pipe it into an IO sink (socket, binary, file, whatever) and that due to complexity of encoding it will not get any support for exporting such data to the configured encoding of the VM (in the way `atom_to_binary` does).

The other alternatives are less interesting because they remove the (most) interesting property of `iolist_to_binary` (or whatever underlying mechanism is used that works with sockets/files etc.) in that it can be done in a single pass by the VM itself. The multiple pass way is what any Erlang project already uses today, either explicitly (homegrow recursive algorithm _before_ the IO sink) or implicitly (via some library that does it wholly or partly, e.g. the new `json` module in OTP).

How are atom text representations stored internally in the BEAM? As UTF-8 binaries somewhere?

---

<div class="post-metadata">

**Author:** ![garazdawi](https://erlangforums.com/user_avatar/erlangforums.com/garazdawi/32/175_2.png) [@garazdawi](https://erlangforums.com/u/garazdawi)\
**Post date:** [September 9, 2024, 8:17am UTC](https://erlangforums.com/t/adding-atoms-to-the-definition-of-io-data/3982/8 "2024-09-09T08:17:35Z")

</div>

What you want is some simpler way to convert atoms to strings without having to litter the code with `s(Atom)` or adding some extra pass. This is very similar to the problem that [EEP-62](https://www.erlang.org/eeps/eep-0062) tries to solve, only that it does it for more datatypes than just atoms. If EEP-62 were to be introduced, would this make adding atoms to `iodata()` still useful? or are they two different solutions to the same problem?

> How are atom text representations stored internally in the BEAM? As UTF-8 binaries somewhere?

yes.

---

<div class="post-metadata">

**Author:** ![eproxus](https://erlangforums.com/user_avatar/erlangforums.com/eproxus/32/452_2.png) [@eproxus](https://erlangforums.com/u/eproxus)\
**Post date:** [September 9, 2024, 9:02am UTC](https://erlangforums.com/t/adding-atoms-to-the-definition-of-io-data/3982/9 "2024-09-09T09:02:53Z")

</div>

The property I would like to keep is to be able to recursively create an IO data structure without flattening it more than once (which happens for free when you pass an IO list to some output sink). Interpolating it myself, with some library or with EEP-62 doesn’t really matter that much because it is still one additional (unnecessary) pass over the structure. If the EEP-62 syntax would be compiled to something that retains the IO data properties, then that of course would solve the problem.

> [@garazdawi](#):
>
> > How are atom text representations stored internally in the BEAM? As UTF-8 binaries somewhere?
> 
> yes.

Right, so in my naive dream world I’d like for something like this to work: `iolist_to_binary(["foo", <<"bar">>, baz], utf8)` 😃 where the atoms would just be piped/encoded automagically (and where the default option would of course be `utf8` as with `atom_to_binary/1`).

---

<div class="post-metadata">

**Author:** ![nzok](https://erlangforums.com/letter_avatar_proxy/v4/letter/n/4af34b/32.png) [@nzok](https://erlangforums.com/u/nzok)\
**Post date:** [September 10, 2024, 1:28am UTC](https://erlangforums.com/t/adding-atoms-to-the-definition-of-io-data/3982/10 "2024-09-10T01:28:50Z")

</div>

It’s difficult to see how EEP-62 would help.  
Suppose you want to include an atom in an IO list.  
There are two cases.  
(1) You have a specific atom in mind.  
In this case, you just turn ‘X!&Z’ into “X!&Z” and there’s no problem.  
(2) The atom will not be known until run-time.  
In this case, EEP-62 is a rather heavy-weight alternative to atom\_to\_list(Atom).  
which could be given a local abbreviation if used often.

This turns our focus to “where are the atoms coming from, and why are they atoms?”

I am actually very sympathetic to the idea of allowing atoms in I/O lists.  
I have a library for another programming language in which its analogues of  
I/O lists (called chartrees and bytetrees) DO allow atoms. If Erlang had  
happened to allow this way back when, I doubt that anyone would have argued for  
removing that feature when making Erlang Unicode-friendly.

Since the feature _isn’t_ present in Erlang (yet), a useful response is to say  
“what does the code look like where this would be helpful?”

Back in the day there used to be a joke about New Zealand.  
When the USA came up with a new educational idea,  
we’d wait 10 years until it was proven to be a very bad idea.  
And _then_ we would adopt it.  
EEP-62 reminds me forcibly of that joke.

---

<div class="post-metadata">

**Author:** ![maxlapshin](https://erlangforums.com/user_avatar/erlangforums.com/maxlapshin/32/1593_2.png) [@maxlapshin](https://erlangforums.com/u/maxlapshin)\
**Post date:** [September 11, 2024, 7:15am UTC](https://erlangforums.com/t/adding-atoms-to-the-definition-of-io-data/3982/11 "2024-09-11T07:15:56Z")

</div>

this is a nice idea if not to consider that `undefined` is also an atom.

It may be really convenient to write some atoms like key names to binary, but `undefined` will become a pain I suppose.

However, the reason to write iolist due to atom may be a bit easier to read.

---

<div class="post-metadata">

**Author:** ![mmin](https://erlangforums.com/user_avatar/erlangforums.com/mmin/32/2193_2.png) [@mmin](https://erlangforums.com/u/mmin)\
**Post date:** [September 11, 2024, 8:42am UTC](https://erlangforums.com/t/adding-atoms-to-the-definition-of-io-data/3982/12 "2024-09-11T08:42:31Z")

</div>

> [@maxlapshin](#):
>
> this is a nice idea if not to consider that `undefined` is also an atom.

Any reason why `undefined` is so special in this case? I don’t see why `undefined` should be written as `"undefined"` by default. If you want to change that behavior you need to manually handle it anyways.

---

<div class="post-metadata">

**Author:** ![maxlapshin](https://erlangforums.com/user_avatar/erlangforums.com/maxlapshin/32/1593_2.png) [@maxlapshin](https://erlangforums.com/u/maxlapshin)\
**Post date:** [September 13, 2024, 8:03am UTC](https://erlangforums.com/t/adding-atoms-to-the-definition-of-io-data/3982/13 "2024-09-13T08:03:34Z")

</div>

`undefined` is special because it is our NULL.

When I write something like `io:write(Fd, ['{',user,':', escape(UserId)])` it is rather clear that I assume that atom `user` will be written as 4-byte binary \<\<“user”\>\>. But `undefined` usually means that it is not what I wanted to write.

---

<div class="post-metadata">

**Author:** ![jhogberg](https://erlangforums.com/user_avatar/erlangforums.com/jhogberg/32/179_2.png) [@jhogberg](https://erlangforums.com/u/jhogberg)\
**Post date:** [September 13, 2024, 8:57am UTC](https://erlangforums.com/t/adding-atoms-to-the-definition-of-io-data/3982/14 "2024-09-13T08:57:11Z")

</div>

I think calling `undefined` our `NULL` is stretching it: sure, record fields that are left unset happen to be set to `undefined`, but practically everywhere else we let absence signal absence, and in the rare cases where sentinel values are used, they’re pretty inconsistent.

---

<div class="post-metadata">

**Author:** ![jhogberg](https://erlangforums.com/user_avatar/erlangforums.com/jhogberg/32/179_2.png) [@jhogberg](https://erlangforums.com/u/jhogberg)\
**Post date:** [September 13, 2024, 9:14am UTC](https://erlangforums.com/t/adding-atoms-to-the-definition-of-io-data/3982/15 "2024-09-13T09:14:47Z")

</div>

Here’s a quick and dirty implementation in case anyone wants to play around with it. As supporting Latin-1 would be annoying in the face of yielding, the encoding is always UTF-8.

(I don’t have a strong opinion on this, but am leaning a bit towards not having it since I think it’s more likely to hide bugs than to help)

> **[GitHub - jhogberg/otp at john/erts/atoms-in-iolists](https://github.com/jhogberg/otp/tree/john/erts/atoms-in-iolists)**
>
> john/erts/atoms-in-iolists

---

<div class="post-metadata">

**Author:** ![mmin](https://erlangforums.com/user_avatar/erlangforums.com/mmin/32/2193_2.png) [@mmin](https://erlangforums.com/u/mmin)\
**Post date:** [September 13, 2024, 9:18am UTC](https://erlangforums.com/t/adding-atoms-to-the-definition-of-io-data/3982/16 "2024-09-13T09:18:07Z")

</div>

> [@maxlapshin](#):
>
> `undefined` is special because it is our NULL.

So could be `'null'`, `'nil'` or `'none'`

> [@maxlapshin](#):
>
> But `undefined` usually means that it is not what I wanted to write.

I’d argue that, but no matter what `undefined` means, currently you still have to write something like:

```erlang
maybe_write(undefined) ->
<<>>;
maybe_write(X) ->
X.

```

if you want to output nothing on undefined. If atoms get into io\_data then you still can do it, **BUT** you don’t need to manually convert every atom to its binary representation. I don’t see `undefined` causing a single problem here

---

<div class="post-metadata">

**Author:** ![wojtekmach](https://erlangforums.com/user_avatar/erlangforums.com/wojtekmach/32/1164_2.png) [@wojtekmach](https://erlangforums.com/u/wojtekmach)\
**Post date:** [September 13, 2024, 9:43am UTC](https://erlangforums.com/t/adding-atoms-to-the-definition-of-io-data/3982/17 "2024-09-13T09:43:37Z")

</div>

I sometimes wished atoms would be handled like proposed here but speaking of `undefined`, I think the following would definitely trip up Elixir developers. In Elixir, nil is handled in many places, including interpolations, here’s a contrived example:

```erlang
iex> "foo#{nil}#{true}#{:bar}"
"footruebar"

```

I’d sometimes change interpolation to iolists to avoid copies but now I’d get a different result:

```erlang
iex> ["foo", nil, true, :bar] |> IO.iodata_to_binary()
"fooniltruebar"

```

---

<div class="post-metadata">

**Author:** ![eproxus](https://erlangforums.com/user_avatar/erlangforums.com/eproxus/32/452_2.png) [@eproxus](https://erlangforums.com/u/eproxus)\
**Post date:** [September 13, 2024, 1:18pm UTC](https://erlangforums.com/t/adding-atoms-to-the-definition-of-io-data/3982/18 "2024-09-13T13:18:36Z")

</div>

For generering strings from atoms `undefined` has no special meaning in general (just as `true` and `false` doesn’t). The principle I want applied here is to automatically generate the _text representation of atoms_ so I don’t have to do it manually everywhere recursively.

If atom has special meaning in some special context, one has to use the right formatting or library (which would be the `json` module in this example, as it looks like you want to encode JSON).

---

<div class="post-metadata">

**Author:** ![williamthome](https://erlangforums.com/user_avatar/erlangforums.com/williamthome/32/471_2.png) [@williamthome](https://erlangforums.com/u/williamthome)\
**Post date:** [September 13, 2024, 5:11pm UTC](https://erlangforums.com/t/adding-atoms-to-the-definition-of-io-data/3982/19 "2024-09-13T17:11:20Z")

</div>

Compared with javascript, and maybe javascript is not a parameter in that case, but what it returns for `undefined` or `null` is:

```erlang
> `foo${undefined}`
'fooundefined'
> `foo${null}`
'foonull'

```

There is no special treatment for them.

---

<div class="post-metadata">

**Author:** ![nzok](https://erlangforums.com/letter_avatar_proxy/v4/letter/n/4af34b/32.png) [@nzok](https://erlangforums.com/u/nzok)\
**Post date:** [September 16, 2024, 11:19pm UTC](https://erlangforums.com/t/adding-atoms-to-the-definition-of-io-data/3982/20 "2024-09-16T23:19:18Z")

</div>

I find the idea of “foo#{nil}#{true}#{:bar}” becoming “footruebar”  
rather shocking.  
If I used Elixir, and were sufficiently unwary as to use string  
interpolation – which is very nearly as bad an idea as the  
“billion-dollar mistake” – I can well imagine falling foul of that.

If atoms are ever allowed in I/O lists, there should be _no_  
exceptions. Every atom should be mapped to its characters.  
Special cases are the bane of programming.

[Next page](https://erlangforums.com/t/adding-atoms-to-the-definition-of-io-data/3982.md?page=2)