Do you make use of term_to_binary?

[AstonJ]

Joe asked this on EF a few years ago:

@joeerl

I may be wrong but I don’t seem much use of term_to_binary in Elixir - this is the great way to store anything on disk or communicate between systems. Is this a well know way of storing and retrieving data to disk?

Wondering whether people make use of it in Erlang - do you use it?

Here’s what he had to say about it:

@joeerl

You have to remember that any SQL database will be a bottleneck since there is an impedance mismatch between the way dayt is represented in a daybase and the way it is represented in the beam VM.

Databases are basically rectangular tables of cells, where the cells contain very simple types like strings and integers - every time you access a row of an external database this list of cells has to be converted to beam internal data structures - this conversion is extremely expensive.

The best way to persist data is in a process - then no conversion is needed, but this is not fault tolerant - so you need to keep a trail of updates to the data and store this on disk.

Often you don’t need a database for example you might like to have a system where you store all the user data as in the file system with “one file per user” this will scale very nicely - just move the files to a new machine if you need more capacity.

Erlang has two primitives term_to_binary and the inverse binary_to_term that serialise any term and reconstruct it - so storing complex terms on disk is really easy.

I have mixed feelings about databases, they are great for aggregate operations (for example, find all users that have these attributes) but terrible for operations on individual users (where a single file per user is far better).

If I were designing a new system I’d go for ‘one file per user’ as much as possible and try to limit databases for operations over all users.

If you look at how many programs are designed you’ll see they follow this principle. Apple stores all images in the file system (hidden a bit) and has a database with metadata about the files. This is good since the database is small and many operations can be performed with minimal use of the database. What they do not do is put all the data in a database - there are good reasons for this.

[dominic]

When I’m going to use term_to_binary , first thing out of my mind is dets and there a question too

How to backup a live dets?

I think that I can’t just copy the dets file or close dets then copy file to finish the job If I have to copy every data in erlang node, read from live dets and write to backup dets BTW, If anyone using dets as data backend in production And I just need a kv disk store, If mnesia is a better choice

@joeerl

I’d go for ‘one file per user’ as much as possible

Great idea!

Finally, I need term_to_binary

[kjnilsson]

Not anymore so much. Mostly use term_to_iovec/1 now

[mworrell]

In Zotonic we are using term_to_binary regularly to serialize arbitrary data structures. For backing stores, but also for data that is send along user requests (we are adding signatures for those).

[caravan_muffin]

It is a great way to “pass by reference” if copying messages between processes becomes a botlleneck. Admittedly with some overhead.

For example, you may have a pool of workers and a dispatcher, and a stream of relatively large messages coming in. In a naive implementation your message will be copied into the dispatcher only to be immediately copied again into one of the workers. If you, instead, transform the message into a binary all you copy between the processes is a reference. You just need to leave enough of a message as a term in order to dispatch it appropriately.

[lpil]

Oh that’s clever! Thanks for the tip

[eproxus]

Also, base64:encode(term_to_binary(Anything)) is a great way to piggyback Erlang things over almost any (text-based or string capable) protocol.

For example, we had a need to ship Prometheus metrics from IoT devices in the field, which already had an existing semi-persistent web socket channel open talking JSON-RPC. What did we do? Exported the data from the Prometheus library on device, base64-encoded it and shipped it in a JSON field in a web socket packet. Then we just unpacked it on the server side and sent it to Prometheus when it was scraping for metrics.

[nickva]

Apache CouchDB uses term_to_binary to store all its data on disk.

[rvirding]

The only thing you have to be aware of here is that when you want to inspect data then you have to unpack the binary into a copy of its original data. So while you might save copying the actual message you still have to make a copy when you view the data. I personally think it is probably more efficient to let the BEAM copy instead of first making the binary then unpacking it every process.

However one situation where it is more efficient is when you want to pass the message through a chain of processes who don’t need to unpack just pass it on. E.g when processing pictures or video.

[starbelly]

I’m interested in this. I saw that rmq is using for writing to files, but did you do any benchmarks outside of that scope?

I did some recent benchmarks and while the tool I used may be folly (need to try with erlperf and making my own test devoid of those tools), but I found term_to_iovec was slower than term_to_binary, which was not expected, but perhaps my expectation is wrong. Tests using term_to_iovec writing to files with a raw file handle were faster, but not significantly faster.

[starbelly]

Quick follow up, bench marking with erlperf I found that term_to_iovec was on average 9% faster