# Do you make use of term\_to\_binary? **URL:** \ **Category:** Chat / Discussions\ **Tags:** joe-armstrong, term\_to\_binary, disk, storage\ **Created:** [February 28, 2023, 10:36pm UTC](https://erlangforums.com/t/do-you-make-use-of-term-to-binary/2416 "2023-02-28T22:36:03Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![AstonJ](https://erlangforums.com/user_avatar/erlangforums.com/astonj/32/9_2.png) [@AstonJ](https://erlangforums.com/u/AstonJ)\ **Post date:** [February 28, 2023, 10:36pm UTC](https://erlangforums.com/t/do-you-make-use-of-term-to-binary/2416/1 "2023-02-28T22:36:03Z")
Joe [asked this](https://elixirforum.com/t/acceptance-of-erlangs-term-to-binary-and-vice-versa-in-elixir/11034) on EF a few years ago: > [@joeerl](#): > > I may be wrong but I don’t seem much use of term\_to\_binary in Elixir - this is _the great way to store anything on disk_ or communicate between systems. Is this a well know way of storing and retrieving data to disk? Wondering whether people make use of it in Erlang - do you use it? Here’s what he had to say about it: > [@joeerl](#): > > You have to remember that any SQL database will be a bottleneck since there is an impedance mismatch between the way dayt is represented in a daybase and the way it is represented in the beam VM. > > Databases are basically rectangular tables of cells, where the cells contain very simple types like strings and integers - every time you access a row of an external database this list of cells has to be converted to beam internal data structures - this conversion is extremely expensive. > > The best way to persist data is in a process - then no conversion is needed, but this is not fault tolerant - so you need to keep a trail of updates to the data and store this on disk. > > Often you don’t need a database for example you might like to have a system where you store all the user data as in the file system with “one file per user” this will scale very nicely - just move the files to a new machine if you need more capacity. > > Erlang has two primitives term\_to\_binary and the inverse binary\_to\_term that serialise any term and reconstruct it - so storing complex terms on disk is really easy. > > I have mixed feelings about databases, they are great for aggregate operations (for example, find all users that have these attributes) but terrible for operations on individual users (where a single file per user is far better). > > If I were designing a new system I’d go for ‘one file per user’ as much as possible and try to limit databases for operations over all users. > > If you look at how many programs are designed you’ll see they follow this principle. Apple stores all images in the file system (hidden a bit) and has a database with metadata about the files. This is good since the database is small and many operations can be performed with minimal use of the database. What they do not do is put all the data in a database - there are good reasons for this. ---
**Author:** ![dominic](https://erlangforums.com/letter_avatar_proxy/v4/letter/d/13edae/32.png) [@dominic](https://erlangforums.com/u/dominic)\ **Post date:** [March 1, 2023, 6:23am UTC](https://erlangforums.com/t/do-you-make-use-of-term-to-binary/2416/2 "2023-03-01T06:23:25Z")
When I’m going to use `term_to_binary `, first thing out of my mind is `dets` and there a question too > [@How to backup a live dets?](https://erlangforums.com/t/how-to-backup-a-live-dets/2079): > > I think that I can’t just copy the dets file or close dets then copy file to finish the job If I have to copy every data in erlang node, read from live dets and write to backup dets BTW, If anyone using dets as data backend in production And I just need a kv disk store, If mnesia is a better choice > [@joeerl](#): > > I’d go for ‘one file per user’ as much as possible Great idea! Finally, I need `term_to_binary ` ---
**Author:** ![kjnilsson](https://erlangforums.com/user_avatar/erlangforums.com/kjnilsson/32/450_2.png) [@kjnilsson](https://erlangforums.com/u/kjnilsson)\ **Post date:** [March 1, 2023, 11:37am UTC](https://erlangforums.com/t/do-you-make-use-of-term-to-binary/2416/3 "2023-03-01T11:37:48Z")
Not anymore so much. Mostly use `term_to_iovec/1` now 🙂 ---
**Author:** ![mworrell](https://erlangforums.com/user_avatar/erlangforums.com/mworrell/32/267_2.png) [@mworrell](https://erlangforums.com/u/mworrell)\ **Post date:** [March 1, 2023, 12:08pm UTC](https://erlangforums.com/t/do-you-make-use-of-term-to-binary/2416/4 "2023-03-01T12:08:13Z")
In Zotonic we are using `term_to_binary` regularly to serialize arbitrary data structures. For backing stores, but also for data that is send along user requests (we are adding signatures for those). ---
**Author:** ![caravan\_muffin](https://erlangforums.com/letter_avatar_proxy/v4/letter/c/e79b87/32.png) [@caravan\_muffin](https://erlangforums.com/u/caravan_muffin)\ **Post date:** [March 1, 2023, 5:37pm UTC](https://erlangforums.com/t/do-you-make-use-of-term-to-binary/2416/5 "2023-03-01T17:37:59Z")
It is a great way to “pass by reference” if copying messages between processes becomes a botlleneck. Admittedly with some overhead. For example, you may have a pool of workers and a dispatcher, and a stream of relatively large messages coming in. In a naive implementation your message will be copied into the dispatcher only to be immediately copied again into one of the workers. If you, instead, transform the message into a binary all you copy between the processes is a reference. You just need to leave enough of a message as a term in order to dispatch it appropriately. ---
**Author:** ![lpil](https://erlangforums.com/user_avatar/erlangforums.com/lpil/32/222_2.png) [@lpil](https://erlangforums.com/u/lpil)\ **Post date:** [March 2, 2023, 8:32pm UTC](https://erlangforums.com/t/do-you-make-use-of-term-to-binary/2416/6 "2023-03-02T20:32:41Z")
Oh that’s clever! Thanks for the tip ---
**Author:** ![eproxus](https://erlangforums.com/user_avatar/erlangforums.com/eproxus/32/452_2.png) [@eproxus](https://erlangforums.com/u/eproxus)\ **Post date:** [March 3, 2023, 6:48pm UTC](https://erlangforums.com/t/do-you-make-use-of-term-to-binary/2416/7 "2023-03-03T18:48:35Z")
Also, `base64:encode(term_to_binary(Anything))` is a great way to piggyback Erlang things over almost any (text-based or string capable) protocol. For example, we had a need to ship Prometheus metrics from IoT devices in the field, which already had an existing semi-persistent web socket channel open talking JSON-RPC. What did we do? Exported the data from the Prometheus library on device, base64-encoded it and shipped it in a JSON field in a web socket packet. Then we just unpacked it on the server side and sent it to Prometheus when it was scraping for metrics. ---
**Author:** ![nickva](https://erlangforums.com/user_avatar/erlangforums.com/nickva/32/578_2.png) [@nickva](https://erlangforums.com/u/nickva)\ **Post date:** [March 3, 2023, 8:49pm UTC](https://erlangforums.com/t/do-you-make-use-of-term-to-binary/2416/8 "2023-03-03T20:49:40Z")
Apache CouchDB uses [term\_to\_binary](https://github.com/apache/couchdb/blob/main/src/couch/src/couch_compress.erl#L49-L52) to store all its data on disk. ---
**Author:** ![rvirding](https://erlangforums.com/user_avatar/erlangforums.com/rvirding/32/178_2.png) [@rvirding](https://erlangforums.com/u/rvirding)\ **Post date:** [March 18, 2023, 1:56pm UTC](https://erlangforums.com/t/do-you-make-use-of-term-to-binary/2416/9 "2023-03-18T13:56:45Z")
The only thing you have to be aware of here is that when you want to inspect data then you have to unpack the binary into a copy of its original data. So while you might save copying the actual message you still have to make a copy when you view the data. I personally think it is probably more efficient to let the BEAM copy instead of first making the binary then unpacking it every process. However one situation where it is more efficient is when you want to pass the message through a chain of processes who don’t need to unpack just pass it on. E.g when processing pictures or video. ---
**Author:** ![starbelly](https://erlangforums.com/user_avatar/erlangforums.com/starbelly/32/209_2.png) [@starbelly](https://erlangforums.com/u/starbelly)\ **Post date:** [September 24, 2023, 7:05pm UTC](https://erlangforums.com/t/do-you-make-use-of-term-to-binary/2416/10 "2023-09-24T19:05:04Z")
I’m interested in this. I saw that rmq is using for writing to files, but did you do any benchmarks outside of that scope? I did some recent benchmarks and while the tool I used may be folly (need to try with erlperf and making my own test devoid of those tools), but I found term\_to\_iovec was slower than term\_to\_binary, which was not expected, but perhaps my expectation is wrong. Tests using term\_to\_iovec writing to files with a raw file handle were faster, but not significantly faster. ---
**Author:** ![starbelly](https://erlangforums.com/user_avatar/erlangforums.com/starbelly/32/209_2.png) [@starbelly](https://erlangforums.com/u/starbelly)\ **Post date:** [September 24, 2023, 7:09pm UTC](https://erlangforums.com/t/do-you-make-use-of-term-to-binary/2416/11 "2023-09-24T19:09:40Z")
Quick follow up, bench marking with erlperf I found that term\_to\_iovec was on average 9% faster 🙂 ---
**Author:** ![SuddenGunter](https://erlangforums.com/user_avatar/erlangforums.com/suddengunter/32/2575_2.png) [@SuddenGunter](https://erlangforums.com/u/SuddenGunter)\ **Post date:** [September 18, 2024, 5:02pm UTC](https://erlangforums.com/t/do-you-make-use-of-term-to-binary/2416/12 "2024-09-18T17:02:24Z")
sorry for necroposting (not sure about this forum rules): we use term\_to\_binary for most of our kafka messages, switching to json for interop with other languages (though I recently started digging into that subject - I think we could parse that term\_to\_binary format in Go directly with some packages) ---
**Author:** ![kanga-roo](https://erlangforums.com/user_avatar/erlangforums.com/kanga-roo/32/1081_2.png) [@kanga-roo](https://erlangforums.com/u/kanga-roo)\ **Post date:** [September 24, 2024, 4:13am UTC](https://erlangforums.com/t/do-you-make-use-of-term-to-binary/2416/13 "2024-09-24T04:13:41Z")
Is ‘one file per user’ in this context to be in the format of .dat or .txt or …? ---
**Author:** ![Maria-12648430](https://erlangforums.com/user_avatar/erlangforums.com/maria-12648430/32/604_2.png) [@Maria-12648430](https://erlangforums.com/u/Maria-12648430)\ **Post date:** [September 24, 2024, 11:38am UTC](https://erlangforums.com/t/do-you-make-use-of-term-to-binary/2416/14 "2024-09-24T11:38:14Z")
You lost me… what do you mean? 🤔 ---
**Author:** ![rlipscombe](https://erlangforums.com/user_avatar/erlangforums.com/rlipscombe/32/459_2.png) [@rlipscombe](https://erlangforums.com/u/rlipscombe)\ **Post date:** [September 24, 2024, 2:59pm UTC](https://erlangforums.com/t/do-you-make-use-of-term-to-binary/2416/15 "2024-09-24T14:59:36Z")
> [@kanga-roo](#): > > Is ‘one file per user’ in this context to be in the format of .dat or .txt or …? > [@Maria-12648430](#): > > You lost me… what do you mean? 🤔 This is originally from Joe’s (quoted) text at the top. I assume that Joe meant that you could use `term_to_binary/1` to serialize each user and then write each to a separate file. This means that none of the files get too large, and you don’t need to worry about concurrent updates (as long as each user is treated serially). It’s basically Joe’s way of saying “shard by user”, I suspect. > [@kanga-roo](#): > > in the format of .dat or .txt or …? I don’t understand this, though. I’ll take a guess. Since `term_to_binary/1` returns binary data (see [External Term Format — erts v15.0.1](https://www.erlang.org/doc/apps/erts/erl_ext_dist.html) for details – I guess you know this already, @Maria-12648430), you’d probably want to name your files `.bin` or `.dat` (if that’s a convention you prefer). ---
**Author:** ![kanga-roo](https://erlangforums.com/user_avatar/erlangforums.com/kanga-roo/32/1081_2.png) [@kanga-roo](https://erlangforums.com/u/kanga-roo)\ **Post date:** [September 25, 2024, 2:53am UTC](https://erlangforums.com/t/do-you-make-use-of-term-to-binary/2416/16 "2024-09-25T02:53:32Z")
Thanks, Roger. Your response has answered my question. Happy coding! ---
**Author:** ![kanga-roo](https://erlangforums.com/user_avatar/erlangforums.com/kanga-roo/32/1081_2.png) [@kanga-roo](https://erlangforums.com/u/kanga-roo)\ **Post date:** [October 27, 2024, 3:26am UTC](https://erlangforums.com/t/do-you-make-use-of-term-to-binary/2416/17 "2024-10-27T03:26:21Z")
With ‘one file per user’ [.dat or .bin], would mnesia be suitable for metadata about those files? If so, is there a link to an example? ---
**Author:** ![rlipscombe](https://erlangforums.com/user_avatar/erlangforums.com/rlipscombe/32/459_2.png) [@rlipscombe](https://erlangforums.com/u/rlipscombe)\ **Post date:** [October 28, 2024, 11:14am UTC](https://erlangforums.com/t/do-you-make-use-of-term-to-binary/2416/18 "2024-10-28T11:14:08Z")
> [@kanga-roo](#): > > would mnesia be suitable for metadata about those files? No. Think about it like this: mnesia is a cluster-replicated data store. “one file per user” (being stored in the filesystem) isn’t. So you have a mismatch between the intended use patterns of the two. ---
**Author:** ![kanga-roo](https://erlangforums.com/user_avatar/erlangforums.com/kanga-roo/32/1081_2.png) [@kanga-roo](https://erlangforums.com/u/kanga-roo)\ **Post date:** [October 30, 2024, 4:11am UTC](https://erlangforums.com/t/do-you-make-use-of-term-to-binary/2416/19 "2024-10-30T04:11:02Z")
Thank you for your insightful and concise explanation, Roger.