Limitations of `term_to_binary` on extremely large lists

Questions / Help
,
1

Hi! First-time poster here!

I’m encountering what I believe is a failure of term_to_binary (term_to_iovec, too) to properly encode extremely large lists. The setup is:

  • A list of maps which contain exactly two key-value pairs – both keys and values are non-empty binaries.
  • The list contains ~ 230 million elements and when external_size is called on this list (no compression options) it returns a value of ~ 15.4 GB
  • Using term_to_binary to encode the list (and then writing the resulting binary to disk) produces no error. However, attempting to decode the list (using binary_to_term, after reading from disk) produces an “invalid external representation of a term” error.
  • Performing the operation my_list |> :erlang.term_to_binary() |> :erlang.binary_to_term() (where my_list is the list in question) results in a core dump/crash. This operation also produces a core dump when executed on arbitrary sub-segments of the list containing >= 100 million elements. However, the core dump is non-deterministic in the sense that repeatedly executing this operation with the same argument will eventually produce a core dump, but an individual call may succeed.
  • All operations are being performed on a machine with 250 GB of memory and this behaviour was observed under both OTP 26 and 27 (I have not tested other versions, yet).

I apologize for the limited info and I hope to provide an actual, reproducible example of this behaviour that is more instructive – at some point in the future. In fact, aspirationally, I am hoping to use this as an opportunity to learn more about BEAM internals, integration with gdb, etc – since I have precisely zero experience with these topics. However, I don’t have time to devote to debugging this issue in the near future (and I was able to solve the issue this error was creating through other, architectural changes).

I was curious, though: has anyone encountered this error, before? Does anyone have any thoughts on what might be causing it? Are there any known limitations on the maximize size of a term or list that term_to_binary supports?

It seems like the issue could be similar to this previous bug…

Thanks!

1 Like
2

Please provide a reproducer for the first three parts (from “- A list of maps …” to “- Using term_to_binary …” inclusive) with concrete self-contained code, preferably Erlang.
Also this would be better suited for the bug tracker than this forum. Either way we need a way to reproduce the issue.

1 Like