I currently have a few large (tot. ~50GB) ets tables. I’ve optimised operations on them a lot, and the performance I get with them is decent. One bottleneck I have, however, is persisting to disk. My workflow involves writing my ETS tables out to disk on shutdown with (
ets:tab2list and similar), and reading them back on start up.
One idea I have is to replace ets with dets/mnesia, to make the reading/writing from disk simpler and more performant. I am open to even having dets/mnesia backed by a RAM disk, and actually persisting it to storage using a native OS command after my program running on the BEAM has terminated, for maximum efficiency.
Does anyone have any experience they can share in moving from ets to dets/mnesia, or otherwise optimising writing out ets tables or other large Erlang term format data to disk?