Out of Memory Reading a file between OTP18 and OTP19

Questions / Help
, ,
1

I am not an expert in Erlang but we use it in the company and I am struggling with an out of memory error reading a ~1GB. I have tried multiple ERL settings to set the max heap size, etc., but nothing seems to address the issue. I am out of luck and not sure what to do next to try and solve for the issue, as this environment need to load and process many files during startup, using Bitcask in the end. So what I noted below is my attempt to rule out any system or environment or dependancy issue.

To be clear, the issue is not present on OTP18 but fails on OTP19, and this is booting the VM with no settings or overrides.

I am running OTP19 on Solaris 10 – yes yes I know this is old and part of my technical debt I am trying to solve.

I would appreciate any help or guidance, thank you!

dd if=/dev/zero of=1GB_file.bin bs=1M count=1024

erl

A = file:read_file("1GB_file.bin").

When executing the last command it crashes.

2

How much memory does your Solaris 10 system have, and are you running a 32-bit version of Erlang or a 64-bit one?

3

We are running 64-bit Solaris and Erlang, as per below.

isainfo -k
>> amd64

erl
>> Erlang/OTP 19 [erts-8.3] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false]

We have tried various zone configurations up to 16GB of memory (on resources).

4

As a workaround, I recommend that you read the file chunk by chunk. For example:

1> {ok,Fd} = file:open("1GB_file.bin", [open,raw,binary]).
{ok,{file_descriptor,prim_file,
                     #{handle => #Ref<0.3613666047.3892707348.102271>,
                       owner => <0.84.0>,
                       r_buffer => #Ref<0.3613666047.3892707336.102735>,
                       r_ahead_size => 0}}}
2> file:read(Fd, 1000000).
{ok,<<0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
      0,0,...>>}
> file:read(Fd, 1000000).
{ok,<<0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
      0,0,...>>}
    .
    .
    .
2 Likes
5

Thank you @bjorng. It is still just strange that the same call works without issue on OTP18 and I am not sure as I could not find a bug reported on OTP19 related to large file reading and memory.

In the end, a module actually processes the file, Bitcask, where the real file is loaded, but I was able to show that using simple file loading also fails, so not related to Bitcask in the end.

Is there any OS or Erlang setting that I can tweak perhaps?