Memory leak - htop shows that the beam processes are using up all memory and the swap file

Questions / Help
, ,
1

I arrive slightly confused. My application is eating up all available memory. I tried to look through observer or recon and I don’t see anything. The application does not consume memory, but htop shows that the beam processes are using up all memory and the swap file

recon_alloc:memory(allocated_types).
[{binary_alloc,425984},
 {driver_alloc,163840},
 {eheap_alloc,3014656},
 {ets_alloc,393216},
 {fix_alloc,425984},
 {ll_alloc,23592960},
 {sl_alloc,425984},
 {std_alloc,688128},
 {temp_alloc,655360}]

recon:node_stats_print(1, 100).
{[{process_count,100},
  {run_queue,2},
  {memory_total,18943352},
  {memory_procs,2637052},
  {memory_atoms,417173},
  {memory_bin,13408},
  {memory_ets,376320}],
 [{bytes_in,8994},
  {bytes_out,0},
  {gc_count,12},
  {gc_words_reclaimed,27756},
  {reductions,61445643},
  {scheduler_usage,[{1,0.6438721760242981},
                    {2,0.002663861349804496},
                    {3,0.0035298204128896508},
                    {4,0.0058989101171309614},
                    {5,0.0},
                    {6,0.0},
                    {7,0.0},
                    {8,0.0}]}]}

erl -version
Erlang (SMP,ASYNC_THREADS) (BEAM) emulator version 13.2.2.6

Erlang/OTP 25 [erts-13.2.2.6] [source] [32-bit] [smp:4:4] [ds:4:4:10] [async-threads:1]

which way to look?

2

Please look at Chapter 7 (Memory Leaks) of Erlang in Anger. I believe it will guide you to find out what’s going on.

2 Likes
3

I read this book first, but everything is fine with the code, a similar problem occurs when I try to run my code as a linux service in systemd

4

If you just run your system as a regular application from the command line you do not see this memory leakage? It only leaks when executed via systemd? That would be quite unusual (but interesting) if true.

5

it looks strange, if I run it as a linux daemon, memory starts to grow, but if I run it as bin /app foreground everything is fine, perhaps there is some kind of peculiarity in launching the erlang daemon

6

I had the same problem when I tried to run the script via rc.local

7

Could this be memory fragmentation and not actual memory consumption? I’ve seen Linux VMs report 90% memory utilized while the application reports its using much less. Once traffic dies down the memory utilization also decreases.

8

I disabled all application consumption. My general server starts empty, it doesn’t work and memory still leaks.

9

Did you try the noshell option when you start as daemon?

10

no I haven’t tried it

11

It leaves your server running as a foreground process, but without the interactive EShell. It can help when you start from another process. In zotonic we have an option for that to run in docker.

12

I think rebar3 does it for me. I’m doing a release of rebar3 and running it as myapp foreground. Correct me if this is not true

13

For some reason, 24 applications are launched at once, is this how it should be?

14

No, that is how Linux lists multi-threaded programs with ps or htop. It’s been like that for decades. A bit weird… linux - Why are there many processes listed under the same title in htop? - Super User

15

Yes that’s right. I try to launch the application manually, the consumption is 93M and is stable, I launch it as a service, the memory starts to grow.

Screenshot_20240215_171638
Screenshot_20240215_1716381577×231 36.4 KB

launched in the console

Screenshot_20240215_171722
Screenshot_20240215_1717221576×255 46.1 KB

Screenshot_20240215_171736
Screenshot_20240215_1717361580×237 39.4 KB

As you can see, memory consumption begins

16

After some experimentation, I see that the problem occurs with the fact that I am capturing output to the console. If I stop outputting to the console, then the memory consumption does not increase. It remains to understand how I can correctly display on the screen from an application running as a service

2 Likes
17

You can always connect to long running process and inspect it with a remote shell.

18

this is not what I want, I display information on the monitor via ncurses and this seems to be the problem