Investigating Slow Boot Time in Small Erlang Release - Seeking Optimization Strategies

Questions / Help
,
1

Problem Description

I am experiencing significant boot time issues with an Erlang release and would appreciate guidance on optimization approaches. My release exhibits boot times between 5-10 seconds, which appears excessive given its relatively small size. This issue mirrors the problem documented in this Elixir forum thread.

Environment Details

Erlang Version: Currently using Erlang 27, though this issue has been consistently observed across previous versions including Erlang 26 and even 28.

Deployment Environment: Release deployed within Docker containers.

Release Characteristics

The release contains a modest number of compiled modules:

# Total modules including OTP components
~$ find _build/prod/rel/mirakle/lib -type f -name "*.beam" | wc -l
804

# Application-specific modules including deps (cowboy, etc.) and excluding OTP (kernel, inets, etc.)
~$ find _build/prod/rel/mirakle/lib -type f -name "*.beam" | wc -l
338

# Application-specific modules only (excluding both deps and OTP)
~$ find _build/prod/rel/mirakle/lib -type f -name "*.beam" | wc -l
56

Current Configuration and Attempted Solutions

I have switched from embedded to interactive mode in an attempt to address this performance issue. While interactive mode successfully reduced memory consumption, the boot time remains approximately 5 seconds, which represents only a marginal improvement from the original 5-10 second range.

Request for Assistance

What additional strategies are available for optimizing Erlang release boot time? The persistence of this issue across multiple Erlang versions suggests that the problem may be related to fundamental configuration parameters, build optimizations, or architectural considerations rather than version-specific bugs.

I am particularly interested in understanding whether there are specific techniques for reducing module loading overhead, optimizing the boot sequence, or leveraging Docker-specific optimizations that could address this performance bottleneck.

Any insights, recommendations, or diagnostic approaches would be greatly appreciated.

For additional context, our infrastructure includes other Docker images written in Rust that boot instantaneously. However, these services must wait for the Erlang Docker image to complete its boot sequence since the primary application logic resides in Erlang. The current boot delay of nearly 10 seconds has become a significant operational burden for our customers during each deployment cycle, particularly as we do not utilize release upgrades and instead deploy fresh containers.

Best regards,
Z.

2

There are two experimental flags you might want to try to understand where time is spent – -init_debug will add some debug and timing printouts, -profile_boot will actually use tracing to track functions called during boot similar to what you’d get from eprof.

2 Likes
3

@michalmuskala thanks for the hints.

This is what i got using -init_debug after excluding module below 1000 microseconds.

{primLoad,[error_handler,application,application_controller,application_master,code,code_server,erl_eval,erl_lint,erl_parse,error_logger,ets,file,filename,file_server,file_io_server,gen,gen_event,gen_server,heart,kernel,logger,logger_filters,logger_server,logger_backend,logger_config,logger_simple_h,lists,proc_lib,supervisor]}
{done_in_microseconds,23312}
{start,logger}
{done_in_microseconds,2632}
{start,application_controller}
{done_in_microseconds,7832}
...
{apply,{application,start_boot,[kernel,permanent]}}
{done_in_microseconds,13111417}
{apply,{application,start_boot,[inets,permanent]}}
{done_in_microseconds,6983}
{apply,{application,start_boot,[sasl,permanent]}}
{done_in_microseconds,3640}
{apply,{application,start_boot,[metrics,permanent]}}
{done_in_microseconds,83357}
{apply,{application,start_boot,[cowlib,permanent]}}
{done_in_microseconds,1536}
{apply,{application,start_boot,[ranch,permanent]}}
{done_in_microseconds,1316}
{apply,{application,start_boot,[cowboy,permanent]}}
{done_in_microseconds,2508}
{apply,{application,start_boot,[syn,permanent]}}
{done_in_microseconds,1576}
{apply,{application,start_boot,[replayq,permanent]}}
{done_in_microseconds,1411}
{apply,{application,start_boot,[runtime_tools,permanent]}}
{done_in_microseconds,4140}
{apply,{c,erlangrc,[]}}
2025-05-24T08:32:59.384311+00:00 [info] [application_controller.erl.2117] Application: runtime_tools. Started at: t5@t5.
{done_in_microseconds,1519}
{progress,started}
4

{apply,{application,start_boot,[kernel,permanent]}}
{done_in_microseconds,13111417}

Kernel starts the network, maybe some network setup issue? I’m wildly guessing here.

1 Like
5

@dgud How to check if it’s a network issue?

This release is not even running epmd

-start_epmd false

Plus:

  {kernel, [ {shell_history, disabled}
           , {connect_all, false}
           , {dist_auto_connect, never}
6

@michalmuskala here is the -profile_boot output (discarding anything below 100us)

            application_controller:conf_param_to_conf/1 -      1 : 1 us
                     application_controller:scan_file/1 -      1 : 1 us
                         application_controller:start/1 -      1 : 1 us
                                                                ...
                                          erlang:send/2 -    104 : 100 us
                                          erlang:send/2 -    104 : 100 us
                                  prim_file:write_nif/2 -      9 : 101 us
                            gen_server:loop_hibernate/4 -     28 : 103 us
                                     zlib:enqueue_nif/2 -    266 : 104 us
                                       beam_ssa:rpo_1/4 -    822 : 104 us
                              logger_formatter:format/2 -     80 : 106 us
                                    filename:pathtype/1 -    622 : 106 us
                                  proplists:get_value/3 -   1468 : 106 us
                                  erl_parse:yeccpars1/7 -   1631 : 107 us
                                          maps:with_1/2 -    576 : 108 us
                                      gen_statem:loop/3 -    190 : 109 us
                                   filename:basename1/3 -   2205 : 109 us
                               erl_scan:reserved_word/1 -    516 : 110 us
                                            ets:match/2 -     13 : 111 us
                                            ets:match/2 -     13 : 111 us
                              erts_internal:open_port/2 -      5 : 113 us
                              erts_internal:open_port/2 -      5 : 113 us
                                         init:request/1 -    156 : 115 us
                                        lists:foldl_1/3 -   1165 : 116 us
                           erts_internal:port_control/3 -     19 : 117 us
                           erts_internal:port_control/3 -     19 : 117 us
                     error_handler:undefined_function/3 -    241 : 117 us
                                gen_server:decode_msg/6 -    512 : 118 us
                          erl_scan:scan_string_no_col/4 -   1784 : 120 us
           application_controller:file_binary_to_list/1 -      1 : 121 us
                           prim_file:list_dir_convert/3 -   1251 : 121 us
                         prim_file:from_posix_seconds/2 -    717 : 123 us
                               erlang:integer_to_list/1 -    802 : 123 us
                               erlang:integer_to_list/1 -    802 : 123 us
                                  persistent_term:put/2 -     74 : 124 us
                                  persistent_term:put/2 -     74 : 124 us
                                   gen_server:init_it/6 -    104 : 124 us
                    erl_prim_loader:get_loader_config/0 -    818 : 124 us
                                      filename:append/2 -    658 : 125 us
                                     filename:flatten/1 -   2561 : 125 us
                               erlang:prepare_loading/2 -    267 : 127 us
                                    erlang:spawn_link/3 -     57 : 128 us
                                    erlang:spawn_link/3 -     57 : 128 us
                               erts_internal:map_next/3 -    296 : 128 us
                               erts_internal:map_next/3 -    296 : 128 us
                                       maps:from_keys/2 -    193 : 129 us
                                       maps:from_keys/2 -    193 : 129 us
                                    lists:splitwith_1/3 -    882 : 129 us
                                prim_file:decode_path/1 -   1202 : 129 us
                                 zlib:inflateInit_nif/3 -    266 : 130 us
                                 code_server:get_name/1 -   1377 : 130 us
                         logger_backend:call_handlers/3 -    245 : 132 us
                             io_lib_pretty:get_option/3 -   1026 : 132 us
                   calendar:system_time_to_rfc3339_do/4 -     80 : 134 us
                                       erl_scan:scan1/5 -   2326 : 135 us
                   code_server:store_module_and_reply/3 -    223 : 140 us
                              code_server:handle_call/3 -    466 : 141 us
                                         zlib:inflate/3 -    266 : 142 us
                            zlib:dequeue_all_chunks_1/3 -    431 : 142 us
                                           maps:merge/2 -    647 : 143 us
                                           maps:merge/2 -    647 : 143 us
                                      erlang:make_fun/3 -    472 : 146 us
                                      erlang:make_fun/3 -    472 : 146 us
                      application_master:start_it_old/4 -     21 : 147 us
                                gen_server:decode_msg/4 -    512 : 147 us
                            erl_prim_loader:read_file/1 -    446 : 148 us
                                  zlib:inflateEnd_nif/1 -    266 : 150 us
                                  prim_file:read_file/1 -    524 : 152 us
                           io_lib_format:collect_cseq/2 -    300 : 153 us
                                gen_server:handle_msg/3 -    512 : 155 us
                              erl_prim_loader:reverse/1 -   1061 : 162 us
                          erl_prim_loader:deep_member/2 -   2730 : 163 us
                                 erlang:module_loaded/1 -    942 : 164 us
                                 erlang:module_loaded/1 -    942 : 164 us
                              erl_prim_loader:request/1 -    179 : 165 us
                                  erlang:list_to_atom/1 -    352 : 165 us
                                  erlang:list_to_atom/1 -    352 : 165 us
                                   code_server:split2/4 -   1522 : 166 us
                           unicode:characters_to_list/2 -    152 : 168 us
                           unicode:characters_to_list/2 -    152 : 168 us
                                          maps:remove/2 -    904 : 170 us
                                          maps:remove/2 -    904 : 170 us
                                   prim_tty:write_nif/2 -     35 : 173 us
                                   init:get_argument1/2 -   2700 : 173 us
                                  persistent_term:get/1 -   1593 : 174 us
                                  persistent_term:get/1 -   1593 : 174 us
                       prim_file:internal_native2name/1 -   1202 : 177 us
                       prim_file:internal_native2name/1 -   1202 : 177 us
                  logger_formatter:linearize_template/2 -   1030 : 182 us
                                   erl_scan:scan_name/2 -   3162 : 182 us
                                  erlang:process_info/2 -    372 : 183 us
                                  erlang:process_info/2 -    372 : 183 us
                                 erl_prim_loader:loop/3 -    181 : 193 us
                                        maps:try_next/2 -   2850 : 195 us
                                       filename:split/1 -   1436 : 196 us
                                prim_file:encode_path/1 -    971 : 197 us
                          code_server:get_object_code/2 -    223 : 199 us
                           logger_formatter:do_format/4 -   1030 : 199 us
                                   erl_scan:scan_atom/6 -    373 : 202 us
                           erl_prim_loader:name_split/2 -    954 : 210 us
                         unicode:characters_to_binary/2 -    372 : 213 us
                         unicode:characters_to_binary/2 -    372 : 213 us
                                  persistent_term:get/2 -    952 : 214 us
                                  persistent_term:get/2 -    952 : 214 us
                                    filename:basename/4 -   3858 : 216 us
                                  prim_tty:is_usascii/1 -   3861 : 223 us
                      prim_socket:is_supported_option/2 -   2812 : 225 us
                                          zlib:gunzip/1 -    266 : 226 us
                                   code:ensure_loaded/1 -    227 : 232 us
                                               re:run/3 -     91 : 233 us
                                               re:run/3 -     91 : 233 us
                            erl_prim_loader:normalize/2 -   4395 : 234 us
                                   filename:extension/3 -   4533 : 242 us
                                        lists:keyfind/3 -   2793 : 246 us
                                        lists:keyfind/3 -   2793 : 246 us
                            code_server:handle_loader/4 -    449 : 247 us
                                   ets:lookup_element/4 -    470 : 247 us
                                   ets:lookup_element/4 -    470 : 247 us
                                  erlang:atom_to_list/1 -    684 : 247 us
                                  erlang:atom_to_list/1 -    684 : 247 us
                                       erlang:whereis/1 -    914 : 249 us
                                       erlang:whereis/1 -    914 : 249 us
                                            gen:reply/2 -    388 : 256 us
                    erlang:universaltime_to_localtime/1 -    797 : 262 us
                    erlang:universaltime_to_localtime/1 -    797 : 262 us
                                   prim_file:open_nif/2 -     27 : 263 us
                                     lists:do_flatten/2 -   2178 : 273 us
                                         lists:append/1 -   1450 : 278 us
                  code_server:'-exclude/2-lc$^0/1-0-'/3 -   1428 : 279 us
                    erl_prim_loader:client_or_request/2 -    817 : 282 us
                                         lists:member/2 -   2952 : 289 us
                                         lists:member/2 -   2952 : 289 us
                                       erlang:monitor/3 -    291 : 294 us
                                       erlang:monitor/3 -    291 : 294 us
                                              ets:new/2 -     47 : 299 us
                                              ets:new/2 -     47 : 299 us
                            io_lib_format:count_small/2 -   3088 : 299 us
                     prim_socket:'-supports/1-fun-2-'/3 -   2812 : 311 us
                            erl_scan:skip_white_space/6 -   6375 : 313 us
                                   erlang:system_info/1 -   3766 : 348 us
                                   erlang:system_info/1 -   3766 : 348 us
                       prim_file:internal_name2native/1 -    971 : 356 us
                       prim_file:internal_name2native/1 -    971 : 356 us
                                              os:type/0 -   3416 : 364 us
                                        zlib:open_nif/0 -    266 : 370 us
                                  lists:thing_to_list/1 -    121 : 387 us
                               erl_parse:modify_anno1/3 -   3895 : 396 us
                            io_lib_format:build_small/1 -   2959 : 397 us
                                      gen_server:loop/5 -    624 : 403 us
                              erlang:iolist_to_binary/1 -    357 : 429 us
                              erlang:iolist_to_binary/1 -    357 : 429 us
                                          maps:fold_1/4 -   2926 : 429 us
                                     file:file_name_1/2 -   2718 : 437 us
                               code_server:mod_to_bin/3 -   2750 : 438 us
                          io_lib_format:build_limited/5 -   2461 : 457 us
                            erl_prim_loader:is_prefix/2 -   5261 : 458 us
                   code_server:get_name_from_splitted/1 -   8580 : 485 us
                                erlang:binary_to_term/1 -      1 : 534 us
                                erlang:binary_to_term/1 -      1 : 534 us
                                    code_server:reply/2 -    466 : 538 us
                                     erlang:spawn_opt/4 -    201 : 569 us
                                     erlang:spawn_opt/4 -    201 : 569 us
                                       erlang:monitor/2 -    930 : 588 us
                                       erlang:monitor/2 -    930 : 588 us
                                          gen:do_call/4 -    400 : 653 us
                                           ets:lookup/2 -    521 : 660 us
                                           ets:lookup/2 -    521 : 660 us
                                           ets:insert/2 -    758 : 696 us
                                           ets:insert/2 -    758 : 696 us
                               code_server:with_cache/4 -   2526 : 789 us
                                     code_server:loop/1 -    470 : 852 us
                          erlang:finish_after_on_load/2 -      3 : 877 us
                          erlang:finish_after_on_load/2 -      3 : 877 us
                                io_lib_format:collect/2 -   4057 : 906 us
                     code_server:discard_after_hyphen/1 -  10577 : 950 us
                                     erlang:demonitor/2 -   1337 : 961 us
                                     erlang:demonitor/2 -   1337 : 961 us
                                          erlang:'++'/2 -   3639 : 984 us
                                          erlang:'++'/2 -   3639 : 984 us
                               prim_file:list_dir_nif/1 -     62 : 1008 us
                                     code_server:call/1 -    466 : 1115 us
                              prim_file:read_info_nif/2 -    356 : 1163 us
                                       filename:join1/4 -  18214 : 1168 us
                                        lists:reverse/2 -  11998 : 1241 us
                                        lists:reverse/2 -  11998 : 1241 us
                                        lists:reverse/1 -  13357 : 1316 us
                        erts_internal:garbage_collect/1 -    352 : 1847 us
                        erts_internal:garbage_collect/1 -    352 : 1847 us
             code_server:'-schedule_on_load/4-fun-0-'/1 -      3 : 2061 us
                          erts_internal:trace_pattern/3 -      1 : 2698 us
                          erts_internal:trace_pattern/3 -      1 : 2698 us
                        erl_prim_loader:archive_split/3 -  34600 : 3111 us
                                       filename:split/4 -  62341 : 3464 us
                              init:debug_profile_mfas/0 -      1 : 5572 us
                              prim_file:read_file_nif/1 -    524 : 6813 us
                                  filename:do_flatten/2 -  76714 : 7250 us
                                     zlib:inflate_nif/4 -    431 : 19522 us
                                  prim_file:close_nif/1 -     16 : 25811 us
                                erlang:finish_loading/1 -    239 : 79217 us
                                erlang:finish_loading/1 -    239 : 79217 us
                                   prim_file:sync_nif/2 -      5 : 111307 us
                        erts_internal:prepare_loading/2 -    267 : 172708 us
                        erts_internal:prepare_loading/2 -    267 : 172708 us
7

To identify the root cause of the performance issue, I developed a minimal application with a simple gen_server that performs no operations.

The following excerpt shows the largest modules from the -init-debug output:

...
{progress,applications_loaded}
{apply,{application,start_boot,[kernel,permanent]}}
{done_in_microseconds,13107990}
{apply,{application,start_boot,[inets,permanent]}}
{done_in_microseconds,5632}
{apply,{application,start_boot,[sasl,permanent]}}
{done_in_microseconds,16802}

Performance Concern

The kernel application is requiring approximately 13 seconds to start, which represents a significant performance bottleneck. This startup time appears excessive for what should be a fundamental system component.

Potential Root Cause

Could this issue be related to Erlang compile options? It was compiled from source using kerl:

$ erl
Erlang/OTP 28 [erts-16.0] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [jit:ns] [sharing-preserving]
Eshell V16.0 (press Ctrl+G to abort, type help(). for help)
8

I have identified the configuration issue causing the 13-second kernel boot delay and require community guidance on the proper solution.

Current Situation

I am starting my release without EPMD using the following vm.args configuration:

-start_epmd false

For this configuration to function within the Docker container, I must also set the ERL_DIST_PORT environment variable. Without this variable, the system generates the following error during boot:

Protocol 'inet_tcp': register/listen error: econnrefused

This error condition results in the kernel application requiring approximately 13 seconds to complete initialization, presumably because the system attempts multiple connection retries before proceeding with startup.

Performance Comparison

When I remove the -start_epmd false setting and allow EPMD to start normally, the release boots efficiently with the kernel application completing initialization in approximately 3 seconds.

...
{apply,{application,start_boot,[kernel,permanent]}}
{done_in_microseconds,3082918}

Request for Assistance

I require guidance on the proper configuration approach for running Erlang releases without EPMD in Docker containers while maintaining optimal boot performance. The current requirement for the ERL_DIST_PORT environment variable suggests there may be additional configuration parameters or alternative approaches that would eliminate both the connection errors and the associated startup delays.

I would appreciate any insights from the community regarding best practices for EPMD-disabled deployments in containerized environments that achieve the same performance characteristics as EPMD-enabled configurations.

9

You need to use -dist_listen false. With -start_epmd false, and no port specified, the system will try to connect to epmd.

Whether you start or don’t start epmd should not impact boot time, as you have shown.

10

@starbelly Thank you for the suggestion regarding -dist_listen false. However, I am already implementing the ERL_DIST_PORT environment variable configuration, which automatically adds both -start_epmd false and -erl_epmd_port ${ERL_DIST_PORT} parameters as documented in the Erlware blog post on EPMDless deployments.

The -dist_listen false configuration presents a significant limitation for our deployment requirements. This setting disables the node’s ability to accept incoming distributed connections, which eliminates essential operational capabilities including remote console access and runtime evaluation commands.

Specifically, the following critical operations become unavailable when -dist_listen false is configured:

~$ /app/bin/mirakle --remote-console
Node is not running!

~$ /app/bin/mirakle eval 'lists:keyfind(encoding, 1, io:getopts()).'
Node is not running!

Our objective remains identifying a configuration approach that eliminates EPMD dependencies while preserving both optimal boot performance and essential node connectivity for operational management purposes.

11

Using emu VM instead of jit VM will speed up loading and most likely will not slow down the application.

2 Likes
12

I see. You indeed want to use erlang distribution, for a second I thought this was not what you wanted. I wonder if the problem you’re running into, at least partially is similar to the locale issue you brought up in another post, which is your environment variables are not making it along to the startup of your release. Just a hunch :smiley:

You can test this hypothesis by passing in -erl_epmd_port $YOUR_PORT in your vm.args along with -start_epmd false, which is all the env var does for you.

If you were passing in -start_epmd false and your ERL_DIST_PORT was not set in your env, then indeed, an attempt to establish a connection from the node to epmd would ensue. The only odd thing still is the fact that you hung for a while. If there was nothing listening on that port, then it should have terminated quite quickly with econnrefused. However, if somehow you did have something listening on that port, you should would have hung while dist erl waited on a reply for the registration of the node, though you should ran into a reg error after about 10 seconds and the node should have terminated. I can replicate that in fact. Though, your original logs, if they are complete, showed you hitting no such error.

As for running epmdless, do note, you will not be able to connect to a running node on the same machine in the way it looks like you are currently trying to do without a custom epmd module and/or a custom release script.

In Tristan’s example project epmdlessless, it’s a different node per container such that each node can bind to the same single static port. In other words and AFAIK without a custom epmd module to facilitate using remote_console on the machine where you’re node is running, well, it won’t work out of the box :smiley:

Good luck :smiley:

Edit: I said remote_console would not work out of the box on the same machine, I was wrong. I hit a DNS issue (it’s always DNS :stuck_out_tongue:). Note for future readers, if you have a machine name such as foo, but this is not defined in your /etc/hosts or inetrc, you will get Node is not running!. This is because the actual error ends up getting sent to /dev/null by the relx app script (error in this case is ei_gethostname related). Alternatively, just use a long name (i.e., -name foo@127.0.0.1).

2 Likes
13

@starbelly Thank you for the excellent guidance. Your suggestion resolved the issue completely. Implementing EPMDless with localhost binding eliminated the boot delay while maintaining full operational functionality.

The kernel application now achieves normal startup performance without the previous 13-second delay. The release operates without EPMD as a separate daemon process, binding directly to localhost on port 4369:

$ ps auxwww | grep epmd | grep daemon
$ netstat -antp | grep 4369
tcp        0      0 127.0.0.1:4369          0.0.0.0:*               LISTEN      1/mirakle

Essential operational capabilities remain fully functional:

~$ /app/bin/mirakle --remote-console
...
(mirakle@127.0.0.1)1>
~$ /app/bin/mirakle eval 'lists:keyfind(encoding, 1, io:getopts()).'
{encoding, unicode}

This configuration delivers both the EPMD-free deployment model and the operational functionality required for production environments. I appreciate the community assistance (@michalmuskala @dgud) in identifying this solution.

1 Like
14

@Led Interesting. With JIT enabled, the kernel application startup time is now around 3sec:

{apply,{application,start_boot,[kernel,permanent]}}
{done_in_microseconds,3087236}

I attempted to use -emu_flavor emu to run the JIT-compiled VM in emu mode, but encountered this error:

-emu_type lcnt -emu_flavor smp
-emu_flavor smp

Is there a proper method for running a JIT-compiled Erlang installation in emu mode to test @Led 's idea? The error suggests the emu flavor may not be available in JIT builds, or additional configuration is required.

15

The non-JIT VM is not built by default and thus not included in most docker images, so you need to build it. You can do so by running make emulator FLAVOR=emu after builing Erlang/OTP with the JIT.

2 Likes
16

@garazdawi does this will build both the emu and jit flavors?

1 Like
17

You need to first build the jit and then emu. That is something like this:

./configure && make && make release && make FLAVOR=emu && make release FLAVOR=emu

then a released Erlang/OTP will appear in release/$TRIPLET/.

2 Likes
18

@garazdawi do you mind explaining how to achieve this with kerl?

19

I don’t know if that is possible.

20

Once upon a time, you could specify --enable-smp-support in the configure and build beam.up and beam.smp in one go.
Why is this now intentionally complicated?