Configuration of Erlang system

Erlang News & Announcements Official Proposals
,
21

Agree on that. As I mentioned above, one of the projects I maintain are supposed to be configured by end users who are non-erlang programmers (and often non-programmers at all) and so more than half of support requests I get are about errors in config files.

Also, there are some situations when “erlang native” notation syntax while is easier to handle in Erlang code, is not that friendly for the configs. See, eg, IP addresses: "127.0.0.1" vs {127, 0, 0, 1} or even worse for IPv6. Datetimes? Binary vs string (eg, it’s better to have binary there due to performance reasons, but it’s not easy to explain why user should put <<"secret-token">> instead of "secret-token").

So yep, for “internal” apps sys.config syntax is ok, but for user-configured some tools to allow custom syntax is must-have.

2 Likes
22

One problem with allowing multiple syntaxes is that the different variants will allow a different set of configuration and may define merging in different ways. For example, let’s say we want to configure this:

{kernel,logger_level,all}.

using toml this might be something like:

[kernel]
logger_level = "all"

How is the parser supposed to know that this is an atom and not a string? Or is it kernel’s job to handle both strings and atoms?

Does anyone have any good ideas about how to deal with this?

4 Likes
23

You’ll need to define the config keys and types (that you accept) and load, parse, validate the foreign format. Easy for an extra config file but a bit more work for a sys.config alternative.

I use the Vapor package which gives you a map function for each config variable that you define.

{:db_port, "DB_PORT", default: 4369, map: &String.to_integer/1}

2 Likes
24

See also this to dynamically create config or VM args before release GitHub - crownedgrouse/esh.mk: esh template engine for erlang.mk

2 Likes
25

My 2 ören. Mostly coming from 3-years-long process of converging 5 different configuration systems into a single pipeline.

  1. I prefer to stay away from code in configuration (especially Turing full code). Been there, done that. Configuration should be considered as potentially malicious input to the system, and changes to it should be treated pretty much like a code deploy events.
  2. I expect DevOps engineers to understand Erlang Term format. If it’s too hard, then we can implement tooling to visualise the format. Say, Erlang Term format browser/editor. I don’t mind tools doing 1:1 conversion from/to another DSL (JSON, *ML, …). But I’d rather keep source of truth (the file read by ERTS) in term format.
  3. Environment variables are evil (will make a longer post on it later). It took me a long time to understand how to get rid of it in our containers. The solution is simple: pass all needed variables explicitly via command line. In our code we use application:get_env(myapp, mykey), and our container starts with command like like /opt/myrel/bin/erlexec -myapp mykey myvalue.
    Restarting emulator (init:restart) can only work when there are no changes in sys.config.src (because init:restart won’t call rebar’s “extended_startup_script”). Therefore entire configuration has to be in sys.config and command line.
  4. ERTS configuration is messy. I would really love ERTS configuration to be done the same way as any other OTP app. Available via application:get_env(erts, "swt"), and updated via set_env where applicable (throwing badarg for read-only properties). Same applies to all other applications, we really suffer from being not able to dynamically update epmd_module variable. Why can’t it be application:get_env(kernel, epmd_module)?
    I also want to replace all OS environment variables (e.g. ERL_CRASH_DUMP_BYTES, or ERL_EPMD_PORT) with application:get_env. Basically, there will be no other way to access configuration other than application:get_env, or some analog (e.g. erlang:get_env). I understand that it will require config table to be implemented differently (as part of ERTS, before application controller), but it’s also a great opportunity to speed up the whole thing, and add guard rails.
    SSL app goes even further by having its own “configuration file” for Erlang distribution over TLS (although I found that I can put it in the same vm.args file, with slightly changed key names, - undocumented but very useful). I’d hope it to also be changed for application:get_env(ssl, ...)
  5. OTP does not provide a concept of “environment” on the application level. Yes, one can make a test release, and a production release, with sys.test.config and sys.prod.config. But for my test release I want test fixtures to be in “priv” folder (e.g. database initialised with test versions, or debug-compiled NIFs), and I don’t want these fixtures in prod.
    Also, default configuration in application spec file (e.g. myapp.app, env section) file should be different, say, for test environment I want to put local database file handle, and staging env should point at staging DB. We sort of tried to have that in rebar3, but haven’t used that yet.
  6. Not sure if it’s configuration, but. Secrets support. Passwords/keys/…, and some “trusted execution engine” or “secure channel” in general. Although I don’t know if it’s possible to have a built-in solution. We use a custom keychain service, but I guess there is some demand for k8s and similar environments.

To summarise:

  1. sys.config is good. sys.config.src is not, especially combined with containers (requires file system mutation, which is not available for read-only containers). One can switch from sys.config.src to sys.config by moving environment variables into command line
  2. Executable config is… just code. Let it remain in code. As I found out, “executable configs” are usually long scripts with very small input, like “environment type” (dev/test/staging/prod) variable. So in fact only this variable is the real configuration. It’s not executable and does not need to be. Put the “executable config” as code in your release, name it “my_config_provider_app”, make all other apps depend on it. It won’t work for ERTS/kernel/stdlib explicitly loaded, but for those you have command line (vm.args) or container script logic.
  3. It’d be good to have tooling support for Erlang Term format in OTP (converters, visualisers), but I don’t think it should be a part of ERTS.

End of 2 ören.

PS: when I say “I want a single configuration source for all OTP apps, including ERTS”, I also mean “happy to work on PRs to make that happen”. Be it application:get_env, erlang:system_info(app, key), does not matter for me, I just want it to be a single source.

7 Likes
26

I too would like to move towards a single source of configuration, and “configuration parameters” (as the application:get_env config is called) is the best way to do that. Today configuration is a mismatch of command-line switches (-sname), environment variables (ERL_CRASH_DUMP), separate files (inet.cfg), and configuration parameters (logger). For ERTS we try to only use + command-line switches for all ERTS configurations, but a lot of legacy configuration options remain the way that they always have. Both ERTS, Kernel, and all the rest would (in my opinion) benefit from moving its configuration to configuration parameters.

For configuring a system at startup, I think most would agree that this would be a good thing™. However, the approach to take after startup differs a lot between different applications.

At the moment the core Erlang applications (erts, kernel, stdlib, sasl) only read the config at startup and you have to use an API to read/change any configuration. The most prominent examples of this are logger and erlang:system_info/1.

Other applications constantly read values using application:get_env to allow setting new values using application:set_env. Some others also use the config_change/3 application callback to update the configuration.

I’m unsure which approach I like the best. Providing an API (such as logger) is very flexible and makes the configuration system in Erlang/OTP much simpler and focused. But it makes it the application developer’s problem to create and maintain an API.

Using application:set_env/3 makes it so that the application constantly has to poll for new changes, which is not a good thing for anything that needs performance. Also, it makes it harder to cache any configuration values (by moving them to persistent_term for instance) for even more speed.

And finally doing upgrades using config_change/3 is today quite a lot of work and possibly a very un-intuitive way to do configuration changes.

Initially, my plan was to add the possibility for application:set_env/3 to trigger a callback in the application module so that the application could verify and do any changes that needed to be done when a variable changes. But was convinced that it was better (from a testing/documentation POV) to make the applications themselves provide APIs. I keep going back and forth between the options on a monthly basis as both approaches have their own pros and cons.

5 Likes
27

I suggest --tmpfs /tmp with RELX_OUT_FILE_PATH=/tmp for .src files in a container.

But I’m pretty sure @garazdawi made it poassible to pass a config through a pipe in the latest Erlang release, so relx should be able to switch to doing it all in memory but I keep forgetting to try that and update the relx scripts.

2 Likes
28

I’m doing my best to follow along here, but what would be great would be a minimal repo with an example of “best practice”.

2 Likes
29

My primary concern with sys.config et al are:

  • secrets have to be provided on disk, in sys.config
  • or you have to re-write the module to accommodate an env var or smarter pattern
  • it’s hard to get environment variables into the system

I wonder if a small improvement on the status quo would be to allow a pre-processor stage, to accept {env, "FOO"} and replace that with whatever $FOO in the environment is?

[
 {kernel,
  [{hist, {env, "HISTORY"}},
   {inet_dist_listen_min, 4370 },
   {inet_dist_listen_max, 4400 }]}
].

which, when erts is started with env HISTORY=false ... beam.smp would translate into:

[
 {kernel,
  [{hist, false},
   {inet_dist_listen_min, 4370 },
   {inet_dist_listen_max, 4400 }]}
].

I would assume that this functions largely like the C pre-processor, so you could have

[{module, 
  [{as_binary, <<"{env, "LOLWUT"}">>}, %% very bad things with quotes
  {as_atom,  {env, "NUCLEAR"}},
  {as_int,  {env, "SOME_NUMBER"}}]}] .

But we know how badly the C pre-processor turned out.

Is there a useful middle ground?

2 Likes
30

Max’s guidelines are pretty good in general, but most people these days use env vars & file-based config mgmt systems. I make heavy usage of https://vaultproject.io/ to obtain secrets securely, but it requires that you (re)write all the code to accommodate this.

2 Likes
31

I agree it would be better as a preprocessor like you describe (the code to do this in relx’s tempaltes is nasty and of course doesn’t work if you just do erl -config ...) but wanted to check you mean the same and not that there are other improvements to the relx env var handling that could be done in the mean time?

On disk is still an issue unless you do something similar to what I describe above with an in memory mount like tmpfs until I add support to relx’s scripts for -configfd.

2 Likes
32

As an additional data point: In our case, we have some configuration shared between different environments, mainly C#, Python and Erlang. Using a common format like TOML simplifies having “on source of truth” a lot, even without centralising the configuration access (just synchronising the deployment).

One thing I have also been missing in the application:get_env style is support for nested configuration, some sort of schema validation and (most of all!) notification events or callbacks for configuration changes.

Our current approach is to use YAML and/or TOML files that we aggregate to a single configuration tree (restricted type-wise to nested maps with binary keys and numeric/bool/binary values), put it in a single persistent term and access this term using `my_config:get_value([<<“root”>>, <<“nested”>>, <<“key”>>]). Since a simple process owns the term, it can call the (gproc-)registered callback functions to inform other processes (often supervisors) of changed configuration, that will then often just compare the maps and restart/start/stop things accordingly.

2 Likes
33

I think that rebar3 (or rather relx I suppose) does a good enough job with environment variables. The secrets to disk thing should be addressed once -configfd is used.

As a crutch, if you did not think of adding an environment variable to your releases sys.config, you can always do this:

ERL_FLAGS="-kernel hist ${HISTORY}" bin/my_release foreground
2 Likes
34

Adding support for “tree” style configs is also on my wish list. I tend to use the config for logger as an example of what I would like to improve.

Regarding validation, my current hope is that it might be possible to leverage the erlang type system to describe what types should be allowed and then also allow for custom logic when some values depend on each other. The problem with using the type system for it is that you would like to be able to re-use the types for usage by dialyzer and also edoc/erl_docgen, so I wonder if it is possible to make all these tools agree on what the type syntax means.

For example, the way that I do scheduler config:

-type scheduler() ::
        #{
          online => 1..1024,
          available => 1..1024,
          busy_wait_threashold => none|very_short|short|medium|long|very_long,
          stacksize => 20..8192,
          wakeup_threshold => very_low|low|medium|high|very_high
         }.
-type schedulers() ::
        #{
          normal => scheduler(),
          dirty => #{ cpu => scheduler(),
                      io => scheduler() },
          %% +sbt
          bind_type =>
              default_bind | no_node_processor_spread |
          no_node_thread_spread | no_spread |
          processor_spread | spread | thread_spread |
          thread_no_node_processor_spread | unbound,
          load =>
              #{
                %% +scl
                compact => boolean(),
                %% +sub
                utilization => boolean()
               },
           %% +sct
           topology => string(),
           %% +sfwi
           forced_wakeup_interval => integer(),
           %% +spp
           port_parallelism => boolean(),
           %% +swct
           wake_cleanup_threashold => very_eager | eager | medium | lazy | very_lazy,
           %% +sws
           wakeup_strategy => default | legacy
         }.

and then also validate that online > available scheduler using code:

check_env(Env, Online, [schedulers, dirty, cpu, online] = Path) ->
    Available = get_path([schedulers, dirty, cpu, available],
                         Env, erlang:system_info(dirty_cpu_schedulers)),
    if Online > Available ->
            [{error, invalid_value,
              fun() ->
                      io_lib:format(
                        "invalid value: ~p.~n  "
                        "Number of online schedulers (~p) must be less than or equal to available (~p)",
                        [Online, Online, Available])
              end, Path}];
       true ->
            []
    end;
check_env(_, _, _) ->
    [].
3 Likes
35

not quite sure what you mean by that, but the general idea is to allow as a last pass, to splice env var contents into the final config, without having an interim stage that persists on disk (even a tmpfs).

If somebody pwns the system they will pop a shell, and immediately scan the file system for goodies. tmpfs or no, its gone. having secrets in the VM is unavoidable today, but in theory its possible to do what openssh now does, and uncloak the keys briefly to (re)establish connections, before disabling them again. slightly better than nothing.

I’ll look at -configfd I was not aware of this possibility.

2 Likes
36

Yea, -configfd is what you want and will be used by relx/rebar3 generated releases once I, or someone, has time to patch it.

Of course if you can get a shell you can just run env and see everything. I see -configfd as more just so you don’t have to setup a tmpfs mount or similar to have a read-only environment.

3 Likes
37

Late response, hadn’t seen this thread until just now:

I’m using conf, which is a standalone variant of the configuration mechanism we have in ejabberd. It ticks your “validation” and “good error indication” requirements: Your application exports config type specs which allows conf to perform proper validation and generate (really) good error messages. It currently only supports YAML, but other backends could be plugged in. It integrates nicely with the standard mechanism in that the configuration is made available via the application environment, and reloads are performed via foo_app:config_change/3. A few examples of config type specs for OTP applications are included with conf, see e.g. those for ssl or mnesia.

4 Likes
38

Thanks for sharing! conf actually looks a lot like what I had in mind. The only thing missing (from what I can find) is a way to sync the implementation with the documentation.

I will definitely take a closer look at what it can do when I get back to this topic!

3 Likes
39

Very late reply. I haven’t look at conf but I will.

I am using Basho cuttlefish (eg Bondy) and I’ve been dreaming about implementing something to replace it by implementing a data constraint language using unification.

Then I spotted CUE Lang which is exactly that. The syntax is like JSON and it uses graph unification. Values accept types, so the same unification operation performs validation!

I started implementing something similar in Erlang but reusing some libraries I build for a Datalog query engine but I had to focus on other things.

If considering using something like an external parser for TOML, yaml etc then maybe it is not a bad idea to consider CUE (written in Go)

1 Like
40

The good thing about building such thing in Erlang is that you can use it in your code to do data validation. That is the main reason why I started implementing this thing in Erlang so that config data and internal request data (eg maps) can be validated using the same schemas and logic.

1 Like