How to build a separate NIF library for each rebar3 profile?

Questions / Help
,
1

I would like to build a separate dynamic library with different gcc options depending on the rebar3 profile (i.e. test and prod). So far I’m unable to find any documentation on accessing the current rebar3 profile from within a Makefile or my Erlang source for when I load the NIF. What’s the recommended way to achieve this functionality?

2

Can you set pre_hook for compile provider and export some env var, e.g.:

{profiles,
 [{custom_nif, [
       {pre_hooks, [{compile, "export MY_PROFILE=custom_nif"}]}
   ]}
 ]}.

and then use that environmental variable in Makefile?

3

That might work, but how do you access the profile to load the NIF? For example, if the test profile produces priv/test/nif.so, how would nif:init/0 work?

init() ->
    erlang:load_nif(???, 0).
4

That’s not achievable without accessing that env var at runtime, but I doubt that’s a reasonable solution. Why is load path dependent on the profile? Can’t you build NIF to the same path no matter what the profile is? After all, you’ll only use one at the time, right?

EDIT: maybe you could define preproc variables per profile and then use ifdef macros in the code? But again, you’ll use one .so file at the time.

5

This is not the ideal solution, but I always wrap rebar3 in a Makefile. make allows to set any environment variables and is generally more flexible than “declarative” build systems.

Please keep in mind that when your project is used as a dependency to other projects using rebar3 or mix, it will likely be called without the wrapper. So you’ll have to have reasonable defaults.

6

This is not an acceptable solution. What if I have multiple instances with different profiles? Elixir’s mix goes further with its “Environments” and maintains separate builds of regular Erlang code for each environment (in case of compile differences) and supports accessing this at build time and runtime.

7

Unfortunately the NIF I’m working will eventually be added as a dependency to opentelemetry, and enter the dependency tree of nearly every Erlang and Elixir project out there. So, wrapping Rebar3 is not an option since the wrapper won’t be called.

8

Edit: I’ll investigate this further as a temporary fix since I primarily need to support the default profile as a dependency. This is still unideal because the debugging experience of dependent projects will be broken.

9

The dependencies likely don’t need the NIF built with test profile. Hence I mentioned having the correct defaults.

and enter the dependency tree of nearly every Erlang and Elixir project out there

The horror.

10

That’d only work if your build system is available at production (which is not usual I believe).

All deps are built under prod profile, no matter what profile you’re using to compile the base project. If you want to expose building your dep with different profiles I believe that’s not doable that way, but is maybe doable with application env variables

11

I have used rebar.config.script to pull some tricks for nif options. Never anything specific to the profile, but I imagine it would be available to the script. Example below

12

I finally solved this with a combination of @mmin original solution and config files.

Passing the profile to make

I used @mmin suggestion and put a pre_hook within the profile, as well as an artifact to check whether the NIF has been built in the correct path:

{profiles, [
    {test, [
        {artifacts, ["priv/test/lz4_nif.so"]},
        {pre_hooks, [
            {"(linux|darwin|solaris)", compile, "make PROFILE=test -C c_src"},
            {"(freebsd)", compile, "gmake PROFILE=test -C c_src"}
        ]},

        {post_hooks, [
            {"(linux|darwin|solaris)", clean, "make PROFILE=test -C c_src clean"},
            {"(freebsd)", clean, "gmake PROFILE=test -C c_src clean"}
        ]}
    ]}
]}.

Accessing the NIF path

I then set a default path in my application env with the lz4_nif_path key in src/lz4.app.src

{application, lz4, [
    {description, "Erlang bindings to the LZ4 compression library"},
    {vsn, "0.1.0"},
    {registered, []},
    {applications, [
        kernel,
        stdlib
    ]},
    {env, [{lz4_nif_path, "priv/default/lz4_nif"}]},
    {modules, [lz4, lz4_nif]},
    {licenses, ["Apache-2.0"]},
    {links, ["https://github.com/Benjamin-Philip/lz4-erlang", "https://github.com/lz4/lz4"]}
]}.

And I could then access this path with application:get_env/2:

init() ->
    {ok, Path} = application:get_env(lz4, lz4_nif_path),
    erlang:load_nif(Path, 0).

Changing the NIF path for each profile

I then defined a config file for the test profile in config/test.config with the following content:

%% -*- mode:erlang; -*-
[{lz4, [{lz4_nif_path, "priv/test/lz4_nif"}]}].

and I overrode the default config within the test profile by passing this file in ct_opts and shell, giving me the final result of:

{profiles, [
    {test, [
        {artifacts, ["priv/test/lz4_nif.so"]},
        {pre_hooks, [
            {"(linux|darwin|solaris)", compile, "make PROFILE=test -C c_src"},
            {"(freebsd)", compile, "gmake PROFILE=test -C c_src"}
        ]},

        {post_hooks, [
            {"(linux|darwin|solaris)", clean, "make PROFILE=test -C c_src clean"},
            {"(freebsd)", clean, "gmake PROFILE=test -C c_src clean"}
        ]},

        {shell, [{config, "config/test.config"}]},
        {ct_opts, [{sys_config, ["config/test.config"]}]}
    ]}
]}.

Disadvantages

I had to manually specify the config for each subcommand-profile combination. Since I was only going to run CT and shell anyway (and CT only runs within the test profile), this boiled down to setting it for just shell for each profile, which was manageable.

2 Likes
13

I only meant to convey that I wanted to get this right, in the most standard way as possible, since a lot of people would be impacted if this breaks.