For my project I’ve been able to do rebar3 as prod tar, then take the tarball and deploy it to my target server and run. The issue now is I have started running stuff from a newer version of Ubuntu than my target servers, so these tarballs do not run anymore on the target system since the included erts has incompatible glibc versions. So now I am having to rethink my deployment strategy.
One thought I have is to run the old version of Ubuntu and build twice until moving to the new version of Ubuntu on the target servers are complete, or something else…of which, I’m not sure what that something else is. I see this in the rebar3 docs: Releases | Rebar3
It could be worth a try. I haven’t used it myself, but I think you could either: 1. not include ERTS in the release and install it on the target server instead, or 2. locally install an ERTS that’s compatible with the new glibc version and give rebar3 its path using include_erts and system_libs.
Do the builds in an environment that’s binary compatible with the target. VMs, Dockers, or spare machines running older OS:es are all possible approaches for this.
Do the builds using a cross-compiler (for C/C++) whose output is compatible with the target. Building cross-compilation versions of the GNU toolchain for mainstream targets is usually not difficult, but I don’t know off-hand how well rebar3 mixes with cross-compilers.
Unless you’re comfortable setting up cross-compilers I’d recommend option 1.
FWIW, we build the portable ejabberd and eturnal binaries we publish using a cross-compilation environment to link against an old glibc version. The script that does this for eturnal might serve as a better example than the ejabberd one (it’s a bit shorter). But it’s still complex, as it goes to some lengths to build all C dependencies and link them statically.
Rebar3 allows for specifying a directory that contains a cross-compiled ERTS to include with the release. However, if cross-compilation is just (ab)used to link against an old glibc version, but otherwise build and target platform are the same, the “cross-compiled” Erlang/OTP can just be used for building the release and Rebar3 wouldn’t need any special setup at all.
We spent some time going down the statically compiled rabbit-hole, and yes, when it comes to some NIFs you’re going to start pulling your hair out.
This is why we stuck with a layered Docker approach for each build target.
It’s a bit of work to build out the builder scripts initially, but definitely worth it.
Layers:
Base OS
SSL and CMake (change of version rebuilds the layer)
Dependencies and Erlang using asdf for simplicity (change of version rebuilds the layer)
App build, release, and packaging (change of version rebuilds layer)
The versions for each layer are simply sourced from external configs.
In most cases only the final layer gets rebuilt and we end up with platform specific releases as .deb or .rpm with statically compiled SSL (we still have to target old platforms)
Thank you all for the answers. I understand the Docker approach here, but it isn’t permitted in my work environment. I think then the path of least resistance is to reset up a VM with the matching OS and deploy from there to old machines, then deploy from the new build machine to new machines until the last one is gone.
You could also use Docker to run a binary compatible local build environment so you can run rebar3 as prod tar inside, then extract the archive from the Docker image and deploy as you normally would. No messy cross-compilation needed.