Advice needed on Build Caching - how to prevent a recompile?

Questions / Help
1

I am attempting to create a build cache. I cache the entire xz’d tar of the _build directory and propagate that between builds. A build with the cached _build directory doesn’t compile any faster than a fresh build. If I run compile twice on the CI agent the second compile is instant. Some NIFs are also recompiled, so maybe this is more systemic in my approach than something specific to rebar3. I’ve seen a few similar issues on the elixir forum, but this project isn’t managed by mix. Any advise on how to prevent a recompile? I did see this topic https://erlangforums.com/t/erlang-deployment-tools-options/ indicating that rebar3 is optimized for incremental builds, and that is the desired behavior here.

  • There are no parse transforms defined.
  • The Erlang / rebar3 version is consistent between the builds.
  • NIF .so files are cached between biulds but not some intermediate .o files.
  • I’ve also tried to recursively touch all of the files in _build, that didn’t have an effect.
  • Tried to compile the deps first, under the same profile using as production do compile --deps_only
2 Likes
2

Two more steps I’ve tried is to update the modified time of all files to their last git commit, as well as remove all macros defined at compile time. Neither of these helped.

1 Like
3

One thing that I do notice is that deps are not recompiled. Just the main application.

1 Like
4

I tried to recompile my project using rebar3 compile and it did not compile modules that did not change. Set env. vars DEBUG and DIAGNOSTIC to 1 and look at the end of the output when running rebar3 compile. When I do fresh build I get:

===> Starting 2 worker(s)
===>      Compiled <mod1>.erl
===>      Compiled <mod2>.erl

, but when I run it again I get:

===> Starting 0 worker(s)

I guess that means that no modules are compiled in the process.

Using rebar3 v3.20.0 and OTP 26.0

2 Likes
5

Yeah, I’d say it is fairly reproducible to not trigger a full recompile locally, and even between two builds on my CI agent when ran back-to-back. I am hitting my head on that initial build with cached artifacts and a git checkout.

1 Like
6

According to this, rebar3 relies on last_modified of the source file to trigger recompilation.

Yes, but I think rebar compares last_modified of source file to the last_modified in the .dag file in the _build/\<profile\>/lib/.rebar3/rebar_compiler_erl directory, not to the last_modified of .beam files. When checking out, try setting last_modified time old the source files way backward, like year ago. I tried it and it did the job.

About caching artifacts, you should somehow create that .dag file. You may create it once when compiling and then add it to cached artifacts.

2 Likes
7

@mmin good shout with DEBUG=1. It looks like the issue was with caching the resulting .erl files generated from several large leex/yecc files. Saving those seems to have done the trick, an easy oversight. It is hard to generalize this as a single solution for future build-cachers.

  • If you’re using git, set the mtime of the source files to the latest commit. This allows for new commits to rebuild the correct files. I relied on this script.
  • If you’re using leex/yecc, cache the generated .erl files.
  • Avoid compile-time macros, defined using -d to the erl compiler, that will change between builds. Especially if they’re resolved in a large header file chain.
  • Try to compile the deps first? Not sure if this one matters all that much.
  • If you’re caching your build in cloud storage like S3, try and tar + gzip or xz the _build directory, it saved a ton of time compared to downloading the individual files (30 minutes vs 5 seconds in my case).
    • Also don’t rely on your build system’s per-file caching. The one I was using modified the mtime of individual artifacts, which seems to have been triggering some builds. tar them all manually, which will preserve mtime.
  • If you’re using NIFs, it doesn’t appear as though caching the .so files matters all that much if they’re very simple, YMMV.
  • This scheme works well for ‘fresh builds’, for example I am not going to untar the artifact on the main branch.
2 Likes