What is the best way to deal with rebar.lock files of dependencies?

Questions / Help
1

We had an internal discussion and couldn’t decide on a best practices for dealing with rebar.lock files of our own dependencies. The documentation explains what happens and tells me rebar.lock should always be committed to the repo for reliable builds.

But this can end up with “diamond” dependencies, where more direct dependencies have different versions in their rebar.lock file and one of them gets picked at random (actually IIRC in alphabetical order).

Our workaround for that would be to list these “diamond” dependencies in the top-level rebar.conf in deps which would result in them being added to the top-level rebar.lock which would override the others.

So my question is: is this the way? Whats best practices around that?

2 Likes
2

Some questions (assumptions are bad):

  1. It sounds like you’re working in an umbrella, is the correct?
  2. Related to 1. , it sounds like you have a need to have different versions on a per app basis in your umbrella, is that correct?
  3. If 1. and 2. are correct, it sounds like pinning in the rebar.lock file on a per basis doesn’t solve your problem and may contribute to it, is that also correct?
3

I think I have to say no to all three questions:

  1. We don’t have an umbrella project but separate repos for our own and open source repos. We have deps that can be re-used, some of them are open-sourced by us.

  2. No special needs, it could just happen that the deps repos have different versions of sub-deps in their rebar.lock

  3. Because 1+2 are not the case this doesn’t really apply

4

Great I didn’t make assumptions then :slight_smile:

So it sounds like some deps maybe use other deps and/or you have applications that use some or all of them, is the correct?

5

Yes that’s the case. Just vanilla no umbrella project with multiple deps that can have nested other deps some of which are open source from hex or GitHub

1 Like
6

Ok next question, in your main application using said deps are you “pinning” the version? i.e., {deps, [foo, "1.1.1"}] ?

7

As I wrote above, no pinning. I’m asking about rebar.lock

Also I’m asking about best practices so in order to have an answer useful for others it should not be too specific. This is going in the wrong direction regarding a general answer

8

I would hope pinning would be the best practice here :slight_smile: Or at least using the ~> op (though that might not work out so great per your problem statement (need verl included to make it better AFAIK).

That said, IMHO this smells like a bug to me. I’m sure Fred or Tristan would have more to say about this though.

9

But that was not the question. It was best practices for using rebar.lock

10

Fair enough, but I would say you’ve already stated the best practice, commit it. Your only other resolve is to not check it in (which I don’t think anyone would recommend, hopefully).

11

Not sure if it’s a best, but definitely a practice: I try to avoid this by upgrading all dependencies under my control to depend on a single (common) version of everything. If it’s not possible and rebar would need to pick one version - I might as well make it explicit and add it to the rebar.config file.

1 Like
12

I would say for even internal projects this can be problematic unless you’re pinning all the things. Resolvers may go with a version you didn’t test with before and now things break or worse there is latent behaviour that is not apparent until you get to prod :slight_smile:

I’m not trying to change your mind or anything, but providing information for all.

13

The dependency chosen is not just alphabetical, but it’s also based on the depth of the dependency within the tree. So something that’s a transitive dep of a transitive dep will be less likely to be chosen as a version than a transitive dependency of a direct dep (app is level 0, the dep is level 1, the transitive dep is level 2, the transitive dep of a transitive dep is level 3).

Only if the conflict exists at the current level do we fall back to alphabetical order of the parent to pick a winner.

In general, our expectation is that if this isn’t acceptable, specifying the dep at the project root is exactly what you should do, as it will reset its depth level and promote it to be more important in the next conflict (if any)

3 Likes
14

Thanks for clarifying this, wasn’t aware of the depth rule and therefore why putting it in the toplevel rebar.conf works.

15

For visual more visual readers, we have diagrams in the docs: Dependencies | Rebar3 (of course knowing where to find the content in the docs is always the problem!):

For a regular dependency tree, such as:

  A
 / \
B   C

the dependencies A, B, and C will be fetched.

However, for more complex trees, such as:

   A
 /   \
B    C1
|
C2

The dependencies A, B, and C1 will be fetched. When Rebar3 will encounter the requirement for C2, it will instead display the warning: Skipping C2 (from $SOURCE) as an app of the same name has already been fetched.

Such a message should let the user know which dependency has been skipped.

What about cases where two transitive dependencies have the same name and are on the same level?

   A
 /   \
B     C
|     |
D1    D2

In such a case, D1 will take over D2, because B lexicographically sorts before C. It’s an entirely arbitrary rule, but it is at least a rule that ensures repeatable fetches.

In the event users disagree with the outcome, they can bring D2 to the top level and ensure it will be chosen early:

  A D2
 /   \
B     C
|     |
D1    D2

Which will yield A, B, C, and D2.

Rebar3 will perform that same algorithm with packages, and will also detect circular dependencies and error out on these.

Dependencies in the _checkouts directory will be left untouched, and are treated as top-level OTP applications.

this mechanism of favoring “closest to the root” is something I believe we had decided to borrow from Maven, and given how unreliable version numbers were back then, it worked surprisingly well and still works surprisingly well today at grabbing deps that work together. “Closest to the root” does all the heavy lifting of course, lexicographical sorting is just a deterministic tie-breaker.

4 Likes
16

Speaking of Rebar documentation, is there any way I can get
the documentation as a single PDF?

17

Not as far as I know. The docsite’s source is at GitHub - tsloughter/rebar3.org: Rebar3.org and uses hugo, which is a static site builder.

Apparently hugo has no PDF generation option. There are apparently some open source options which I have never tried. My past experience with these tools is that they’re going to be disappointing, but I can always see if I can find time to try them and report back if that can be useful.

Considering the document structure closely matches a website, I don’t have great expectations there regardless though. That would have required a more linear way of building the content than what the site allows.