I wonder what is the real meaning of runtime_dependencies in application resource files. Documentation says it is “A list of application versions that the application depends on”. Thing about versions is all clear, but what makes an application dependent on another app? E.g. stdlib’s runtime deps are erts, sasl, compiler and kernel, but only kernel is listed as dependency in applications list. So, when creating a release, only kernel is included from stdlibs dependencies. Shouldn’t runtime_dependencies also be automatically included in the release? Or should runtime_dependencies should also be listed in optional_applications list?
Current mechanism makes system_information:sanity_check() to fail on releases where you don’t explicitly specify all runtime dependencies (which is not something you usually want to do).
I do believe runtime_dependencies is used mostly by the OTP team. You can have of course use it, but chances are there’s not a lot of value in doing so.
Applications across various releases of Erlang/OTP can be mixed and matched. This property in app source file dictates the smallest version that an application can depend on and properly function with.
As an example, ssl app lists stdlib 4.1 as a runtime_dependency in the main branch in git, yet the stdlib in the same git ref is 5.1.1. This means I can take stdlib from a previous release and run it with ssl 11.0.3 (provided I’ve met the bare minimum for all other dependencies of ssl).
Likewise, in release notes by the OTP you’ll notice entries like the following, this one was for the latest update to ERTS in 26 point release.
Full runtime dependencies of erts-14.1.1: kernel-9.0, sasl-3.3, stdlib-4.1
This tells me that if I want to use this version of erts, I need at least these versions of kernel, sasl, and stdlib. If I check erts/preloaded/src/erts.app.src in the source, that holds true.
I agree with everything you said, but if stdlib-5.1needs at least sasl-3.3, shoudn’t sasl be automatically included in every release which uses stdlib?
stdlib works without sasl, but some specific parts of stdlib use sasl - to me, it sounds like sasl should be listed in optional_applications in stdlib.app.src.
Okay, sasl may not be a good example (prolly is runtime dep cuz of code upgrade), but stdlib also depends on compiler and crypto and you can find usages of compile and crypto modules in source of stdlib
You could add them there, but it would have no effect. Maybe you see value in information or compile time checks?
TL;DR optional deps will be started when application:ensure_all_started/1 (and friends) is called, but in the case of kernel or stdlib, the app will already be started, and thus any optional dependencies it defines will be skipped.
FWIW, these applications (and compiler is a special case) are loaded on demand in stdlib itself.
A counter example to this is observer, it defines optional dependencies, such that if you load and start it via application:ensure_all_started/1 (and friends), it will also start up et, wx and runtime_tools, if available.
But if e.g. crypto app is not manually included in the release, stdlib can’t load/start it. Shouldn’t release handler take care of that and include runtime dependencies in the release automatically?
Imagine having a minimal release of just 3 apps: stdlib, kernel and my_app. my_app contains only 1 module: my_mod which contains only 1 function my_fun/5 which calls beam_lib:decrypt_chunk/5 which uses crypto module. You test my_app in the rebar3 shell and it works - yay. Than you make it into a release and include only those 3 apps to make it minimal - all of a sudden, calling my_mod:my_fun/5 will give you a runtime error. Whose fault is this? stdlib correctly assumes that crypto module is present because crypto is specified as a runtime_dep. How any why should user know that calling beam_lib:decrypt_chunk/5 requires an extra dependency for his app?
EDIT:
okay, beam_lib:decrypt_chunk/5 is not exported, but some exported functions uses it somewhere, it’s taken for the sake of an example
Right, so that’s the compile time check I was hinting at
That would be indeed terribly useful. As far automatically including it in a release, perhaps… Explicitness is still desirable though, I think. But yes, what a release handler or some other mechanism could do is yell at you. “You’re trying to call crypto:aeaddeadbeef/42 but you do not have this defined as a dependency in your app, do this : …”
^ that’s good tooling IMHO
Edit:
FWIW, I think there’s a lot of hunting we could do for places where we think principle of least astonishment is violated. This is a good start
Exactly, if you think about explicitly stating applications’ versions in release spec, I’m absolutely for that.
I think xref could do that, but that is the problem of stdlib, not my_app. I don’t think you want to run static analysis on your deps. The other mechanism I used is system_information:sanity_check/0 which failed on what should be a valid release - it failed not on my app’s deps, but on stdlib’s
I don’t think this should be an error/warning, because it’s just not your fault. As you mentioned before:
Notice that word need is bold, because an application needs some other applications in order to work properly. That need should be addressed by including those applications in a release.
I think that every release which contains no user-errors should pass sanity check.
EDIT: FWIW, this shouldn’t be mandatory, but at least there should be a way to include runtime deps in releases (even though they should be included by default :D)
Right, I conflated another case with your stdlib case. I’m thinking about the case where it’s a problem in your app, but you know later rather than sooner.
See above
Agreed, as this can be a bit tricky with kernel, stdlib, etc. Not every case requires crypto as an example, yet stdlib is needed by everything.
So what I hear and agree with is an option {bundle_optional_dependencies, true}. This of course doesn’t have to live inside OTP, it could be part of relx, mix, etc. though it’s nice to have core features available to all tooling outside of OTP to avoid duplicated effort.
EDIT: FWIW, I didn’t edit this, I just wanted to keep symmetry from the previous two posts going
Exactly, but we can’t be sure in compile time whether that part is being used or not.
I was thinking about bundle_runtime_dependencies, but actually, both options could have it’s purpose (+ dunno if included_applications get into releases).
It doesn’t have to, but I still think that current mechanism is incomplete. Although It’s a corner case, I don’t see why something called dependency is not being treated as one. I may go with a PR to release_handler directly, that way every build tool can just easily adapt that feature (I’d rather call that fix, not a feature )
Well, you wouldn’t want to make use included_applications, and I would suggest reading the documentation around that. I wonder if that should be deprecated actually.
I think this notion of bundling optional dependencies needs a little more thought, though I’d be quite interested to see a PR, as I’m sure there are bits on your end I’m not picking up. One problem I see is you have one option that you put in an app source file and all optional dependencies get bundled. This would create for a huge release and is a heavy handed way of getting just a few dependencies you care about it (albeit indirectly) in there without having to manually specify them. If size of a release is not a concern, then well that may be of no concern, but there’s perhaps other reasons to avoid bigger releases as well (startup time and in general just more surface area).
The idea of some kind of static analysis is nifty I think, but it would also be very slow in a large application, and as you said, it may be some what odd to perform this kind of analysis on your deps (but not invalid).
Only documentation I found on it is here and I can’t see why this should be deprecated and not supported in releases.
I don’t think huge is the right word here, I’d rather say bigger (equal in some cases), but that would ensure that generated releases are sane. Currently, we only have an option to exclude all runtime_deps/optional_apps and include them manually (not very scalable tho, especially when those deps are not directly yours, but deps of your deps!). Why don’t we introduce an option to include all runtime_deps/optional_apps and exclude them manually if you want smaller releases? Then, if the release doesn’t work as it should - its your problem.
Isn’t a pain when you have to handle your deps’ deps? This example is with stdlib is trivial because there is only one level of indirection, but imagine you’re packaging apps that have multiple levels of dependencies that use runtime_deps/optional_deps. When I include app X in a release, shouldn’t everything else needed for X to work also be included?
I know that included_apps, optional_apps and runtime_deps are rarely used, but I see each item listed in one of those as a dependency of the application.
