A new EEP (New EEP: add support for experimental features by lisztspace · Pull Request #40 · erlang/eep · GitHub) has been added, describing support for adding experimental features to Erlang/OTP. The plan is to make this part of the next RC and then OTP25.
Comments and suggestions very welcome!
edit: change link to new PR due to being too quick on the merge button
My opinion on this subject is a bit torn 
On the one hand, the ability to enable experimental features is nice. Up to now, something like this has been done by either a conditional compile via ?OTP_RELEASE, or in the code by checking the existence of a function that appeared in the same release as the new feature (not always possible, eg if the new feature was something more background-ish, like maybe).
On the other hand, the more fine-grained control ?FEATURE_AVAILABLE offers may also lead to even more cluttered code, especially if several such features become available over the range of OTP versions an application/library wants to support (for perspective, MySQL/OTP for example claims backward compatibility all the way back to R16B). And while code using experimental features should make it to production only with care, it is hard to afterwards find and remove all the conditional code that was introduced to try out a feature, if you remember it at all.
Another question is what is supposed to happen once a feature reaches a final stage, ie rejected or permanent. Remove it from the future list of features, or keep it?
If we keep them, the list of features may become large, and naming new ones will become more and more awkward in order to distinguish them from older features.
If we remove them, code with conditionals will revert to using the alternatives, probably unnoticed, when compiled with an OTP version where the feature is removed from the list and ?FEATURE_AVAILABLE subsequently returns false for them, even though it have been made permanent.
So maybe go down an in-between route, remove them from the list of displayed features but keep them internally so that ?FEATURE_AVAILABLE will still return true (given that they were made permanent, not rejected), and maybe issue a warning? Naming may still be a little awkward then, but it would be easier for users if they only saw easily distinguishable features instead of a myriad of similarly named ones.
Yes, there is definitely a risk of clutter when you need to have a code base that can run on several platforms/versions. This will always be true, regardless of the support in the language. Without support, you will build it outside (m4 comes to mind). I really prefer the support to be inside the language as it makes additional tooling so much easier. The main goal of this is to make it possible to evolve Erlang/OTP in a more controlled way, while both making it easier to test and get feedback on new features as well as making a transition easier/slower. Currently, we have the deprecation warnings, but when the change comes there is no way around it. Even with this, there will be a cutoff point, but it can be made softer. We do not want to clutter the code base of Erlang/OTP with long term support for a combinatorial explosion of features.
The idea is that features will not be removed when they become permanent (or rejected), so ?FEATURE_AVAILABLE will continue to return true for features that have become permanent. When obtaining a list of features, the ones being controllable (possible to enable or disable depending on state) are the most interesting. That should probably be made the default, while still having the possibility to obtain the full list and history of features.
Regarding finding features that have become permanent, we can add warnings in the compiler when trying to enable a permanent feature (easy) or check whether it is enabled with ?FEATURE_ENABLED (less easy, but should be doable). We already have the option to get warnings for current use of atoms that are keywords in experimental features and adding more warnings to find and reduce potential clutter is reasonable.
Along the lines above, the meta feature all should be clarified to cover only features that are controllable.
Sounds reasonable to me 
Yes, same here 
@cons, maybe edit the EEP text to reflect/clarify this?
Btw, where should discussions such as this one take place, regarding this EEP/feature and future ones, in the forum or in the PR on GitHub? While I think there is a wider audience following and chiming in here in the forum, the reasons for possible edits to the PR resulting from it will be hard to follow since the discussion is elsewhere.
Yes, @Maria-12648430 the EEP will be updated as changes are needed and collected. I will notify in this thread when it happens. Some comments/suggestions will probably warrant more internal discussions before they affect the EEP PR and implementation.
Regarding where discussions should take place, there is no easy answer. It would be more convenient if there was exactly one place, but I will be monitoring the forums and the PR (at least) and try to put the pieces together. If a topic is discussed on both places, I might try to focus it on one place, but I’ll take that problem when/if it arrives.
I think something like this is probably needed for development practices within Erlang/OTP to scale out in the future. Rust’s “unstable model”, while slightly less complex as they really only have to worry about compile-time features, has worked very well for them and allowed them to confidently introduce new features faster and allow them to mature into either a stable or abandoned feature over time.
Some of my thoughts on potential areas to explore more:
code:load_binary/3 with an input that has compile-time features enabled that the runtime doesn’t currently support?gen_server, what should enabling/disabling the experimental module look like?tokio Feature Flags).nightly, beta, and stable versions of Rust are released). For example: stable release of Erlang/OTP could be built with all experimental features explicitly disabled for both compile-time and runtime.As noted in the section Implementaton Notes, when compiling a module with features enabled, these features are recorded in a new chunk containing meta data. If the corresponding feature(s) are not enabled at runtime (using the -enable-feature option to erl) code:load_binary/3 will return an error indication that the load is not allowed.
Also, as noted in the section Backwards Compatibility, a module compiled with features enabled can be loaded into an older OTP release, as there is no checking of the new chunk. There might be other issues preventing loading and/or running, though, such as using an opcode that does not exist.
Minor note: the section Preventing that Experimental Features are used too Much in Production Code you refer to belongs to the appendix and is present as reference, being the initial report on this.
See above regarding how a feature is enabled at runtime. This would be valid for pure runtime only features as well, but is admittedly currently not well covered in the EEP. For global swapping of the sort you mention, that would rather be a property of how your runtime is built and started,e.g., with different load paths, and I’m not really sure that should be accomplished by an option to the runtime. While not expanded in the EEP the sometimes mentioned module_alias feature was envisioned for local module swapping, but that would then be a compile time feature as well.
Could you please expand on this? I’m not quite sure I understand what you mean.
Again, not really sure what this means and I am not current on Rust to quickly understand the link. Is this about the possibility for a non-Erlang/OTP lib to define a feature set of their own? That is not within the scope of the current EEP. If, however, this is about using the features provided by Erlang/OTP then it should be more or less a compile, build and artifact handling issue. I’ll admit to hand waving here since I really don’t understand what you are after.
It should be noted that there is support both in the compiler and runtime to enable/disable all features that can be controlled.
I was typing up a whole thing but then thought better of it, the rust reference talks about them well:
https://doc.rust-lang.org/cargo/reference/features.html
In short, if a library exposes a feature then it define it in its build file (format is the same as any identifer), so like:
[features]
blah = []
This does nothing on its own other then just keep things from getting an error if they try to enable the blah feature on your library or the compile command line.
There are a few ways you can use it though, such as marking a dependency as optional:
[dependencies]
some_dep = { version = "1.2", optional = true }
[features]
blah = ["some_dep"]
Every dependency becomes an internal feature name as well, but here I marked the some_dep dependency as an optional dependency, so it isn’t compile in or enabled or anything by default, however the blah feature depends on the some_dep feature (dependency in this case) so enabling blah will enable some_dep to be downloaded and compiled in.
The other main big use of features is for conditional code, you can do something like:
if cfg!(feature = "blah") {
...
} else {
...
}
Where the cfg macro becomes either true or false at compile-time depending on if the feature is enabled or not, all branches are still compiled as normal code though so you still can’t, for example, reference some_def in it and expect it to compile, these are for just simple things, the other way to do conditional compilation is by making it an attribute on something, here’s three examples:
#[cfg(feature = "blah")]
use some_dep::Vwoop; // This `use` only exists if `blah` is enabled
#[cfg(feature = "blah")] // This entire function only exists if `blah` is enabled
pub fn some_func() -> i32 {
let v = get_something();
Vwoop::calculate(v)
}
#[cfg(not(feature = "blah"))] // This entire function only exists if `blah` is NOT enabled
pub fn some_func() -> i32 {
16
}
// Another way to do the same as the above:
pub fn another_func() -> i32 {
#[cfg(feature = "blah")] // Putting this attribute around a whole block
{
let v = get_something();
Vwoop::calculate(v)
}
#[cfg(not(feature = "blah"))]
{
16
}
}
In the last one only one or the other block will be compiled in. But features allow for not only conditional compilation but also optional dependency inclusion.
Features are completely additive, so once anything enables the blah feature on the above library then it will be enabled in that library for all users of it in the entire project, no matter how many times libraries may include it, and once enabled it can’t be disabled (except for the “default” feature name can be requested to be disabled, but if its explicitly enabled then it stays enabled regardless). Thus features are additive, they only add functionality, they must never ever remove functionality and they should not ever be contradicting (where you can only enable a subset of a set of features, though some libraries do that by just throwing a compile-time error if someone does that even though it’s generally regarded as bad form and they should instead either make more features for each combination or split it up in to more libraries, thankfully it’s rare).
Features can also be exposed by the compile at the language level (the standard library uses features like any other library and those can be requested or not as normal), but these are enabled or disabled differently by adding a top-level module attribute that forces it to be enabled (which can of course be in a cfg attribute if you want it conditional) and so the compiler enables the feature for just that file.
There’s also unstable build system features defined in the Cargo.toml file that can be enabled from within the file on nightly too.
Enabled a language features might enable “access” to a standard library function as an example (in addition to enhanced language functionality) but that function was always there and always compiled in (as it’s file mandates the feature, but you just can’t use it without enabling the feature in your file as well).
The extra language level features cannot be enabled at all in a stable build, they are strictly that, unstable test features, they are expected to be unfinished and broken (though maybe sufficient for a use here or there, but then you have to compile on nightly and probably want to lock to a specific nightly version), but they are extremely useful for testing. Once it’s fully stabilized and complete then the feature is technically kept in by added to the “stable” features, which are always enabled at all times, including in stable builds.
Thanks for both the write up and the link. So, in short, this is the possibility of user defined features for third party libs. Yes, this would be possible to implement without too much problems. I’d probably opt for having it separate from the features of this EEP, though, since the user defined features is about the behavior/implementation of the lib, but not about changes to grammar, syntax and/or properties of the OTP runtime. It could still piggyback on the existing mechanism, but use other options. Also, one would probably want to construct a naming scheme to avoid collisions both with internal features and features of other libs.
No promises that it will happen now, though, but I do like the idea.
Yes, in addition to new language level features (that are only compiled into nightly builds before being forced on in release builds once it passes all tests and has been tested by the community enough in those cases, but that’s all features that third party libs can even do themselves if they want, it’s very usable not just by the language core but by third party code, just not for syntax extensions in third party code).
+1
Rust has no such collision as syntax level compiler features are forced required by #![feature(doc_cfg)] for example (though you can make that exist or not depending on if one of your own features are set, so if someone enabled a feature you set then it can enable that, which then means that a nightly compiler is then required), where library features are restricted to within the library (you can’t “test” another libraries features, instead you need to define your own feature that then requires that other library’s features, so if your feature is enabled it transitively enables that other libraries features, in addition to whatever code you specify it on in your own library, remember features are additive so nothing ‘extra’ should ever be done by a missing feature, they should only add functionality).
I like the idea.
I am relatively new to the Erlang community, so I do not have much to add about specifics or implementation.
However, I do have some comments and questions about the EEP itself.
The first question is regarding the prefix of a file introduced in New -feature(enable|disable, <feature>) directive.
The concept is explained very well, but it was not clear to me whether this is an existing concept which is expanded upon, or an entirely new concept introduced by the EEP.
I am not familiar with the standards for an EEP, so this may be caused by my own ignorance. However, if it is an existing concept I would suggest referring to some material where it is possible to learn more about it.
If on the other hand it is a new concept, I would suggest expanding on the motivation and the reasoning behind it.
I also have a comment about the newly added Meta chunk.
If is only meant to contain information on features, I would suggest a more specific name.
If this is not the case I am curious to know if this newly added chunk is intended to be used to store other information as well.
I look forward to hearing your thoughts.
This is a new concept introduced with EEP. I guess it has existed informally before, e.g., limiting where user defined attributes are allowed, but here I wanted to make it somewhat formal and precise. Making this more precise is a good idea for the documentation.
Currently, the new chunk only holds information about the features used when compiling the module, it is perfectly possible to add more data to it later. Naming is one of the more difficult problems we face in our area, and especially so when having limited space. In this chunk names can only be 4 characters, so Meta captures the use quite well. One could use Info as well. As a side note, other existing chunks in the BEAM file are named Abst, Dbgi, Attr, CInf, LocT, Atom and Type.
Actually, CInf contains compile time info, so we will add the features info there as well. The Meta chunk will still be needed, though, since it is needed when loading to check if we are allowed to load the module.