What do you think should be in OTP?

Off the top of my head I can’t think of anything that should be in OTP.

On the other hand I can think of a lot of applications and modules that should NOT be (debugger dialyzer diameter edoc eldap et eunit ftp inets megaco mnesia observer odbc snmp ssh tftp wx xmerl, to name a few).

OTP should be the small core on top of which everything else is built, nothing else. It doesn’t make any sense to have the entire world inside OTP directly.

9 Likes

By this reason was created topic What do you think should not be in OTP? :upside_down_face:

2 Likes

What are you imagining here? :thinking: Something like gen_statem timers, but without gen_statem maybe?

I mean, we do have the timer module. It got a big overhaul under the hood, which will appear in OTP 25 I guess. In a nutshell, it should be a lot less prone to overload and becoming a bottleneck.

But while timer offers some conveniences like interval timers and apply_after, exit_after, kill_after on top of send_after, it doesn’t offer much in terms of timer management, so I think this is not what you are looking for?

4 Likes

I am commenting in the correct topic. My response to this topic is “nothing should be added to OTP, only things should be removed from it”. The list of things I would remove is a good example of the type of things we shouldn’t add.

There shouldn’t be two topics for this anyway. The premise should be discussed first, aka “Should anything be added to OTP?” or “What should the scope of the OTP project be?”

6 Likes

I don’t have a clear picture in my mind yet. By the code I tend to repeat is usually smth like

reset_timer(#state{timer_ref = undefined, cfg_timeout = Timeout} = State) ->
    State#state{timer_ref = erlang:send_after(self(), timeout, Timeout)};
reset_timer(#state{timer_ref = Ref} = State) ->
    erlang:cancel_timer(Ref),
    State#state{timer_ref = undefined}.

and I have to call this function as last step of gen_server’s handle_* callbacks:

handle_call(..., State) ->
    ...
    {reply, Response, reset_timer(State)}.

It’s a lot of boilerplate IMO. And also it causes a lot of timer re-creation putting some pressure on the ERTS timer wheel if it happens on a hot path (but this is probably not an issue, I did no benchmarks).

But also I sometimes might want to have some kind of FSM of the timeouts - have different Timeout values and maybe timeout messages depending on which state my gen_server currently is (is in “connecting” state - timeout is 10s, in “handshake” state - timeout 60s, in “idle” state - timeout 10m).

I made an attempt to make some abstraction here / usage example but was not very happy with the result never had time to polish it.

4 Likes

I share the same opinion as @essen. For instance, adding Rebar3 into Erlang/OTP itself would be a downside to me: what if Rebar in Erlang/OTP 24 has more options & commands than its version in Erlang 23 because something was not backported or not released yet by the Erlang team?

When more things are bundled together inside Erlang, we would all have to make compromises somewhere between the ease and speed of maintenance/updates to various components and the release engineering of Erlang/OTP. It also adds new commitments and support load on the Erlang/OTP team.

So like @essen, I would vote for “nothing added” :slight_smile: That way, I can use Rebar or any JSON library and get the same behavior regardless of the version of Erlang/OTP and I don’t have to wait for OTP 25.0 to get a big feature. Of course, there is the problem of “curating the ecosystem”. We have multiple build systems (Rebar, Erlang.mk, …), multiple JSON libraries and so on. It’s more difficult for everyone, beginners in particular, to pick dependencies and tools.

6 Likes

For something to be part of OTP it must not have dependencies outside of OTP. rebar3 has a lot of dependencies where many of them are unnecessary in my opinion. For some there could be additions in OTP which would make them unnecessary. Something like getopt could for example be interesting to have in OTP.

7 Likes

Yup, everything would have to be vendored in and/or added to OTP. getopt would definitely be a good one IMO. @max-au’s argparse comes to mind.

3 Likes

For things like what rebar needs and uses, doesn’t mean it has to be in the standard library. Like Rusts’ cargo and crate ecosystem that comes with rust (cargo is the build system, crates are the dependency system that’s managed by cargo) has a lot of dependencies that are not in the standard library, but they get built along with just those programs (effectively vendered in).

3 Likes

Happy to contribute argparse/cli, although it will need some better testing. Can do the experimental PR to spin up a discussion.

One thing that should really be in OTP, some bootstrap to get the root CA. Or, have OTP to understand OS-provided CA store.

8 Likes

I think we should add support to use the OS provided ca store.

18 Likes

Well, one thing that annoys me a bit right now in our production system, is that we currently have two different JSON libraries pulled in by different dependencies (and one of them we also use directly).

In the past, we also had multiple logging libraries, which was a real pain. This got a lot better with the introduction of logger, and I’d hope introducing a common json module would have the same effect. It could (like logger) still be built in a configurable manner, but having well-defined interfaces(!) in OTP is a plus for me.

11 Likes

Erlang supporting to just read from the OS’s root CA store would likely be the best way to drastically increase security of all Erlang apps by making cert checking built-in, and following existing update mechanisms other applications can use.

11 Likes

It would be interesting if we have something like rustup that download components to the machine without compiling or downloading it individually by yourself. :slight_smile:

4 Likes

Oh yes, another huge plus for that! rustup is such an amazing tool in the rust ecosystem. It’s installable via a simple copyable curl .. | bash script (though they of course encourage you to look at it first if you want) or via most package managers now or grab it individually. What it does is manage the toolchains and targets for rust as well as the components of the toolchains. In essence:

  • Toolchain: A full build system for Rust, this includes the compiler, formatter, lots of other things even including the source code of the standard library or even the compiler itself (great for IDE introspection), each of those parts are a Component, so you’d add a component of, say, the standard library source code to a given toolchain. Toolchains can be things like:
    • stable a keyword for the latest stable version (built every 6 weeks with possible point releases in the interim) toolchain. The stable toolchain is the most stable form of the language, always backwards compatible, what everything should be built with by default (and is the only toolchain installed by default with rustup)
    • beta a keyword for the latest beta version (built weekly, updates to beta are pushed to stable when they have sufficient, usually months to a year or more of testing and usage, if not enough usage or bugs are encountered with the feature then the feature is removed, there is some churn in beta but not much).
    • nightly a keyword for the latest CI build, built every single day, can break on occasion but it lets you test the newest and greatest possible features (that may get promoted to beta eventually once feature-complete or will get removed if it is deemed not worth it or has some underlying issues discovered).
    • Or you can specify a precise version, in any of a stable, beta, or nightly formats as a semver string.
  • Targets: This is probably less useful to the erlang ecosystem (maybe?) but it’s the archetecture specific code, generally backend generation, in standard LLVM-triple-format, for example here right now I have x86_64-unknown-linux-gnu (64 bit general linux builds, currently set as default), x86_64-pc-windows-gnu (to cross-compile to windows 64 bit), and wasm32-unknown-unknown to cross-compile to wasm.

To specify a target or toolchain override in cargo (the rust build tool) you specif it on the commandline. To override the toolchain you specify it as the first option, where target is specified on a specific command that cares about targets, so to build from linux here to a native windows program using the nightly toolchain then something like cargo +nightly build --target=x86_64-pc-windows-gnu (where just cargo build would use the defaults you have set, so for me that would be stable and x86_64-unknown-linux-gnu, but of course I can change those if I want, or even specify unique options in the cargo build configuration files or all kinds of things).

But it makes it really really easy to jump in to rust, the rustup tool is for every system that is also a tier 1 (and a few tier 2) level build systems the same as rust itself.

5 Likes

Have you tried kerl? It felt quite powerful for me.

4 Likes

It’s not nearly as comfortable as rustup, though. It doesn’t run on Windows and requires a complete toolchain as it always builds, rustup will download binary releases.

4 Likes

I think we should add support to use the OS provided ca store

I can’t believe I never thought of this addition. What a fantastic idea, it would benefit most likely everyone in a meaningful way

5 Likes

The disadvantage of kerl is it requires the local machine to have a package/libraries to build the OTP. But the rustup idea is they load the pre-built binaries that you without any installation package/libraries to build them.

4 Likes

For logger:

  • TCP handler
  • UDP handler
  • HTTP handler
  • syslog formatter
  • Something like trace from Lager in the logger or merge telemetry into logger
  • Exported logger_olp or exposed overload protection in other way. I am working on enough library, but ideally it would be part of the kernel/stdlib

Other than that:

  • QUIC support
  • Argon2 support
  • systemd support for socket activation, status notifications, etc.
  • Maybe lanuchd socket activation support as well
  • Approach to fetch the information about the process that is OS specific, like UID, EUID, GID, EGID, etc.
  • Approach to fetch the information about basic sizes of the OS values like int size, which would greatly help with new socket implementation as not all cmsg_* values can be represented in the Erlang (as these differ between OSes) and building them “manually” without such information potentially can be dangerous
  • Create file:fd/0 form the OS provided FD (currently the only solution is to use ports)
  • Way to pass FD to the spawned process for better communication with spawned processes rather than relying only on the stdio
  • Allow to close the stdin in port without closing the stdout - this would greatly help with integration with tools like ImageMagick or other processors that wait for closing stdin before any work
13 Likes