The crash only occurs when using qemu user mode emulation (by just running ./bin/erl -emu_flavor jit). In that mode debugging like you suggested does not work at all because ptrace is not supported. I tried using qemu gdbstub, but that does also just segfaults.
In qemu system mode the jit version that was compile within that VM works. When I try to run the binaries produced in the VM in qemu user mode, it goes to 100% CPU load and just does nothing (no CLI prompt, no other output).
The binaries compiled with the user mode emulation work fine when moving them into the VM image and running them under system emulation.
The root images for both version are the same. So the tooling for both builds was identical.
The host CPU is a Intel Core 7, so nothing spectacular.
QEMU is 6.2 (Ubuntu Jammy host) and Linux 5.17.5-051705-generic.
Reproducing this should be simple. Enable QEMU user mode emulation for aarch64, grab the Alpine 3.15.4 AArch64 root tar, extranct, chroot to it and build otp in it.
I’m going to try this with a Debian AArch64 root to see if there is difference in muslc vs. glibc based systems.
Thanks, I’ve found a bug in QEMU that explains this. The gist of it is:
Instead of interpreting guest code, QEMU dynamically translates it to the host architecture. When the guest overwrites code for one reason or another, the translation is invalidated and redone if needed.
Our JIT:ed code is mapped in two regions to work in the face of W^X restrictions: one executable but not writable, and one writable but not executable. Both of these regions point to the same physical memory and writes to the writable region are “magically” reflected in the executable one.
I would’ve expected QEMU to honor the IC IVAU / ISB instructions we use to tell the processor that we’ve altered code at a particular address, but for some reason QEMU just ignores them and relies entirely on trapping writes to previously translated code.
In system mode QEMU emulates the MMU and sees that these two regions point at the same memory, and has no problem invalidating the executable region after writing to the writable region.
In user mode it instead calls mprotect(..., PROT_READ) on all code regions it has translated, and invalidates translations in the signal handler. The problem is that we never write to the executable region – just the writable one – so the code doesn’t get invalidated.
Sure, if it runs quickly enough I’m open to adding it once they fix this bug.
I want to try maybe_expr feature introduced in OTP 25.0. I installed 25.0-rc3 locally using kerl and when I run r3 compile or r3 shell I get no errors at all. However, when I try to run (in rebar3 shell, OTP 25, ERTS v13.0) exported functions from module using maybe expression I get the following error:
Loading of /home/marko/<project_root>/_build/default/lib/<project_name>/ebin/pfun.beam failed: not_allowed ** exception error: undefined function pfun:<fun_i'm_trying_to_run>.
In pfun module I’ve got the following attributes: -feature(maybe_expr, enable). -compile({feature, maybe_expr, enable}).
Also, erl_opts in rebar.config contains {feature, maybe_expr, enable}.
Any idea of what is wrong here?
The option -enable-feature maybe_expr has to be given to the runtime to allowing loading modules that have been compiled with the feature enabled. This is to make an active decision in allowing experimental features.
While erl_features:enable_feature/1 is available in RC3, it will not be available in the final release of OTP25. Enabling features in the runtime is only allowed during startup and changing (enabling/disabling) is not allowed thereafter.
Full documentation for the features support will be included with OTP25.
Note that you only need one of these and using the -feature directive is the preferred use. To be honest, while a -compile attribute is allowed anywhere in a file, the enabling and disabling of features is only allowed in a prefix of the file, so using -compile to enable/disable a feature anywhere might have unexpected results, i.e., I have to test and fix this.