Erlang 29 built with -O2 on MIPS64(n32) crashes

Questions / Help
,
1

Have you ever found the following situation with Erlang/OTP?

I’m not asking about the same platform that I’m using, but the symptoms related to -Ox GCC flag.

Platform: MIPS64 ABI n32
Erlang 29 (maint-29 branch)

GCC version 6.4.0 (CFLAGS and CXXFLAGS: “-meb -mabi=n32 -mhard-float -march=mips64r2”)

Build was ok and erl worked as expected when running on qemu-system and qemu-user.

GCC version 13.4.0 (default build)

By default erlang build uses -O2 -g

export CFLAGS="-meb -mabi=n32 -mhard-float -march=mips64r2"
export CXXFLAGS="-meb -mabi=n32 -mhard-float -march=mips64r2"
./configure && make

Build was ok. But erl crashed when qemu-user mode:

qemu: uncaught target signal 10 (Bus error) - core dumped
Bus error (core dumped)

GCC version 13.4.0 (with -O0)

Build and run were ok:

export CFLAGS="-meb -mabi=n32 -mhard-float -march=mips64r2 -O0 -g -fno-omit-frame-pointer"
export CXXFLAGS="-meb -mabi=n32 -mhard-float -march=mips64r2 -O0 -g -fno-omit-frame-pointer"
./configure && make

GCC version 13.4.0 (with -O1)

Build and run were ok:

export CFLAGS="-meb -mabi=n32 -mhard-float -march=mips64r2 -O1 -g -fno-omit-frame-pointer"
export CXXFLAGS="-meb -mabi=n32 -mhard-float -march=mips64r2 -O1 -g -fno-omit-frame-pointer"
./configure && make

On other hand, when using a real MIPS hardware like -march=octeon3 Erlang/OTP runs fine on the target platform.

It might sounds like some GCC optimization is not ok for MIPS. While this is a niche case, my point is to try to identify where it is and how the bug happens; because it could trigger other issues for other platforms too.

I’ll post some results for Erlang 27, 28 builds.

Thanks.

2

You might try building and running a debug build? If you’re lucky an assert will give you more information and help you track it down.

See “How to Build a Debug Enabled Erlang RunTime System” in the build instructions: Building and Installing Erlang/OTP — Erlang System Documentation v29.0.4

3

Hi,
Sure, that will help. But I could not get even the bootstrap erl working. As I’m not making cross build but native build (inside qemu-user mode). Maybe I could switch to cross build, then get a erlang runtime build with debug support.

4

Hi,

After cross building OTP29 with TYPE=debug, Erlang VM worked. Then I rebuild with TYPE=asan then I got the following trace (it could be a false positive one):

bin/erl -emu_type asan
=================================================================
==315129==ERROR: AddressSanitizer: unknown-crash on address 0x2c809e20 at pc 0x5567bbc0 bp 0x2b2aaab0 sp 0x2b2aaab8
WRITE of size 4 at 0x2c809e20 thread T0
    #0 0x5567bbb8 in beam_load_emit_op beam/emu/emu_load.c:1131
    #1 0x556c1b4c in load_code beam/beam_load.c:650
    #2 0x556c31f0 in erts_prepare_loading beam/beam_load.c:227
    #3 0x556c3370 in erts_preload_module beam/beam_load.c:114
    #4 0x557b63f4 in load_preloaded beam/erl_init.c:528
    #5 0x557bb308 in erl_start beam/erl_init.c:2534
    #6 0x555be03c in main sys/unix/erl_main.c:35
    #7 0x2bb26054  (/lib/libc.so.6+0x26054) (BuildId: 377579b50e14ecd5db89c5ee788fdbd6aada7583)
    #8 0x2bb26180 in __libc_start_main (/lib/libc.so.6+0x26180) (BuildId: 377579b50e14ecd5db89c5ee788fdbd6aada7583)
    #9 0x555bdeb8  (/home/joafreit/xyz/code/otp-29-mips64-cross/RELEASE/erts-17.0.3/bin/beam.asan.smp+0x6deb8) (BuildId: d1ef77f0094fff5b610a50b3af71f5855843ef1f)

Address 0x2c809e20 is located in stack of thread T0 at offset 32 in frame
    #0 0x55678b4c in beam_load_emit_op beam/emu/emu_load.c:879

  This frame has 1 object(s):
    [32, 96) 'stack' (line 1122) <== Memory access at offset 32 is inside this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
      (longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: unknown-crash beam/emu/emu_load.c:1131 in beam_load_emit_op
Shadow bytes around the buggy address:
  0x2c809b80: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5
  0x2c809c00: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5
  0x2c809c80: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5
  0x2c809d00: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5
  0x2c809d80: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5
=>0x2c809e00: 00 00 00 00[00]00 00 00 00 00 00 00 00 00 00 00
  0x2c809e80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x2c809f00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x2c809f80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x2c80a000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x2c80a080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==315129==ABORTING

With ldd, it’s possible to see that beam.asan.smp uses libasan.

ldd ./erts-17.0.3/bin/beam.asan.smp
        libasan.so.8 => /usr/lib/libasan.so.8 (0x2c060000)
        libm.so.6 => /lib/libm.so.6 (0x2c6a0000)
        libtinfo.so.5 => /lib/libtinfo.so.5 (0x2c770000)
        libz.so.1 => /usr/lib/libz.so.1 (0x2c7c0000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x2c800000)
        libc.so.6 => /lib/libc.so.6 (0x2c860000)
        /lib32/ld.so.1 => /lib/ld.so.1 (0x55550000)
        librt.so.1 => /lib/librt.so.1 (0x2ca50000)
        libpthread.so.0 => /lib/libpthread.so.0 (0x2ca80000)
        libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x2cab0000)

file ./erts-17.0.3/bin/beam.asan.smp
./erts-17.0.3/bin/beam.asan.smp: ELF 32-bit MSB pie executable, MIPS, N32 MIPS64 rel2 version 1 (SYSV), dynamically linked, interpreter /lib32/ld.so.1, BuildID[sha1]=d1ef77f0094fff5b610a50b3af71f5855843ef1f, for GNU/Linux 5.15.0, with debug_info, not stripped