I posted this on the Elixir Forums. But I think I’ll find more answers here.
https://elixirforum.com/t/ethr-mutex-h-fatal-error-in-ethr-mutex-lock-invalid-argument-22-aborted-core-dumped/49762
What can I do to avoid seeing this error again? (Answer: Program in Pascal) What can I do to avoid seeing this error and still program in Elixir?
../include/internal/ethr_mutex.h:657: Fatal error in ethr_mutex_lock(): Invalid argument (22)
Aborted (core dumped)
I’m guessing it’s this Erlang file:Erlang/OTP: erts/include/internal/ethr_mutex.h | Fossies
652 static ETHR_INLINE void
653 ETHR_INLINE_MTX_FUNC_NAME_(ethr_mutex_lock)(ethr_mutex *mtx)
654 {
655 int res = pthread_mutex_lock(&mtx->pt_mtx);
656 if (res != 0)
657 ETHR_FATAL_ERROR__(res);
658 }
And it also looks like error code 22 is EINVAL
EINVAL The mutex was created with the protocol attribute having
the value PTHREAD_PRIO_PROTECT and the calling thread's
priority is higher than the mutex's current priority
ceiling.
https://man7.org/linux/man-pages/man3/pthread_mutex_lock.3p.html
Elixir/Erlang/OS specs:
elixir --version
Erlang/OTP 25 [erts-13.0.3] [source] [64-bit] [smp:128:128] [ds:128:128:10] [async-threads:1] [jit:ns]
Elixir 1.13.2 (compiled with Erlang/OTP 24)
uname -rmo
5.15.62-1-lts x86_64 GNU/Linux
Did I run out of resources? Should I stop doing things like max_connections: (:erlang.system_info(:logical_processors_available) * 4) or running async_stream() when I have LOTS of cores?
1 Like
How easy is it for you to reproduce the error? Could you provide a minimal version where I could reproduce it?
2 Likes
Difficult to reproduce. Sadly I haven’t seen it again. The system ran all night without any issues. I’ve reduced the number of TCP connections since it crashed the previous night. Both directly used by the client/server Elixir code AND I turned off some other heavy network traffic apps (torrents).
I’ll kick up the number of processes elixir is using to try to recreate it. But anytime I see a bug with threads… 
My understanding of ethr_mutex_lock is:
Ethr → Ethernet, maybe;
Mutex → Threads;
Lock → Be greedy, and puke if you don’t get your way
Besides increasing the number of TCP connections + processes running at random, can I do anything else to try and make it so “the calling thread’s priority is higher than the mutex’s current priority”. Ideally in a less random way.
1 Like
ethr stands for erlang threads, which is the erlang emulator’s own threading library which on many OSs is only a wrapper around pthread.
The ethread library does not create mutexes with PTHREAD_PRIO_PROTECT set.
The only time I’ve ever seen pthread_mutex_lock return an error was when a linux kernel bug caused the memory of the mutex to be overwritten. So my guess would be that there is some wild memory write happening somewhere.
I’m assuming that you do not have a linux core file from when this happened?
2 Likes
I only have one beam dump in /var/lib/systemd/coredump. But the date doesn’t seem correct. Unless my brain is having an off by one error, this dump was 24 hours too early. I’d be happy to put it online for you to look at if you wish. zgrep’ed it enough to know it was the project that crashed, and it doesn’t have any sensitive personal information. Wouldn’t want anyone to know my mothers maiden name is Silverman, my SSN is 943-84-9931 and I bank at Bailey Building and Loan. That would be bad.
I pulled a copy of it so systemd doesn’t rotate it out. Let me know if you want it and how. (1.4 G… I know it’s big. It’s not yet production code.)
-rw-r-----+ 1 root root 1.4G Aug 24 06:35 'core.beam\x2esmp.1000.2133636ae8e04b29943442afbd2ba380.1415242.1661344507000000.zst'
And thanks for the mini-lesson on Erlang threads. Makes me wish I keep up with my C programming. And at the same time, makes me glad I don’t have to program in C anymore. *ptr are powerful, but &suck too. Not to mention threading, shared memory, 
1 Like