OTP 25.3 Nodes Permanently Locked at 100% CPU: timerfd_settime Dominates (48.5%) in Frame-Sync Game with 33ms Timers

Questions / Help
,
1

Environment: 8-core 64GB server, Erlang 25.3, 30+ frame-synchronized game nodes.
Symptoms:

Stable CPU ≈ 3% initially
Three nodes spiked to 100% CPU at 17:22/19:20/next day 12:30 and never recovered (unaffected by player decrease).
Process reds normal; business logic checks revealed no issues.
Key Data:

VTune hotspots:
timerfd_settime 48.5%
epoll_wait 23.5%
check_fd_events 7.7%
Frame-sync timers:
% High-frequency micro-timers (33-44ms)
erlang:send_after(33, Pid, {frame_loop})
Network config:
{nodelay, true}, {delay_send, false} % Small-packet direct send
Core Questions:

Why does high-frequency send_after cause permanent 100% CPU?
Do timerfd_settime + epoll_wait dominance indicate timer/I/O model flaws?
Are there known OTP 25.3 issues related to this?
Request:

Root cause analysis for CPU lock
Optimization strategies for frame-sync timers
High-performance alternatives to timerfd_settime

2

vtune: Executing actions 75 % Generating a report Elapsed Time: 7.200s
CPU Time: 7.540s
Effective Time: 7.530s
Spin Time: 0.010s
Overhead Time: 0s
Total Thread Count: 10
Paused Time: 0s

Top Hotspots
Function Module CPU Time % of CPU Time(%)

timerfd_settime libc.so.6 3.660s 48.5%
epoll_wait libc.so.6 1.769s 23.5%
check_fd_events beam.smp 0.582s 7.7%
ethr_native_atomic32_read beam.smp 0.140s 1.9%
ethr_rwmutex_runlock beam.smp 0.130s 1.7%
[Others] N/A 1.259s 16.7%

top - 16:12:26 up 660 days, 20:34, 3 users, load average: 6.86, 7.14, 6.86
Tasks: 561 total, 2 running, 559 sleeping, 0 stopped, 0 zombie
%Cpu0 : 23.8 us, 42.6 sy, 0.0 ni, 32.3 id, 0.3 wa, 0.0 hi, 1.0 si, 0.0 st
%Cpu1 : 25.0 us, 37.2 sy, 0.0 ni, 36.8 id, 0.3 wa, 0.0 hi, 0.7 si, 0.0 st
%Cpu2 : 24.8 us, 43.7 sy, 0.0 ni, 29.1 id, 1.7 wa, 0.0 hi, 0.7 si, 0.0 st
%Cpu3 : 26.0 us, 40.1 sy, 0.0 ni, 31.9 id, 1.3 wa, 0.0 hi, 0.7 si, 0.0 st
%Cpu4 : 26.2 us, 41.6 sy, 0.0 ni, 31.1 id, 0.3 wa, 0.0 hi, 0.7 si, 0.0 st
%Cpu5 : 25.6 us, 45.5 sy, 0.0 ni, 27.6 id, 1.0 wa, 0.0 hi, 0.3 si, 0.0 st
%Cpu6 : 23.5 us, 38.1 sy, 0.0 ni, 36.1 id, 1.3 wa, 0.0 hi, 1.0 si, 0.0 st
%Cpu7 : 27.8 us, 47.4 sy, 0.0 ni, 23.5 id, 1.0 wa, 0.0 hi, 0.3 si, 0.0 st
11210 xxxx 20 0 5748492 1.3g 15644 S 111.6 2.1 634:25.37 beam.smp
8530 xxxx 20 0 4447160 699848 5140 S 102.6 1.1 92:35.70 beam.smp
9881 xxxx 20 0 5281592 1.1g 15636 S 101.0 1.8 1791:21 beam.smp
11078 xxxx 20 0 5238632 1.2g 5148 S 4.7 2.0 255:45.64 beam.smp

msacc:start(), timer:sleep(10000), msacc:stop(), msacc:print().
Average thread real-time : 20002492 us
Accumulated system run-time : 3549078 us
Average scheduler run-time : 440553 us

    Thread      aux check_io emulator       gc    other     port    sleep

Stats per thread:
async( 0) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00%
aux( 1) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00%
dirty_cpu_( 1) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00%
dirty_cpu_( 2) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00%
dirty_cpu_( 3) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00%
dirty_cpu_( 4) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00%
dirty_cpu_( 5) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00%
dirty_cpu_( 6) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00%
dirty_cpu_( 7) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00%
dirty_cpu_( 8) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00%
dirty_io_s( 1) 0.00% 0.00% 0.00% 0.00% 0.01% 0.00% 99.99%
dirty_io_s( 2) 0.00% 0.00% 0.00% 0.00% 0.01% 0.00% 99.99%
dirty_io_s( 3) 0.00% 0.00% 0.00% 0.00% 0.01% 0.00% 99.99%
dirty_io_s( 4) 0.00% 0.00% 0.00% 0.00% 0.02% 0.00% 99.98%
dirty_io_s( 5) 0.00% 0.00% 0.00% 0.00% 0.01% 0.00% 99.98%
dirty_io_s( 6) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00%
dirty_io_s( 7) 0.00% 0.00% 0.00% 0.00% 0.01% 0.00% 99.99%
dirty_io_s( 8) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00%
dirty_io_s( 9) 0.00% 0.00% 0.00% 0.00% 0.03% 0.00% 99.97%
dirty_io_s(10) 0.00% 0.00% 0.00% 0.00% 0.01% 0.00% 99.99%
poll( 0) 0.00% 0.01% 0.00% 0.00% 0.00% 0.00% 99.99%
scheduler( 1) 0.01% 3.14% 0.72% 0.02% 6.00% 0.10% 90.01%
scheduler( 2) 0.01% 1.51% 0.20% 0.01% 2.83% 0.04% 95.39%
scheduler( 3) 0.01% 0.72% 0.14% 0.01% 1.38% 0.02% 97.72%
scheduler( 4) 0.00% 0.06% 0.01% 0.00% 0.11% 0.00% 99.83%
scheduler( 5) 0.00% 0.00% 0.00% 0.00% 0.01% 0.00% 99.98%
scheduler( 6) 0.00% 0.12% 0.05% 0.00% 0.23% 0.01% 99.58%
scheduler( 7) 0.00% 0.04% 0.00% 0.00% 0.08% 0.00% 99.87%
scheduler( 8) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00%

Stats per type:
async 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00%
aux 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00%
dirty_cpu_sche 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00%
dirty_io_sched 0.00% 0.00% 0.00% 0.00% 0.01% 0.00% 99.99%
poll 0.00% 0.01% 0.00% 0.00% 0.00% 0.00% 99.99%
scheduler 0.00% 0.70% 0.14% 0.00% 1.33% 0.02% 97.80%

|Home(H)|Network(N)|System(S)|Ets(E)|Mnesia(M)|App(A)|Doc(D)|Plugin(P)|recon:proc_window(reductions, 40, 1500) Interva | 2Days 07:08:35 |
|Erlang/OTP 25 [erts-13.2.2.5] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] xxx@sxxxxgame.com |
|System | Count/Limit | System | Status | Stat Info | Size |
|Proc Count | 1044/262144 | Version | 25 | Active Task | 1 |
|Port Count | 83/131072 | ps -o pcpu | 84.3% | Context Switch | 197715046 |
|Atom Count | 57805/1048576 | ps -o pmem | 1.4% | Reds(Total/SinceLast | 136585645307/1556429 |
|Mem Type | Size | Mem Type | Size | IO/GC:(1500ms) | Total/Increments |
|Total | 1.1710 GiB | 100.0% | Binary | 44.4894 MiB | 03.71% | IO Output | 4.3991 GiB/33.0498 KiB |
|Process | 779.6087 MiB | 65.02% | Code | 49.8805 MiB | 04.16% | IO Input | 1.3118 GiB/7.9688 KiB |
|Atom | 2.1244 MiB | 00.18% | Port Parallelism (+spp) | false | Gc Count | 9178202/95 |
|Ets/361 | 295.3850 MiB | 24.63% | RunQueue/ErrorLoggerQueue | 0/0 | Gc Words Reclaimed | 98912029143/1595296 |
|No | Pid | Reductions |Name|>Label|>Initial Call | Memory | MsgQueue |Current Function |
|1 |<0.932.0> |803185 |rankings_mgr |192.3828 KiB| 0 |gen_server:loop/7 |
|2 |<0.26963.43>|73124 |role_worker:init/1 | 2.7851 MiB| 0 |gen_server:loop/7 |
|3 |<0.527.0> |23439 |server_cache_worker:init/1 | 4.5019 MiB| 0 |gen_server:loop/7 |
|4 |<0.2751.44> |18415 |battle_worker:init/1 |279.2031 KiB| 0 |gen_server:loop/7 |
|5 |<0.2903.44> |16210 |battle_worker:init/1 |107.9219 KiB| 0 |gen_server:loop/7 |