How to get recvtos working on Arch Linux

Questions / Help
1

I’m encountering an issue with OTP 27 on Arch Linux. I can’t replicate it on Debian/Ubuntu and I’m wondering if someone can point me in the right direction to solve it. I need to use a kernel socket option and for some reason it reports it is not supported:

❯ erl -noshell -eval 'io:format("~p", [socket:is_supported(options, ip, recvtos)]),init:stop().'
false
❯ docker run --rm -it erlang:27.1 erl -noshell -eval 'io:format("~p", [socket:is_supported(options, ip, recvtos)]),init:stop().'
true

I compiled & installed OTP using asdf. Using docker it works like expected. I’m fine with digging deeper, but I’d need a little bit of advice where to dig :slightly_smiling_face:

1 Like
2

I’d look into what the OTP VM does for this call, and double-check that the needed support (header files, manifest constants) was present at OTP compile-time.
If OTP does the check at runtime, check if libc fails immediately or if it’s the kernel that signals failure. A combination of printf() and strace should give you that.

3

The socket nif only tests for if IP_RECVTOS is known (defined).

This prints a list of all the (“known”) ip options and if they are supported or not:

io:format(“~p~n”, [socket:supports(options, ip)]).

Also, if its ok in your case, try do the update with debug enabled:

socket:debug(true), % Enable global (socket-) debug
socket:setopt(Sock, {otp, debug}, true). % Enable debug for you socket
socket:setopt(Sock, {ip, recvtos}, Value), % Try set value
socket:setopt(Sock, {otp, debug}, false). % Disable debug for you socket
socket:debug(false), % Disable global (socket-) debug

Finally, you could try the really dirty approach, just to test if its actually supported,
by using the integer value for the option. On my Ubuntu its 13, but I do not know
if its a different value on Arch.

find /usr/include -name ‘*.h’ | xargs grep IP_RECVTOS

Then

socket:setopt(Sock, ip, 13, Value).

4

socket:supports/2 prints an empty list on my host. Running erl in docker it does print the options and recvtos is supported.

Here is the output of the other steps (which with erl in docker work):

1> {ok, Sock} = socket:open(inet, dgram, udp, #{port => 5557}).
{ok,{'$socket',#Ref<0.2770375395.3382312969.61217>}}
2> socket:debug(true).
SOCKET [08-Apr-2025::11:52:42.308057] [unknown] nif_command -> done with result:
   ok
ok
3> socket:setopt(Sock, otp, debug, true).
SOCKET [08-Apr-2025::11:52:56.628972] [unknown] nif_setopt -> entry with argc: 5
   esock:  #Ref<0.2770375395.3382312969.61217>
   elevel: otp
   eopt:   1001
   eval:   true
   enval:  0
SOCKET [08-Apr-2025::11:52:56.630026] [unknown] esock_setopt_otp_debug {20} -> ok
   eVal: true
ok
4> socket:setopt(Sock, ip, recvtos, true).
{error,{invalid,{socket_option,{ip,recvtos}}}}
5> socket:setopt(Sock, ip, 13, true).
SOCKET [08-Apr-2025::11:53:40.458510] [unknown] esock_down {20} -> entry with
   pid:   <0.89.0>
   Close: false (false)
UNIX-ESSIO [08-Apr-2025::11:53:40.458768] [unknown] essio_down {20} -> controller process exit
   initiate close
UNIX-ESSIO [08-Apr-2025::11:53:40.458920] [unknown] essio_down_ctrl {20} ->
   Pid: <0.89.0>
SOCKET [08-Apr-2025::11:53:40.459047] [unknown] esock_close_socket(20) -> try socket close
SOCKET [08-Apr-2025::11:53:40.459389] [unknown] env free - esock_msg_send - msg-env: 0x0
SOCKET [08-Apr-2025::11:53:40.459533] [unknown] esock_down -> done
** exception throw: {invalid,{socket_option,{ip,13}}}
     in function  prim_socket:invalid/1
     in call from prim_socket:setopt_common/4

For me it was also 13 as can be seen below:

❯ find /usr/include -name '*.h' | xargs grep IP_RECVTOS
/usr/include/linux/in.h:#define	IP_RECVTOS	13
/usr/include/bits/in.h:#define IP_RECVTOS	13	/* bool */
5

On my Ubunu:

$ find /usr/include -type f | xargs fgrep IP_RECVTOS
/usr/include/x86_64-linux-gnu/bits/in.h:#define IP_RECVTOS	13	/* bool */
/usr/include/linux/in.h:#define	IP_RECVTOS	13
/usr/include/i386-linux-gnu/bits/in.h:#define IP_RECVTOS	13	/* bool */

The file included by prim_socket_nif.c:

$ grep 'in[.]h' /usr/include/netinet/in.h
#include <bits/in.h>
#endif	/* netinet/in.h */

So the socket NIF includes netinet/in.h, that includes bits/in.h, then by compiler header magic it resolves to x86_64-linux-gnu/bits/in.h through the compile target.

Something seems to go wrong on your platform. The socket NIF includes netinet/in.h, and by that it should find IP_RECVTOS. Are target specific header files missing?

6

I think they’re not missing, it’s just that the target doesn’t seem to be in the path on my system. Shouldn’t the compiler be able to figure that out anyway? Or is this a mismatch between Arch & Ubuntu/Debian?

❯ grep 'in[.]h' /usr/include/netinet/in.h
#include <bits/in.h>
#endif	/* netinet/in.h */
7

I guess that the compiler has some include path that depends on the target. How that works and how it is set up (per system / installation) I don’t know. It usually “just works”.

8

To confirm the recvtos option is working I asked a computer to write some code for me to test it. It’s in a gist. This code works fine.

Running configure, I see it actually finds netinet/in.h. I’m also used to the fact it “just works” :smile:

9

As it turns out, when you use a “native opt” you have to encode the value yourself and
pass it as a binary (socket does not know how to encode the value).

10

socket:setopt_native/3 takes a Value argument as either an integer, boolean, or binary. It assumes that Value is of the correct type and encodes it, for example: socket:setopt_native(Sock, {ip,13}, true).

11

I was on the same track and indeed using the native function it works as expected:

1> {ok, Sock} = socket:open(inet, dgram, udp, #{port => 5557}).
{ok,{'$socket',#Ref<0.3356808869.2251948033.19868>}}
2> socket:setopt_native(Sock, {ip, 13}, true).
[pid 693596] setsockopt(20, SOL_IP, IP_RECVTOS, [1], 4) = 0
[pid 693596] setsockopt(20, SOL_IP, IP_TOS, [0], 4) = 0
[pid 693596] setsockopt(20, SOL_SOCKET, SO_PRIORITY, [0], 4) = 02> socket:setopt_native(Sock, {ip, 13}, true).
ok
3>

I feel it doesn’t really solve the problem though. I’d have to get this upstream whereas it doesn’t pose an issue on other architectures.

12

I did some more work on this. I found the option is seen by the compiler and there is a different reason why it doesn’t work like expected. On Arch IPPROTO_IP is 0, same as on Debian. However IPPROTO_HOPOPTS also is 0. I confirmed by testing it:

1> {ok, Sock} = socket:open(inet, dgram, udp, #{port => 5557}).
{ok,{'$socket',#Ref<0.521348907.360316931.236634>}}
2> socket:setopt(Sock, {hopopt, recvtos}, true).
ok

On Debian (truncated):

ip	0	IP		# internet protocol, pseudo protocol number
hopopt	0	HOPOPT		# IPv6 Hop-by-Hop Option [RFC1883]

On Arch (truncated):

❯ grep 0 /etc/protocols
hopopt         0 HOPOPT

After adding ip 0 IP in /etc/protocols before hopopt everything works as expected.

Thanks for helping, I really appreciate it!

1 Like
13

Interesting enough IANA doesn’t have 0 as IP either: Protocol Numbers

14

Interesting!

What does the following produce on your system with and without your workaround?

6> maps:get(ip, prim_socket:p_get(protocols)).
0
7> maps:get(0, prim_socket:p_get(protocols)).
[ip,'IP']
15

Well, these protocols are protocols transported on the IP protocol, right? So the IP protocol itself cannot have an IP protocol number. But you need a number to specify the IP protocol level for e.g setsockopt().

You also need a number to specify the Socket level, which is a layer over the IP protocols; the libc “user” layer. Linux has it as 1, which collides with ICMP. FreeBSD has it as 0xffff.

16

With the workaround I get the same result as you. Without the workaround I get:

1> maps:get(ip, prim_socket:p_get(protocols)).
0
2> maps:get(0, prim_socket:p_get(protocols)).
[hopopt,'HOPOPT']

Though, I’m not sure I’d label it as workaround. It’s a difference between distro’s.

17

Well, these protocols are protocols transported on the IP protocol, right? So the IP protocol itself cannot have an IP protocol number. But you need a number to specify the IP protocol level for e.g setsockopt().

That makes sense. Though we could reason that the default (0 or ip) should not be relied upon /etc/protocols but translated to the correct value by otp.

18

The socket NIF actually does not rely on /etc/protocols, or rather that getprotoent() enumerates the essential protocols. (getprotoent() probably just loops over /etc/protocols, but it could be implemented smarter)

The protocols ip, ipv6, tcp, udp, sctp, rm and igmp have fallbacks that one can see in that ip gets translated to 0 even when 0 gets translated to hopopt.

Therefore it seems that the socket NIF translates ip correctly into 0.

But I think the fallback implementation is a bit flawed. If getprotoent() enumerates a protocol it overrides the fallback, so if getprotoent only enumerates hopopt for 0 it is lost that ip is an alias.

Does socket:setopt(S, {hopopt,recvtos}, true) work?

19

Yes that works, I used it in an earlier post

Therefore it seems that the socket NIF translates ip correctly into 0.

Yes, except for the way the ip options are stored as persistent term. This is actually the only place I can think of it’s a bit flawed. When checking if recvtos is a valid ip option it’s being checked on the atom ip and not on 0. And while the nif returns recvtos as an option on 0 it is being put in persistent term storage as hopopt.

20

Sorry, I had a vague memory of that earlier post but didn’t find it.

All valid combinations are cached in the persistent term to require just a single lookup for socket:setopt/3. So in this case we should have #{ {hopopt,recvtos} := {0,13}, 'HOPOPT' := {0,13} } and are missing the fallback {ip,recvtos} := {0,13} because it was lost that ip also is an alias for 0.

This is to avoid first looking up Level to NumLevel and then looking up {NumLevel,Opt} to NumOpt.

1 Like