I’ve spent a bit of time on the socket layer of OTP as I’ve been working on erlang DNS servers recently. You all know the classic behaviours, gen_tcp and gen_udp, but there’s also the much less common socket module, that gives a lot more granular power and lets you use kernel features like timestamping and whatnot. So I wanted to check how it’d behave performance-wise in comparison to the traditional modules, and notice there’s one feature not implemented: sendmmsg and recvmmssg, which help with accepting packets from the sockets in batches, therefore amortising the cost of switching from Erlang space to NIF space, and most importantly, from User space to Kernel space, and the traditional modules do have support for batching using {active, N}. So I decided to implement this for the socket module:
As in the little excerpt from the benchmark, for similar size batches gives a 50% improvement over gen_udp and it allows to use all the new functionality socket gives
Looking forward for feedback, support, or just thumbs up
Yes, you can call recvmmsg with nowait or a Ref as the timeout, and if there’s no messages available immediately it will return a select handle with a new ref or the one you provided, just like recvmsg does. In reality at the C level both functions call the kernel-level recvmsg/recvmmsg with no timeout and do the timeout on the erlang level, if you do for example a timeout of 5000ms, what happens is that under the hood the kernel is called with no timeout, if no message is returned then a kernel-select happens, and then do an erlang wait. The proposed recvmmsg works exactly the same way (which makes me think, probably need to fix the function spec )
I understand that this is used with multiple messages, what happens if I call the function in a hot loop with a message per call?
I should really benchmark this in my SQL library, to get the answer, although I did try to accumulate messages and use regular send which did not perform any better, as you usually need a bit of latency to benefit from reduced system calls.
If you’re calling sendmmsg/recvmmsg with only one message it’s probably a couple of CPU cycles faster to just call the equivalent sendmsg/recvmsg singular counterparts, the plural equivalents have the overhead of wrapping in lists so M is smaller than [M], but otherwise they behave pretty much equivalently, the point of using them is to do batching though, in hot loops all the better. Both the already existing singulars and my proposed plurals are just wrappers around kernel APIs, recommend the kernel docs too
sendmmsg and recvmmsg are present in Linux and in {Free,Net,Open}BSD and AIX.
I’ve been unable to find them in the Solaris documentation.
Everything I have been able to find says that they’re not in Windows.
What does this new code (nice performance improvement!) do when built on
platforms that don’t offer sendmmsg and recvmmsg?
Yes, those are present only in Linux and in most BSD variations, but not in Darwin (MacOS) nor Windows. Windows has some really weird modes around WSARecvMsg and MacOS has a well-known and completely undocumented recvmsg_x that everybody fears using, there’s some read about Firefox trying to do this in a multiplatform way.
In the unsupported platforms it would raise a notsup error, for example on my Mac:
1> {ok, Socket} = socket:open(inet, dgram, udp).
{ok,{'$socket',#Ref<0.1088206482.1928724481.132731>}}
2> socket:bind(Socket, #{addr => any, port => 8080, family => inet}).
ok
3> socket:recvmmsg(Socket, 10, 0, 0, [], nowait).
** exception error: notsup
in function prim_socket:nif_recvmmsg/6
called as prim_socket:nif_recvmmsg(#Ref<0.1088206482.1928724481.132731>,10,0,0,0,
#Ref<0.1088206482.1928593409.132841>)
in call from prim_socket:recvmmsg/6
in call from socket:recvmmsg_nowait/6 (socket.erl:6298)
in call from socket:recvmmsg/6 (socket.erl:6269)
)