Hello everybody, I just want to know if the Kernel Polling functions ( epoll
for Linux and kqueue
for FreeBSD) are used by default to handle sockets and connections when the Kernel Poll Space is enabled in Erlang/OTP ?
I asked this questions because I had read the code source of TCP Section of Erlang Network Driver ( inet_drv.c
) and there was any Kernel Poll Function Call so I thinked that the OS use the Kernel Polling by default when creating , binding, listening and accepting sockets if the Kernel Poll Space is enabled.
3 Likes
inet_drv.c
is a Port Driver which interacts with the BEAM’s internal I/O system by calling driver_select
. Normally, there is no direct interaction with the underlying OS polling system within a Port Driver.
However, if you follow the call path for driver_select
, you’ll find that erl_check_io.c
contains most of the I/O handling for the BEAM and that erts/emulator/sys/common/erl_poll.c
and erts/emulator/sys/win32/erl_poll.c
contain the OS-specific parts.
There is a good comment at the top of erts/emulator/sys/common/erl_poll.c
which describes the fallback behavior:
* The interface is currently implemented using:
* - select
* - poll
* - /dev/poll
* - epoll with poll or select as fallback
* - kqueue with poll or select as fallback
In older versions, there used to be an erl +K true
flag to enable or disable kernel polling at runtime, but today it looks like it’s something determined more at compile-time.
As of OTP 24, you can run erlang:system_info(check_io)
to see information about the BEAM’s internal I/O checkers.
6 Likes
thank you very much,
FIRST : the driver tcp_inet
is reponsible for handling all Erlang tcp requests, this driver’s Entry is defined in inet_drv.c
so we can say this way that this C program is directly used by ERTS
SECOND : driver_select
is used in inet_drv.c
only when Defining win32
and not in POSIX systems (Unix and Linux)
Iam sure you have great understood to internal ERTS but here you didn’t specify in more details how that works
2 Likes
At this instant, I had read the inet_drv.c
code from creating a listen socket to accept a connection on it and create a new connected socket and there is no sock_select
call, so as my knowledge (Iam not very experimented with C language) the New Accepted Socket’s FD should added to the Kernel Pollset immediately when accepting the connection ?
I had not read the code yet for sending and receiving data via the socket
2 Likes
For gen_tcp:accept/1,2
, see how sock_select
is only called if accepting on the socket results in ERRNO_BLOCK
.
For receiving there are a couple options: (1) use gen_tcp:recv/2,3
or (2) use inet:setopts/2
with active
set. The Port Driver’s tcp_inet_drv_input
callback will be called as it is defined as the callback for ready_input
in the TCP driver entry. See the case section for TCP_REQ_RECV
.
2 Likes
Yeah sock_select
is called only if socket==INVALID_SOCKET
when accepting a socket fails and returns (-1) but not called when accept()
succeed and returns FD so what you can say about that?
I have just read the code until accepting a socket like I said, so send and receiving data not yet
2 Likes
That’s just the way non-blocking accept(2)
functions in Linux, FreeBSD, Windows, etc.
If there’s a connection ready to accept, then it will return a new file descriptor.
If there are no connections available, it returns -1
(or INVALID_SOCKET
) and errno
gets set to EAGAIN
or EWOULDBLOCK
; this is the case where epoll (on Linux) would then be used as a result of sock_select
so the Port Driver can be notified once there is a connection ready to accept.
2 Likes
Yeah I said that you are great thank you, I think that I understand know, I did a look to the gen_tcp:recv
function track and I think that it works the same way as accept
so first the Thread check if there is any ready data to read from the socket if it’s true he will return the data using recv()
system call but if there is no data he will call sock_select
that result in adding the socket’s FD to the Kernel Pollset and the ERTS will be notified by the Kernel (using epoll
or kqueue
) if there is any data arrives to this socket.
1-Iam confused about accept
, in the case of receiving data that’s ok like I explained above but in the case of accepting connections, if there is no connections how can using the Kernel Polling on just one FD (the listen socket) so as my knowledge epoll
or kqueue
used on a set of FD.
2-Last thing, how can you know that driver_select
results in using erl_poll.c
since driver_select
is used from the header erl_driver.h
and as my knowledge the library implementing this header is not open source code ?
2 Likes
The implementation of driver_select
is in erts/emulator/sys/common/erl_check_io.c
and it’s open-source along with the rest of OTP.
For epoll
specifics, check out the functions called by erl_check_io.c
in erts/emulator/sys/common/erl_poll.c
. Much of the epoll
specific parts are dependent on the C macro definitions that are determined during the ./configure
step when building OTP on a Linux host. So, as long as the build system was able to detect epoll
support, it should be used, but if not then poll or select will be used as a fallback.
In my opinion: inet_drv.c
might be more difficult to understand if you’re trying to connect it to the underlying non-blocking I/O functionality compared with the socket
module with nowait
. It might be helpful to do some experiments with listening and accepting a TCP connection using socket
to see how it mirrors the underlying non-blocking OS functionality.
3 Likes
Thanks again, I know the socket
module and it’s lowest level than gen_XXP
but it’s recommended in the above of the module to use gen_tcp
instead of and all Erlang Softwares that I know use gen_XXp
so tracking it will be more helpful, but my problem is not at the Erlang part but in C implepentation, I think that I understand a lot of internal working of the ERTS and as I said my last thing is using Kernel Polling (epoll
and kqueue
) to do non-blocking accepts in One Listen Socket (One FD), as my knowledge Kernel Polling use a Set of FDs that can updated and check periodically any news about each FD of the set if there’s news the Kernel behaves, but what exactly happens in non-blocking accepts when using Kernel Polling ? this is very important for me and if you can answer this I will be very thankful Sir.
1 Like