Erlang Kernel Polling Functions

Hello everybody, I just want to know if the Kernel Polling functions ( epoll for Linux and kqueue for FreeBSD) are used by default to handle sockets and connections when the Kernel Poll Space is enabled in Erlang/OTP ?

I asked this questions because I had read the code source of TCP Section of Erlang Network Driver ( inet_drv.c ) and there was any Kernel Poll Function Call so I thinked that the OS use the Kernel Polling by default when creating , binding, listening and accepting sockets if the Kernel Poll Space is enabled.

3 Likes

inet_drv.c is a Port Driver which interacts with the BEAM’s internal I/O system by calling driver_select. Normally, there is no direct interaction with the underlying OS polling system within a Port Driver.

However, if you follow the call path for driver_select, you’ll find that erl_check_io.c contains most of the I/O handling for the BEAM and that erts/emulator/sys/common/erl_poll.c and erts/emulator/sys/win32/erl_poll.c contain the OS-specific parts.

There is a good comment at the top of erts/emulator/sys/common/erl_poll.c which describes the fallback behavior:

 *		The interface is currently implemented using:
 *		- select
 *		- poll
 *		- /dev/poll
 *		- epoll with poll or select as fallback
 *		- kqueue with poll or select as fallback

In older versions, there used to be an erl +K true flag to enable or disable kernel polling at runtime, but today it looks like it’s something determined more at compile-time.

As of OTP 24, you can run erlang:system_info(check_io) to see information about the BEAM’s internal I/O checkers.

5 Likes

thank you very much,
FIRST : the driver tcp_inet is reponsible for handling all Erlang tcp requests, this driver’s Entry is defined in inet_drv.c so we can say this way that this C program is directly used by ERTS
SECOND : driver_select is used in inet_drv.c only when Defining win32 and not in POSIX systems (Unix and Linux)

Iam sure you have great understood to internal ERTS but here you didn’t specify in more details how that works

2 Likes
  1. Correct, it just happens to be using the Port Driver interface for interacting with the OS. There is a newer alternative module, socket, which is written using the NIF interface. There is some support for this already in gen_tcp, see the note at the top of the gen_tcp docs that talks about using Backend = socket instead of Backend = inet.
  2. Take a look at the macro definitions for inet_driver_select and sock_select which both result in a call to driver_select in both POSIX and Windows.
2 Likes

At this instant, I had read the inet_drv.c code from creating a listen socket to accept a connection on it and create a new connected socket and there is no sock_select call, so as my knowledge (Iam not very experimented with C language) the New Accepted Socket’s FD should added to the Kernel Pollset immediately when accepting the connection ?

I had not read the code yet for sending and receiving data via the socket

2 Likes

For gen_tcp:accept/1,2, see how sock_select is only called if accepting on the socket results in ERRNO_BLOCK.

For receiving there are a couple options: (1) use gen_tcp:recv/2,3 or (2) use inet:setopts/2 with active set. The Port Driver’s tcp_inet_drv_input callback will be called as it is defined as the callback for ready_input in the TCP driver entry. See the case section for TCP_REQ_RECV.

2 Likes

Yeah sock_select is called only if socket==INVALID_SOCKET when accepting a socket fails and returns (-1) but not called when accept() succeed and returns FD so what you can say about that?

I have just read the code until accepting a socket like I said, so send and receiving data not yet

2 Likes

That’s just the way non-blocking accept(2) functions in Linux, FreeBSD, Windows, etc.

If there’s a connection ready to accept, then it will return a new file descriptor.

If there are no connections available, it returns -1 (or INVALID_SOCKET) and errno gets set to EAGAIN or EWOULDBLOCK; this is the case where epoll (on Linux) would then be used as a result of sock_select so the Port Driver can be notified once there is a connection ready to accept.

2 Likes

Yeah I said that you are great thank you, I think that I understand know, I did a look to the gen_tcp:recv function track and I think that it works the same way as accept so first the Thread check if there is any ready data to read from the socket if it’s true he will return the data using recv() system call but if there is no data he will call sock_select that result in adding the socket’s FD to the Kernel Pollset and the ERTS will be notified by the Kernel (using epoll or kqueue) if there is any data arrives to this socket.

1-Iam confused about accept, in the case of receiving data that’s ok like I explained above but in the case of accepting connections, if there is no connections how can using the Kernel Polling on just one FD (the listen socket) so as my knowledge epoll or kqueue used on a set of FD.

2-Last thing, how can you know that driver_select results in using erl_poll.c since driver_select is used from the header erl_driver.h and as my knowledge the library implementing this header is not open source code ?

2 Likes

The implementation of driver_select is in erts/emulator/sys/common/erl_check_io.c and it’s open-source along with the rest of OTP.

For epoll specifics, check out the functions called by erl_check_io.c in erts/emulator/sys/common/erl_poll.c. Much of the epoll specific parts are dependent on the C macro definitions that are determined during the ./configure step when building OTP on a Linux host. So, as long as the build system was able to detect epoll support, it should be used, but if not then poll or select will be used as a fallback.

In my opinion: inet_drv.c might be more difficult to understand if you’re trying to connect it to the underlying non-blocking I/O functionality compared with the socket module with nowait. It might be helpful to do some experiments with listening and accepting a TCP connection using socket to see how it mirrors the underlying non-blocking OS functionality.

3 Likes

Thanks again, I know the socket module and it’s lowest level than gen_XXP but it’s recommended in the above of the module to use gen_tcp instead of and all Erlang Softwares that I know use gen_XXp so tracking it will be more helpful, but my problem is not at the Erlang part but in C implepentation, I think that I understand a lot of internal working of the ERTS and as I said my last thing is using Kernel Polling (epoll and kqueue) to do non-blocking accepts in One Listen Socket (One FD), as my knowledge Kernel Polling use a Set of FDs that can updated and check periodically any news about each FD of the set if there’s news the Kernel behaves, but what exactly happens in non-blocking accepts when using Kernel Polling ? this is very important for me and if you can answer this I will be very thankful Sir.

1 Like