Introduce kernel TLS in ssl application?

Not sure if this is the right place to initialize an OTP enhancement proposal, but we are currently doing experiments on using linux kernel TLS as the transport layer of dist communications. It shows significant Memory and CPU savings on dist traffic in our services, so we are wondering if it is viable to support kTLS in OTP, and we can work on the majority of the implementation.

The saving is not because linux kernel encryption being significantly faster than crypto application, but rather because it saves the TLS sender and receiver processes (tls_sender and ssl_gen_statem). It also removes two rounds of length wrap encoding: the current inet_tls_dist sends <<TotalLen, Len1, DistMsg1, Len2, DistMsg2, …>>, but with kTLS and a dist protocol similar to inet_tcp_dist, it can send << DistMsg>> directly.

There is also some drawback of kTLS. Since the inet_drv port_driver handles all traffic in single thread. The data encoding and network send will be processed sequentially. As a result, the single socket data throughput is decreased by a large factor (30% ~ 50%). However we usually don’t have this large data transfer, and we can start more sockets to send data in parallel if needed.

For implementation, I think ssl_gen_statem could support an ssl option ‘use_ktls’. When this option is enabled, the ssl_gen_statem can just do some kTLS set-ups, change the controlling process, and then close the supervision tree (tls_dyn_connection_sup) after the handshake is done.

Example (hacky) code for TLS_CIPHER_AES_GCM_256 after ssl:handshake or ssl:connect:

set_ktls(#sslsocket{pid = [Receiver | _], fd = {_, Socket, _, _}}) ->
    ControlPid = self(),
    State = sys:replace_state(
        Receiver,
        fun(State) ->
            gen_tcp:controlling_process(Socket, ControlPid),
            State
        end
    ),
    inet:setopts(Socket, [list, {active, false}]),
    {_, #state{connection_states = ConnectionStates}} = State,
    CurrentWrite = maps:get(current_write, ConnectionStates),
    CurrentRead = maps:get(current_read, ConnectionStates),
    #cipher_state{iv = <<WriteSalt:4/bytes, WriteIV:8/bytes>>, key = WriteKey} = maps:get(cipher_state, CurrentWrite),
    #cipher_state{iv = <<ReadSalt:4/bytes, ReadIV:8/bytes>>, key = ReadKey} = maps:get(cipher_state, CurrentRead),
    WriteSeq = maps:get(sequence_number, CurrentWrite),
    ReadSeq = maps:get(sequence_number, CurrentRead),
    inet:setopts(Socket, [{raw, 6, 31, <<"tls">>}]),
    inet:setopts(Socket, [{raw, 282, 1, <<4, 3, 52, 0, WriteIV/binary, WriteKey/binary, WriteSalt/binary, WriteSeq:64>>}]),
    inet:setopts(Socket, [{raw, 282, 2, <<4, 3, 52, 0, ReadIV/binary, ReadKey/binary, ReadSalt/binary, ReadSeq:64>>}]),
    Socket.

After setting up kTLS, one can technically send/recv encrypted message through gen_tcp:send(Socket, Data) and gen_tcp:recv(Socket, 0) API, it can be simply wrapped by ssl APIs.

There are several works to be done and several problems/questions though:

  1. All kTLS-supported cipher constants need to be integrated (Option code, Cipher code, Salt length, Key length, IV length), right now we can only use raw option bytes to set them
  2. TLS post handshake data might not be supported in kTLS
  3. {packet, 4} might not be supported in kTLS
  4. Is there any concern for removing the double layer of length wrapped encoding? Why it was initially added?
  5. Is there any plan of migrating to socket NIF in dist/ssl? Or any planned change on ssl application? Will there be any conflict with this proposal?
8 Likes

Just to add onto the initial post by @zzydxm, here is an example built on top of the socket NIF using kTLS: Example of Linux Kernel TLS (KTLS) with Erlang/OTP 24 `socket` NIF · GitHub

See the following for more information about kTLS itself:

6 Likes

I do not have any definite answer to this. I would say that we are probably more interested in making more optimization efforts to the Erlang code. I would not say that we are short of ideas but as usual, it is a question about prioritizing and tradeoffs to other works and features. Using the new sockets for the ssl application is definitely on the horizon. @kennethL and @raimo do you have any comments on the subject at the moment?

3 Likes

We will do the actual implementation so it won’t use much of your time. I just wonder if this change will have some conflict or some problem that I didn’t see. Maybe it would be better to do an actual pull request and discuss in it? (Wouldn’t happen soon though)

3 Likes

If it is going to be included in OTP we will have to maintain it, even if you do initial implementation with tests and documentation. I think to be really useful there should be a way to handle renegotiation in TLS-1.2 and key update messages in TLS-1.3. This would probably need similar mechanisms as if someone would want to implement QUICK. I think it could be a long term goal.

2 Likes

To my understanding, kernel TLS (kTLS) is essentially a flag to socket options (although it can only work with Linux). I would not expect significant maintenance cost there.

As for TLS renegotiation and related portions, I would be very interested in knowing the context, why OTP uses only the crypto part part of openSSL, but not the connectivity.

2 Likes

Once upon a time erlang used OpenSSL connectivity part. The solution was hard to maintain and did not perform well (mostly not due to OpenSSL itself though). Nobody seemed interested to contribute to make it better, people rolled their own solutions instead. So we decided to use Erlang instead for the parts that Erlang is really good at and use OpenSSL for the encryption parts that you should not be implementing yourself and that Erlang is not the best fit for. Since then we have received a lot of user contributions and fixing and finding bugs is much easier. And some types of errors such as buffer overflows in the protocol code we do not need to worry about.

7 Likes

Would it make sense to only support TLS 1.3 in the kTLS implementation? TLS 1.2 will be deprecated and TLS 1.2 renegotiation implementation with kTLS would be really complicated, but TLS 1.3 key update might be able to be handled by the inet driver/socket nif themselves.

The core thing is we don’t want to have processes to proxy the dist data like what tls_sender and receiver does. The two processes added most cost on CPU and memory in my view.

3 Likes

I think kTLS solves the same part that OpenSSL does today, just in a slightly different way. All of the Erlang implementation is still needed in order to perform the handshake, renegotiation, key update requests, etc.

I think it would mostly be an alternative to sections like tls_record_1_3:decode_cipher_text/2, so that instead of calling the crypto functions provided by OpenSSL, the encrypt/decrypt operations would instead by handled within the kernel (either in software or by offloading to a hardware network device).

But, you’re right, there are some unknowns are this point, like how exactly can we detect sequence exhaustion on TLS 1.3 and know when to emit a key_update while using kTLS.

However, I also understand the concern about maintainability in the ssl application. Would it make more sense to start with adding support in inet_drv.c and prim_socket_nif.c, independent of the ssl application?

I imagine it would just be an alternative to using raw along with inet:setopts/2 today, and would only be supported on Linux for now. However, I think FreeBSD may also have some form of support for kTLS, too, so it’s possible that this pattern winds up being more common in the future (see Improving NGINX Performance with Kernel TLS).

3 Likes

If we are looking at this as a distribution optimization I see no problem with restricting it to TLS-1.3.

4 Likes

I think it makes some sense to first add the building blocks and then see what you can build. We have a long-term goal to have some kind of pluggable “separation” between the TLS record protocol and the other parts to for instance enable QUICK implementations.

4 Likes

@zzydxm has submitted the following PR which implements the first parts in inet of what we discussed here: [RFC] Add kernel TLS options in prim_inet and add related constants in inet_drv by zzydxm · Pull Request #5840 · erlang/otp · GitHub

@ingela As mentioned in the PR, TLS key update may not be fully supported right now and may, potentially, be handled by the hardware or kernel itself in the future (in kTLS TLS_HW_RECORD mode, using hardware like NVIDIA Mellanox ConnectX-6 or ConnectX-7).

We also discovered that, over the years, there has been a lot of debate amongst the Linux community related to whether “hardware based TLS” ought to exist or not in the first place, as it annoyingly replaces parts of the software networking stack, which broke a lot of existing assumptions in the kernel. I suspect that integration into Erlang/OTP will likely be similarly annoying as kTLS continues to mature. However, as @zzydxm mentioned in the PR, a potential for 10-20%+ CPU savings is significant enough that I think there’s a good chance that kTLS will continue to improve in the future and won’t be going away anytime soon.

3 Likes