Not sure if this is the right place to initialize an OTP enhancement proposal, but we are currently doing experiments on using linux kernel TLS as the transport layer of dist communications. It shows significant Memory and CPU savings on dist traffic in our services, so we are wondering if it is viable to support kTLS in OTP, and we can work on the majority of the implementation.
The saving is not because linux kernel encryption being significantly faster than crypto application, but rather because it saves the TLS sender and receiver processes (tls_sender and ssl_gen_statem). It also removes two rounds of length wrap encoding: the current inet_tls_dist sends <<TotalLen, Len1, DistMsg1, Len2, DistMsg2, âŠ>>, but with kTLS and a dist protocol similar to inet_tcp_dist, it can send << DistMsg>> directly.
There is also some drawback of kTLS. Since the inet_drv port_driver handles all traffic in single thread. The data encoding and network send will be processed sequentially. As a result, the single socket data throughput is decreased by a large factor (30% ~ 50%). However we usually donât have this large data transfer, and we can start more sockets to send data in parallel if needed.
For implementation, I think ssl_gen_statem could support an ssl option âuse_ktlsâ. When this option is enabled, the ssl_gen_statem can just do some kTLS set-ups, change the controlling process, and then close the supervision tree (tls_dyn_connection_sup) after the handshake is done.
Example (hacky) code for TLS_CIPHER_AES_GCM_256 after ssl:handshake or ssl:connect:
set_ktls(#sslsocket{pid = [Receiver | _], fd = {_, Socket, _, _}}) ->
ControlPid = self(),
State = sys:replace_state(
Receiver,
fun(State) ->
gen_tcp:controlling_process(Socket, ControlPid),
State
end
),
inet:setopts(Socket, [list, {active, false}]),
{_, #state{connection_states = ConnectionStates}} = State,
CurrentWrite = maps:get(current_write, ConnectionStates),
CurrentRead = maps:get(current_read, ConnectionStates),
#cipher_state{iv = <<WriteSalt:4/bytes, WriteIV:8/bytes>>, key = WriteKey} = maps:get(cipher_state, CurrentWrite),
#cipher_state{iv = <<ReadSalt:4/bytes, ReadIV:8/bytes>>, key = ReadKey} = maps:get(cipher_state, CurrentRead),
WriteSeq = maps:get(sequence_number, CurrentWrite),
ReadSeq = maps:get(sequence_number, CurrentRead),
inet:setopts(Socket, [{raw, 6, 31, <<"tls">>}]),
inet:setopts(Socket, [{raw, 282, 1, <<4, 3, 52, 0, WriteIV/binary, WriteKey/binary, WriteSalt/binary, WriteSeq:64>>}]),
inet:setopts(Socket, [{raw, 282, 2, <<4, 3, 52, 0, ReadIV/binary, ReadKey/binary, ReadSalt/binary, ReadSeq:64>>}]),
Socket.
After setting up kTLS, one can technically send/recv encrypted message through gen_tcp:send(Socket, Data) and gen_tcp:recv(Socket, 0) API, it can be simply wrapped by ssl APIs.
There are several works to be done and several problems/questions though:
- All kTLS-supported cipher constants need to be integrated (Option code, Cipher code, Salt length, Key length, IV length), right now we can only use raw option bytes to set them
- TLS post handshake data might not be supported in kTLS
- {packet, 4} might not be supported in kTLS
- Is there any concern for removing the double layer of length wrapped encoding? Why it was initially added?
- Is there any plan of migrating to socket NIF in dist/ssl? Or any planned change on ssl application? Will there be any conflict with this proposal?