VerneMQ SSL Termination issue `unsupported_record_type` and `no_suitable_signature_algorithm` errors

Hello,

I’ve set up a vernemq 2.0.1 container and with SSL enabled and I’m facing errors when trying to connect with emqtt_bench with a user name and password. emqtt_bench supports tls v1-> 3 by default. The server logs unsupported_record_type and no_suitable_signature_algorithm error below and the emqtt_bench client is unable to connect. I see in the config that vernemq TLS default is v1.2. Not sure where to head from here, would appreciate any support.

Vernemq Server Logs:

{"line":561,"pid":"<0.1177914.0>","time":1738941936697199,"file":"tls_record.erl","gl":"<0.115.0>","msg":"TLS server: In state hello at tls_record.erl:561 generated SERVER ALERT: Fatal - Unexpected Message, - {unsupported_record_type,71}","level":"notice","mfa":["tls_record","validate_tls_records_type",8],"depth":20,"report_cb":"fun ssl_logger:format/1"}

{"line":1740,"pid":"<0.1175942.0>","time":1738940031281844,"file":"ssl_handshake.erl","gl":"<0.115.0>","msg":"TLS server: In state hello at ssl_handshake.erl:1740 generated SERVER ALERT: Fatal - Insufficient Security, - no_suitable_signature_algorithm","level":"notice","mfa":["ssl_handshake","do_select_hashsign",3],"depth":20,"report_cb":"fun ssl_logger:format/1"}

Settings:

- name: DOCKER_VERNEMQ_LISTENER__SSL__DEFAULT
  value: "0.0.0.0:8883"
- name: DOCKER_VERNEMQ_LISTENER__SSL__ALLOWED_PROTOCOL_VERSIONS
  value: "3,4,5"
- name: DOCKER_VERNEMQ_LISTENER__SSL__CAFILE
  value: "/etc/ssl/certs/ca-certificates.crt"
- name: DOCKER_VERNEMQ_LISTENER__SSL__CERTFILE
  value: "/etc/ssl/certs/erc/tls.crt"
- name: DOCKER_VERNEMQ_LISTENER__SSL__KEYFILE
  value: "/etc/ssl/certs/erc/tls.key"

SSH Debug:

openssl s_client -connect mqtt.edge.redacted.com:8883 -servername mqtt.edge.redacted.com
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = E6
verify return:1
depth=0 CN = *.edge.redacted.com
verify return:1
---
Certificate chain
 0 s:CN = *.edge.redacted.com
   i:C = US, O = Let's Encrypt, CN = E6
   a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA384
   v:NotBefore: Feb  3 03:02:25 2025 GMT; NotAfter: May  4 03:02:24 2025 GMT
 1 s:C = US, O = Let's Encrypt, CN = E6
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   a:PKEY: id-ecPublicKey, 384 (bit); sigalg: RSA-SHA256
   v:NotBefore: Mar 13 00:00:00 2024 GMT; NotAfter: Mar 12 23:59:59 2027 GMT
---
Server certificate
redacted

subject=CN = *.edge.redacted.com
issuer=C = US, O = Let's Encrypt, CN = E6
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 2364 bytes and written 428 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-ECDSA-AES256-GCM-SHA384
Server public key is 256 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-ECDSA-AES256-GCM-SHA384
    Session-ID: redacted
    Session-ID-ctx:
    Master-Key: redacted
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1738939370
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
---

emqtt_bench call:

              ./bin/emqtt_bench sub \
              --host mqtt.edge.redacted.com \
              --version 5 \
              --ssl \
              --port 8883 \
              --username user \
              --password pass \
              --qos 1

Hello Ashton,

I do not know why you can connect with openssl s_client but not with emqtt_bench.

Maybe setting the listener to v1.3 and setting specific ciphers will generate more insights?

listener.ssl.default.tls_version = tlsv1.3
listener.ssl.default.ciphers = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384

The ciphers are just an example to show how to add them, I don’t know what’s appropriate in your case.

Thanks for the reply!

So explicitly settings the tls/ciphers as suggested works with emqtt_bench to some degree. Oddly if I set the broker and the client to v1.1 or v1.3 it works, but not with v1.2. If I try to set the broker to v1.1 and the client to v1.3 it also does not work. Ideally I’d like the broker to be forgiving and negotiate the version. If I set the broker to listener.ssl.default.tls_version=tlsv1.1,tlsv1.2,tlsv1.3 it allows the config but the client is unable to connect with an explicit version set or a negotiation so it doesn’t seem to work at all.

It seems like verne is enforcing one version at a time, is there a way to have verne negotiate the version to the highest the client allows?

Another interesting note that is concerning is the vmq-admin tls check fails, even when tls connections are working. The tls show command returns two certificates successfully.

vmq-admin tls check
RPC to 'VerneMQ@10.42.1.238' failed: {'EXIT',
                                      {badarg,
                                       [{erlang,hd,
                                         [[]],
                                         [{error_info,
                                           #{module => erl_erts_errors}}]},
                                        {stdout_formatter_table,
                                         draw_row_horizontal_border,3,
                                         [{file,
                                           "/usr/src/vernemq/_build/default/lib/stdout_formatter/src/stdout_formatter_table.erl"},
                                          {line,725}]},
                                        {stdout_formatter_table,format_lines,
                                         6,
                                         [{file,
                                           "/usr/src/vernemq/_build/default/lib/stdout_formatter/src/stdout_formatter_table.erl"},
                                          {line,471}]},
                                        {stdout_formatter_table,format,2,
                                         [{file,
                                           "/usr/src/vernemq/_build/default/lib/stdout_formatter/src/stdout_formatter_table.erl"},
                                          {line,97}]},
                                        {stdout_formatter,to_string,2,
                                         [{file,
                                           "/usr/src/vernemq/_build/default/lib/stdout_formatter/src/stdout_formatter.erl"},
                                          {line,325}]},
                                        {vmq_cli_human_writer,write_table,1,
                                         [{file,
                                           "/usr/src/vernemq/apps/vmq_server/src/vmq_cli_human_writer.erl"},
                                          {line,54}]},
                                        {vmq_cli_human_writer,write_status,2,
                                         [{file,
                                           "/usr/src/vernemq/apps/vmq_server/src/vmq_cli_human_writer.erl"},
                                          {line,44}]},
                                        {clique_status,parse,3,
                                         [{file,
                                           "/usr/src/vernemq/_build/default/lib/clique/src/clique_status.erl"},
                                          {line,49}]}]}

Thanks for the report. Yes, Verne currently enforces a TLS version per listener.
The output of the vmq-admin tls check command seems to indicate some sort of format related error rather than an internal issue. I’ll look into it.

1 Like