We were using self signed CA certificate to connect MQTT device with vernemq version ‘1.12.3’ and it works without any issue. But with the same certificate, connectivity is not getting established with latest vernemq version - 1.12.6.2
From the pcap we observe that connectivity is getting reset from server side after client responds Client Hello.
Are there any configuration changes to make connectivity allowed?
Awaiting for solution,
Thanks in Advance,
Shashikiran
Let’s see & collect more information here.
First, ensure you have a minimal TLS listener configured in the vernemq.conf file.
listener.ssl.default=0.0.0.0:8883
listener.ssl.default.keyfile=/etc/ssl/vernemq/tls.key
listener.ssl.default.cafile=/etc/ssl/vernemq/ca.crt
listener.ssl.default.certfile=/etc/ssl/vernemq/tls.crt
Instead of 0.0.0.0 you might want to use a specific IP/interface.
Please verify what listeners you have running with vmq-admin listener show
Use a command like openssl _client -connect localhost:8883 to check for a client side opinion on the error. Maybe use a couple of variations of this command, showing the certs, or using -cipher ALL
Also, Verne 1.12.3 is compile with OTP 23.3.2 while Verne 1.12.6.2 is compiled with OTP 24.3.4.5. I’ll check what has possibly changed between those versions in the OTP TLS implementation.
Last but not least: Does VerneMQ log any exception for that connection attempt?
Thanks for quick response.
We have below configuration in values.yaml file of vernemq
listener.ssl.default.keyfile=/etc/ssl/vernemq/tls.key
listener.ssl.default.cafile=/etc/ssl/vernemq/ca.crt
listener.ssl.default.certfile=/etc/ssl/vernemq/tls.crt
And we don’t have this configuration ‘listener.ssl.default=0.0.0.0:8883’
vernemq@vernemq-0:~$ vmq-admin listener show
±-------±--------±-------------±------±-----------±----------+
| type | status | ip | port | mountpoint | max_conns |
±-------±--------±-------------±------±-----------±----------+
| http | running | 127.0.0.1 | 8888 | | 10000 |
±-------±--------±-------------±------±-----------±----------+
| http | running | 192.168.1.22 | 8888 | | 10000 |
±-------±--------±-------------±------±-----------±----------+
| vmq | running | 192.168.1.22 | 44053 | | 10000 |
±-------±--------±-------------±------±-----------±----------+
| mqttws | running | 192.168.1.22 | 8080 | | 10000 |
±-------±--------±-------------±------±-----------±----------+
| mqtts | running | 192.168.1.22 | 8883 | | 10000 |
±-------±--------±-------------±------±-----------±----------+
| mqtt | running | 127.0.0.1 | 1883 | | 10000 |
±-------±--------±-------------±------±-----------±----------+
| mqtt | running | 192.168.1.22 | 1883 | | 10000 |
±-------±--------±-------------±------±-----------±----------+
We ran below to check client side opinion
shashis@N-20S5PF2WA3KC MINGW64 ~
$ openssl s_client -connect 10.75.217.198:31714
CONNECTED(0000016C)
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 293 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
write:errno=10054
We used CA part from the certificate as a certificate ( cacert.pem) and tlskey section for key file (tlskey.pem)
shashis@N-20S5PF2WA3KC MINGW64 /c/SHASHI_OFFICE
$ openssl s_client -connect 10.75.217.198:31714 -cert cacert.pem -key tlskey.pem -state -quiet
error setting private key
15164:error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch:../openssl-1.1.1k/crypto/x509/x509_cmp.c:303:
And No logs are printed at Vernemq as connection itself ressting from server side PCAP are attached
Who is listening on port 31714? It’s not VerneMQ.
Vernemq is listning on 31714 for MQTT over tls connection.
We are not using the standard ports rather we defined the different ones for secure and non-secure in the yaml file as defined below.
Below is the configuration in vernemq
mqtt:
enabled: true
port: 1883
# This is the port used by nodes to expose the service
nodePort: 31712
mqtts:
enabled: true
port: 8883
# This is the port used by nodes to expose the service
nodePort: 31714
I missed that you’re using Kubernetes, apologies.
This is a connection reset by the server, so it points to a server side problem, possibly with the reading or loading of the certificates.
I will do some verification of the Verne image to check whether there could be some TLS issues. Should you have additional findings in the mean time, please keep us posted here.
Are you using vernemq/vernemq:1.12.6.2 or vernemq/vernemq:1.12.6.2-alpine?
We are using ‘vernemq/vernemq:1.12.6.2’
One additional question is whether this listener has loaded certs. To check this, you can attach to Verne in the Pod:
vernemq attach
Then in the Verne shell, run the following command:
ranch:info({{192, 168, 1, 22}, 8883}). and hit Enter (adapt the IP)
Then detach from Verne:
ctrl + g, then q.
Does that command show a configured certificate in the output?
@Shashikirana There really seems to be an issue in the Docker 1.12.6.x packages; this does not seem related to configuration issues. Doing some more digging on this.
There’s an open Github issue here on exactly this: Can't reconfigure mqtts listener · Issue #349 · vernemq/docker-vernemq · GitHub
I ran the above command and it did not show any certificate in the output.
Below is the output.
[root@shashimqttqa-control-01 ~]# kubectl exec -it bash
error: you must specify at least one command for the container
[root@shashimqttqa-control-01 ~]# kubectl exec -it vernemq-0 bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
vernemq@vernemq-0:~$ vernemq attach
Remote Shell: Use "Ctrl-G q" to quit.
q() or init:stop() will terminate the vernemq node.
Erlang/OTP 24 [erts-12.3.2.5] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [jit]
Eshell V12.3.2.5 (abort with ^G)
(VerneMQ@vernemq-0.vernemq-headless.default.svc.cluster.local)1> ranch:info({{10.75.217.198}, 8883})
(VerneMQ@vernemq-0.vernemq-headless.default.svc.cluster.local)1> ranch:info({{192.168.1.22}, 8883})
(VerneMQ@vernemq-0.vernemq-headless.default.svc.cluster.local)1>
User switch command
--> q
The . is part of the command.
You need to ensure you see paths for ca.crt, tls.crt, and tls.key
An easy fix for the Github issue I linked is to use the full linked path for the ca.crt file, as below (adding the ..data). This fixes CA cert loading for the 1.12.6.2 images, so you will need to apply that too.
- name: DOCKER_VERNEMQ_LISTENER__SSL__CAFILE
value: "/etc/ssl/vernemq/..data/ca.crt"
- name: DOCKER_VERNEMQ_LISTENER__SSL__CERTFILE
value: "/etc/ssl/vernemq/tls.crt"
- name: DOCKER_VERNEMQ_LISTENER__SSL__KEYFILE
value: "/etc/ssl/vernemq/tls.key"
can you share the .yaml for deploying in kubernetes
