Unexpected interaction between ssl_config.erl, yaws and letsencrypt

Questions / Help
, ,
1

ssl_config.erl treats ‘certfile’ and ‘cacertfile’ in two slightly different ways. I’m not sure why. Either way, it turned into a gotcha for me.

What happened

Yesterday, my site monitoring alerted me to a certificate problem:

curl: (60) SSL certificate problem: unable to get local issuer certificate

Studying the logs, I saw this happened shortly after ‘certbot’ automatically renewed the site’s certificates. I also saw that the problem went away the next time the webserver (yaws) was restarted, for log rotation.

Why are there two certificates?

‘Letsencrypt’, and probably other CAs, use a two-step way of signing certificates involving two certificates:

leaf certificate: “Letsencrypt’s delegated signer says this certificate belongs to Corelatus”
intermediate certificate: “The delegated signer belongs to Letsencrypt”

Letsencrypt could have used their root certificate to sign my leaf
certificate, but they don’t, for practical reasons. The result is that
the browser needs both certificates.

Where does Certbot put certificates?

Certbot puts certificates in /etc/letsencrypt/archive, e.g.

cert15.pem // the leaf
chain15.pem // the intermediate
fullchain15.pem // both in one file

Certbot then symlinks all three of them from /etc/letsencrypt/live/corelatus.com, basically

cert.pem → /etc/letsencrypt/archive/cert15.pem
fullchain.pem → /etc/letsencrypt/archive/fullchain15.pem

What does Yaws do?

Yaws just passes its configuration on to SSL. In my case, that was

certfile: /etc/letsencrypt/live/corelatus.com/cert.pem
cacertfile: /etc/letsencrypt/live/corelatus.com/fullchain.pem

What does Erlang’s SSL code do?

Erlang’s SSL code has a cache for the certificates, i.e. it doesn’t re-read them all the time. You can snoop the cache because it’s in an ETS table:

ets:tab2list(ssl_pem_cache).
[{<<“/etc/letsencrypt/live/corelatus.com/cert.pem”>>,
{<<“/etc/letsencrypt/live/corelatus.com/../../archive/corelatus.com/fullchain15.pem”>>

There’s a difference between the leaf and the full chain: the leaf gets the symlink. The fullchain got resolved to the actual file. That was a bit surprising. It turns out that the ssl_config.erl treats the two options differently. Only "opt_cacerts() calls unambiguous_path() to change the symlink into a link-free filename.

The result is that when ‘certbot’ updates the certificate, Erlang’s SSL code picks up the new leaf, but not the new intermediate.

How did I fix this?

I fixed it by changing my ‘yaws’ configuration:

old:    certfile=/etc/letsencrypt/live/corelatus.com/cert.pem         cacertfile=/etc/letsencrypt/live/corelatus.com/fullchain.pem

fixed:  certfile=/etc/letsencrypt/live/corelatus.com/fullchain.pem

Another possible fix would have been to add a post-deployment hook to ‘certbot’, i.e. automatically restart yaws after certbot updates certificates.

Why am I posting this to the forum?

Mainly because I would have liked to have stumbled across this explanation before I started :wink:

But also a bit because someone might know something I don’t. I’m left wondering

  • If you can put the full chain in ‘certfile’, why do we need the ‘cacertfile’ option? Why not just always put them in the same file? (Historical? Maybe some types of certificates can’t be combined?)
  • Why is unambiguous_path() called at all? What’s the benefit in eliminating symlinks?

Matthias

2 Likes
2

Yes. The ability to specify a full chain in certfile (and cert) was added in SSL 10.2 (in OTP-23.2); see SSL Release Notes — ssl v11.4

The cert and certfile options will now accept a list of certificates so that the user may specify the chain explicitly.

3

Ah, thanks, that’s interesting.

4

I dug into my second question a bit: “why is unambiguous_path() called at all?”

It was introduced in this commit:

commit 8c7fa307ed07b89e445ffea1cb3dbdf89fba64c8
Date: Wed Jul 27 14:10:08 2022 +0200

ssl: adjust cacertfile option

- perform conversions before loading cert data from file
- convert relative path to absolute for avoiding collisions
- convert symlink to actual file path

The commit message doesn’t give me much of a clue as to “why?”, but the test suite has a better explanation. The concern seems to be that the ETS cache shouldn’t contain multiple copies of the same certificate just because it has different filenames which point to the same file.

5

No the thing was that the cache should not become confused if someone hade certificate-files with the same name but under different directories that contained different certificates.

2 Likes