Please, my certs, they’re very sick
tl;dr: LetsEncrypt revoked certs issued via TLS challenge, traefik doesn’t ‘know’ that so you need to manually remove them so that traefik generates a new cert
What happened?
I was checking on one of the services I self-host locally, and Firefox loudly complained:
Why had this happened?
Recently, LetsEncrypt issued a notice about certificates obtained via TLS challenge:
[Urgent] Let’s Encrypt revocations affecting yourTLScertificates
Please immediately renew yourTLScertificate(s) that were issued from
Let’s Encrypt using theTLS-ALPN-01 validation method and the following
ACME registration (account) ID(s):
[ID]
We’ve determined that an error made it possible forTLS-ALPN-01
challenges, completed before today, to not comply with certificate
issuance requirements. We have remediated this problem and will revoke
all unexpired certificates that used this validation method at 16:00 UTC
on 28 January 2022. Please renew your certificates now to ensure an
uninterrupted experience for your site visitors.
We apologize for any inconvenience this may cause. If you need support
in the renewal process, please comment on our forum post. Our staff and
community members are available to help:
https://community.letsencrypt.org/t/170449
When I set up traefik many moons ago I used the tlsChallenge method, ie:
[certificatesResolvers.myresolver.acme]
[certificatesResolvers.myresolver.acme.tlsChallenge]
so those certs were affected.
How do I fix it?
There is an open request on GitHub asking if there is a way to ‘force’ a renewal ahead of schedule in traefik. While the request dates to 2018, there are — unsurprisingly — some folks commenting in the last few days saying how useful it would be to have the feature.
Members suggested a couple of workarounds:
- delete
acme.json
(the cert store for traefik) - replace with
"status": "invalid",
in acme.json - delete
Certificate
lines in acme.json
basically these boil down to removing or invalidating the certs from traefik’s point of view so that it requests and issues updated certificates.
Idez, one of the traefik devs created a tool to help with this: traefik certs cleaner:
The process:
– run the tool with the right arguments
– copy the content of the generated file to youracme.json
file
– restart Traefik
Whatever method you choose, traefik does request and issue new certs either via a cert validator or via command line:
curl --insecure -vvI https://example.org 2>&1 | awk 'BEGIN { cert=0 } /^\* SSL connection/ { cert=1 } /^\*/ { if (cert) print }'
you may need to do a cache refresh (ctrl+F5) a couple times before the browser stops complaining about the cert.
My heart skipped a beat when I had fixed the certs and Firefox was still complaining…