Your certificate (or certificates) for the names listed below will expire in 10 days (on 12 Jan 21 06:00 +0000). Please make sure to renew your certificate before then, or visitors to your website will encounter errors. [...] *.wikibooks.beta.wmflabs.org *.wikimedia.beta.wmflabs.org *.wikinews.beta.wmflabs.org *.wikipedia.beta.wmflabs.org *.wikiquote.beta.wmflabs.org *.wikisource.beta.wmflabs.org *.wikiversity.beta.wmflabs.org *.wikivoyage.beta.wmflabs.org *.wiktionary.beta.wmflabs.org etc.
12th January arrived and it was still old so I went to take a look.
I found:
- acme-chief on deployment-acme-chief03 had not begun to create a new certificate. It still seemed quite content with version 0d829dc82393450fa34fabd837364efa, with the usual hourly reloading. At around 02:22 I found and killed the acme-chief-backend process, which caused it to get restarted and it produced sudden activity, issuing the new cert as version 1c321ac6da0c4103bf630165d91bca2c. Why was this necessary?
- After this, puppet on deployment-cache-text06 happily updated its local files in /etc/acmecerts, but ATS was still serving the old versions. I had to service trafficserver-tls restart to get it to load new certs from disk.