Though Trellis VPS setups should auto renew the Let’s Encrypt certificate automatically I had a certificate expire for one of my Trellis sites the other day. Here is how I found out about the Trellis Let’s Encrypt Expired SSL / TLS certificate and how I worked out solving it.
Google Search Console Warning
The reason I found out about it was on time and not by visiting the site itself. I got a warning from the Google Search Console:
Expired SSL/TLS certificate on https://domain.com To: Webmaster of https://domain.com Google has noticed that the SSL/TLS certificate for https://imagewize.nl/ has expired. This means that your website is not perceived as secure by some browsers. As a result, many web browsers will block users by displaying a security warning message when your site is accessed. This is done to protect users’ browsing behavior from being intercepted by a third party, which can happen on sites that are not secure. Recommended Action: Get a new certificate To correct this problem, renew or get a new SSL/TLS certificate. This should be from a Certificate Authority (CA) that is trusted by web browsers.
And because Trellis uses HSTS I could not even continue to the site despite the warning I should not normally having an option to continue anyways..
Renew Certificate Manually
I checked Roots Discourse and found out I could use the following command to renew the certificate manually:
ansible-playbook server.yml -e env=production -K --tags letsencrypt
That was not enough though. I had to restart NGINX as well. And when I logged in I was told a reboot was needed as well:
firstname.lastname@example.org:~$ sudo service nginx restart email@example.com:~$ sudo reboot
Cause Failed Auto Renew
Perhaps the during an auto renew request the server was not accessible. Did not see errors it was down though. I could also not find errors like:
2016/07/08 02:41:31 [error] 7259#7259: ocsp.int-x3.letsencrypt.org could not be resolved (110: Operation timed out) while requesting certificate status, responder: ocsp.int-x3.letsencrypt.org
as mentioned in a thread at Roots Discourse here. So I did not have issues accessing Let’s Encrypt for the renewal either.
Trellis Cron jobs
I checked for cron jobs next using:
for user in $(cut -f1 -d: /etc/passwd); do crontab -u $user -l; done
as root, but no cron jobs were shown for any users . And we do need one to run the Let’s Encrypt auto renew of course. Then I mentioned this on Roots Discourse and got this comment by CFX:
Please ensure your cron job (normally at /etc/cron.d/letsencrypt-certificate-renewal) has the full path to service in its reload command (original PR2). That was apparently my issue too—renewal was just fine but nginx never reloaded after renewal.
This way I learned the location of the cronjobs. And also that the path to reload NGINX could sometimes be the issue and a full path is needed in those cases. So I adjusted the path. And it was confirmed a little later that this was the main cause.
So through the comment by CFX I found out the cron jobs for Trellis are stored at:
and the Let’s Encrypt one is therefore at:
And this is the location for cron jobs I did not check. Now that file contains:
30 4 1,11,21 * * root cd /var/lib/letsencrypt && ./renew-certs.py && /usr/sbin/service nginx reload
This is a perfectly normal cron job. Now it also has the full path to reload NGINX properly. This being the issue I will now have the auto renewal work again like it should be.
NB Reading up on cron jobs here at Ubuntu Wiki it is mentioned too so I found out. It is a place where often one liners for particular users are stored instead of using a crontab.