Though Trellis VPS setups should auto renew the Let’s Encrypt certificate automatically I had a certificate expire for one of my Trellis sites the other day. Here is how I found out about the Trellis Let’s Encrypt Expired SSL / TLS certificate and how I worked out solving it.

Google Search Console Warning

The reason I found out about it was on time and not by visiting the site itself. I got a warning from the Google Search Console:

Expired SSL/TLS certificate on https://domain.com

To: Webmaster of https://domain.com

Google has noticed that the SSL/TLS certificate for https://imagewize.nl/ has expired. This means that your website is not perceived as secure by some browsers. As a result, many web browsers will block users by displaying a security warning message when your site is accessed. This is done to protect users’ browsing behavior from being intercepted by a third party, which can happen on sites that are not secure.

Recommended Action:

Get a new certificate
To correct this problem, renew or get a new SSL/TLS certificate. This should be from a Certificate Authority (CA) that is trusted by web browsers.

And because Trellis uses HSTS I could not even continue to the site despite the warning I should not normally having an option to continue anyways..

Renew Certificate Manually

I checked Roots Discourse and found out I could use the following command to renew the certificate manually:

ansible-playbook server.yml -e env=production -K --tags letsencrypt

That was not enough though. I had to restart NGINX as well. And when I logged in I was told a reboot was needed as well:

dhc-user@domain.com:~$ sudo service nginx restart
dhc-user@domain.com:~$ sudo reboot

Cause Failed Auto Renew

Perhaps the during an auto renew request the server was not accessible. Did not see errors it was down though. I could also not find errors like:

2016/07/08 02:41:31 [error] 7259#7259: ocsp.int-x3.letsencrypt.org could not be resolved (110: Operation timed out) while requesting certificate status, responder: ocsp.int-x3.letsencrypt.org

as mentioned in a thread at Roots Discourse here. So I did not have issues accessing Let’s Encrypt for the renewal either.

Trellis Cron jobs

I checked for cron jobs next using:

for user in $(cut -f1 -d: /etc/passwd); do crontab -u $user -l; done

as root, but no cron jobs were shown for any users . And we do need one to run the Let’s Encrypt auto renew of course. Then I mentioned this on Roots Discourse and got this comment by  CFX:

Please ensure your cron job (normally at /etc/cron.d/letsencrypt-certificate-renewal) has the full path to service in its reload command (original PR2). That was apparently my issue too—renewal was just fine but nginx never reloaded after renewal.

This way I learned the location of the cronjobs. And also that the path to reload NGINX could sometimes be the issue and a full path is needed in those cases. So I adjusted the path. And it was confirmed a little later that this was the main cause.

Cron.d

So through the comment by CFX I found out the cron jobs for Trellis are stored at:

/etc/cron.d/

and the Let’s Encrypt one is therefore at:

/etc/cron.d/letsencrypt-certificate-renewal

And this is the location for cron jobs I did not check. Now that file contains:

30 4 1,11,21 * * root cd /var/lib/letsencrypt && ./renew-certs.py && /usr/sbin/service nginx reload

This is a perfectly normal cron job. Now it also has the full path to reload NGINX properly. This being the issue I will now have the auto renewal work again like it should be.

NB Reading up on cron jobs here at Ubuntu Wiki it is mentioned too so I found out. It is a place where often one liners for particular users are stored instead of using a crontab.