Keeping your Trellis Server Updated

When you manage client servers with Trellis you will every now and then have to update the server. This to keep Ubuntu, NGINX, MariaDB and PHP updated. But also to patch security holes and update the latest Linux libraries. So keeping your Trellis server updated is pretty important. And fortunately with Trellis it is all pretty straight forward.

Local Server Update

When there is a new update of the Ubuntu Bento Box you will see that when you start your Vagrant Box from the command line:

==> default: Removing hosts
==> default: Checking if box 'bento/ubuntu-16.04' is up to date...
==> default: A newer version of the box 'bento/ubuntu-16.04' is available! You currently
==> default: have version '2.2.9'. The latest is version '2.3.0'. Run
==> default: `vagrant box update` to update.

I really love that. No need to ssh into your box to do updates unless you want specific tweaks. Just keep up with the latest Bento Box updates and you should be just fine in the majority of the cases.

You can do a local update using the following command:

vagrant box update

This will then download the latest Bento Box from the Hashicorp Atlas Server and update your Vagrant Box. Bento is a great box based on an encapsulated Hashicorp Packer template for Vagrant Boxes maintained by the Chef team. The repo is here.

When all goes well you will see a message like:

==> default: Successfully added box 'bento/ubuntu-16.04' (v2.3.0) for 'virtualbox'!

If something did go wrong you will see a warning and or errors. Often this means you have to do it all again. From my location that will mean you will be spending another 20-30 minutes. So keep your fingers crossed at all times!

Remote Staging or Production Server Update

To update your live or production server or your staging server – if you use one – you will need to fire the following command:

ansible-playbook server.yml -e env=<environment>

This command is for provisioning a server, but also for updating the Trellis server. It is an Ansible playbook as Trellis is basically a set of playbooks to manage your server locally as well as remotely. To do this, this means you have to open the terminal, go to the trellis folder and type in

ansible-playbook server.yml -e env=production

NB server.yml can be viewed here

for example to update the production box. Do remember the remote site will not be accessible or fully accessible when you do this. So make sure you do this during slow traffic hours. Sometimes the update does not go well. Due to connection issues, issues with the Let’s Encrypt certificates or other issues. Run it again and all should work out well in the end. If things do go well you will see the following:

PLAY RECAP *********************************************************************

imagewize.nl               : ok=102  changed=6    unreachable=0    failed=0   

localhost                  : ok=0    changed=0    unreachable=0    failed=0   

So as you can see it all went well and six packages were changed:

TASK [fail2ban : ensure fail2ban is configured] ********************************
changed: [imagewize.nl] => (item=jail.local)
ok: [imagewize.nl] => (item=fail2ban.local)

TASK [ferm : ensure iptables INPUT rules are added] ****************************
ok: [imagewize.nl] => (item={u'dport': [u'http', u'https'], u'type': u'dport_accept', u'filename': u'nginx_accept'})
changed: [imagewize.nl] => (item={u'dport': [u'ssh'], u'type': u'dport_accept', u'saddr': [u'89.148.54.121']})
ok: [imagewize.nl] => (item={u'dport': [u'ssh'], u'seconds': 300, u'hits': 20, u'type': u'dport_limit'})

TASK [users : Setup users] *****************************************************
changed: [imagewize.nl] => (item={u'keys': [u'ssh-rsa 
key== user@imagewize.com', u'https://github.com/jasperf.keys'], u'name': u'web', u'groups': [u'www-data']})
changed: [imagewize.nl] => (item={u'keys': [u'ssh-rsa 
keyw== user@imagewize.com', u'https://github.com/jasperf.keys'], u'name': u'dhc-user', u'groups': [u'sudo']})

TASK [users : Add SSH keys] ****************************************************
changed: [imagewize.nl] => (item=({u'name': u'web', u'groups': [u'www-data']}, u'ssh-rsa A
key== user@imagewize.com'))
changed: [imagewize.nl] => (item=({u'name': u'web', u'groups': [u'www-data']}, u'https://github.com/jasperf.keys'))
changed: [imagewize.nl] => (item=({u'name': u'dhc-user', u'groups': [u'sudo']}, u'ssh-rsa 
key== user@imagewize.com'))
changed: [imagewize.nl] => (item=({u'name': u'dhc-user', u'groups': [u'sudo']}, u'https://github.com/jasperf.keys'))

TASK [mariadb : Add MariaDB MySQL deb and deb-src] *****************************
ok: [imagewize.nl] => (item=deb http://nyc2.mirrors.digitalocean.com/mariadb/repo/10.0/ubuntu trusty main)
ok: [imagewize.nl] => (item=deb-src http://nyc2.mirrors.digitalocean.com/mariadb/repo/10.0/ubuntu trusty main)
TASK [mariadb : Restart MariaDB MySQL Server] **********************************
changed: [imagewize.nl]

RUNNING HANDLER [fail2ban : restart fail2ban] **********************************
changed: [imagewize.nl]

In this case keys were changed, MariaDB upgraded and Fail2Ban configured anew.

Ansible Package Updates

Sometimes you do need quicker updates or just want to take care of other package updates. You may want to patch security holes or update packages you need the latest version for urgently. There is a Roots Discourse thread on updates where this is mentioned here. Swalkinshaw mentions here you can do three things for manual updates:

  • Add apt upgrade as a task for a server wide upgrade – did it before myself but do test locally or on staging first!
  • manually specify version for Ansible tasks being run for provisioning – see apt Ansible Books
  • or add latest=yes to any apt action

Real Guess has some scripts that I may implement in the future for Ubuntu updates here. It uses the mentioned Ansible apt module. But the examples Ansible mentions are also very useful.

Quick Security Updates

I do ran updates manually every now and then. And I sometimes do them on the server. To deal with security issues. Not as deterministic as I would like, but it does do the trick. I will incorporate some commands into tasks as soon as I can and write another blog post on it.

Earlier I had this shown on logging into one of my servers at Digital Ocean:

Welcome to Ubuntu 16.04.1 LTS (GNU/Linux 4.4.0-34-generic x86_64)

* Documentation: https://help.ubuntu.com
 * Management: https://landscape.canonical.com
 * Support: https://ubuntu.com/advantage

System information as of Thu Oct 6 06:49:26 UTC 2016

System load: 0.09 Processes: 119
 Usage of /: 20.8% of 24.58GB Users logged in: 0
 Memory usage: 48% IP address for eth0: 208.113.134.123
 Swap usage: 1%

Graph this data and manage this system at:
 https://landscape.canonical.com/

Get cloud support with Ubuntu Advantage Cloud Guest:
 http://www.ubuntu.com/business/services/cloud

95 packages can be updated.
44 updates are security updates.

This was after a general provisioning. So to take care of the security packages I did a:

sudo unattended-upgrades -d
  • add the -d flag to get extra information

which works when the unattend-upgrades package is installed. It is on the Bento box. Running this package will update all unattended security packages – SO Thread. Sometimes including regular packages submitted by security maintainers or so it seems. And when I logged in again I saw:

59 packages can be updated.
6 updates are security updates.

Packages that were updated were:

dnsutils
libbind9-140
libdns-export162
libdns162
libisc-export160
libisc160
libisccc140
libisccfg140
liblwres141
libnss-myhostname
libpam-systemd
libssl1.0.0
libsystemd0
libudev1
linux-headers-generic
linux-headers-virtual
linux-headers-virtual-lts-xenial
linux-image-virtual
linux-libc-dev
linux-virtual
linux-virtual-lts-xenial
ntp
openssl
systemd
systemd-sysv
udev

So quite a few security packages were updated including SSL. New ones popped up and those can be dealt with the same way. This way is also pretty safe and clean. Better to build it into your Trellis workflow of course. I am working on another task for that. However, all in all a good solution to deal with security on your server in a semi automated and clean way. There are ways to run this in the background too, but so far I have preferred to do it this way, especially with the -d flag to keep everything under control.

So there you have it – keeping your Trellis server updated in a nutshell. You got feedback or questions? Leave a comment below and feel free to retweet or post to Facebook.

PS WordPress Core, plugins and theme updates are handled with

./deploy.sh production site.domain

following composer update. Another blog post on that later on.

Setting up a Trellis Server at Digital Ocean including SendGrid and Google Mail

Here is a the story of setting up a Trellis server at Digital Ocean including SendGrid and Google Mail for a client. I took care of this recently to move him from a shared Dreamhost server (testing ground) to a Trellis LEMP with Digital Ocean using the Bedrock Modern WordPress Stack. This post also includes dealing with general email using Google and external site email using SendGrid. Took longer mainly due to SendGrid issues. Normally is should just take a few hours depending on the propagation of the domain name and the provisioning and deployment of your server.

Backups

I made a backup of the existing website using BackupBuddy (awesome backup plugin by iThemes) and files backup with SFTP. I also made a database backup using Sequel Pro. On top I exported all data using the WordPress Export Tool. If you are doing a bare bone installation or had been developing with Trellis from scratch and are going to production you can skip this step.

Digital Ocean Setup

I then did the initial set up a US Digital Ocean Ubuntu 16.0.4 LTS Server. To set up Trellis later you need this type of bare bone setup. The client wanted to have a US located server so I picked NYC. I also checked backups and added my SSH keys:

Digital Ocean Droplet
 You need SSH to have smooth and secure access and to make installing Trellis a breeze using the Ansible Playbooks.

Gmail for Mail

I also started the process at Gmail to take care of email for the client as they have the best email service out there for business. Digital Ocean does not take care of this for you. So you need to another provider for this. Google Mail is the best and is around $5 per month for a basic package. You set this all up at Google Apps for Work. Create a general account with them, add email aliases (accounts) and send verifications to your clients. This so the accounts can be activated.

Gmail Domain Verification

Next step is to confirm domain ownership. To use your domain with Gmail you have to be able to confirm you own it adding a tag to the index page’s header section:
 Gmail Verification
 This was not possible yet due to the domain not pointing to the Digital Ocean server as discussed next.

Domain Propagation

Domain propagation to point the domain name to DO was in progress at so I needed to wait a bit for the new domain to load from the new server. For this the What’s My DNS site is your friend:
 What's my DNS

At Dreamhost I already had changed the DNS and had an A record pointing to the new ip address. See article on pointing hosting away from Dreamhost here. This in case you are also using Dreamhost for DNS. If you use another provider they should have documentation for you too.

NB Emptying local DNS might be useful too. Especially when most DNS Servers globally are showing the correct ip address for the domain already. For OSX use:

sudo dscacheutil -flushcache;sudo killall -HUP mDNSResponder;say cache flushed

NNB You may also have to remove the SSH key when you tried to access the site with the old ip address. Remove the offending line at

/Users/your-user/.ssh/known_hosts

Server Provisioning with Trellis

Once all pointed to the Digital Ocean Droplet’s ip address xxx.xx.x.xxx.xxx I could install the Trellis Server at Digital Ocean. You always first set all things up locally and then add all to the server. This means you have several files to edit. Eight when you set up the local server as well as production. A few more when you take care of staging as well. Check things locally. See if they work and do not forget to:

  • add commercial plugins to .gitignore
  • remove www as url if you are only using non www urls
  • activate build-before.yml deploy hook if you are working with Sage

You can read more about it at Roots.io and in this blog post. Better do it well or you will bump into Trellis errors.

 Server Deployment

When all is well and the server is provisioned you can deploy the web application to the server. You can do this using the following command:

./deploy.sh production domain.com

As you can see I am deploying to production. If you are deploying to staging then change accordingly. When all goes well the site should then load and you should be able to install WordPress. Once that is done you can log in and check if all plugins and needed theme have been added.

Importing Content

Content at the installed Trellis Server at Digital Ocean can be imported with WP CLI. This comes pre-installed with the Trellis setup and is an awesome CLI tool for WordPress developer. Importing can be done from the command line with the WordPress Import plugin active. Check that it is installed and active. When it is use the following command:

admin@domain:/srv/www/domain.com/current$ wp import --authors=skip domain.wordpress.year-mo-da.xml

That should allow you to import all. You will see notices and or warnings, but the tool does the job most of the time without issues.

Gmail Domain Confirmation

Once the deployment was done well and I had included the Google Domain Verification email I could proceed with setting up Gmail for Work. In the Dreamhost Panel I indicated I would use Google Mail for email. Dreamhost can then with ease add the necessary MX records. With that Google Aps for Work had enough information and things setup to process all:

Verify your domain and set up email

And as you can see this can take up to 50 minutes for Google Aps for Work to verify the domain and set up your email!
When all is done you will see this popup saying all is well and that you should enter payment details before the trial has ended:
Google Apps - You're almost done.

Sendgrid – External Mail

For the external email server of  the Digital Ocean server we use Sendgrid .

SendGrid

The service is for free up to 12,000 emails per month and 2,000 contacts. Just for getting emails when comments are left, users are registered and such. Better to do this once the domain is all setup. Otherwise Sendgrid may stop the provisioning.

SendGrid Provisioning Issues

That is the issue we ran into. The stated:

Unfortunately, your account did not pass provisioning. We’ve sent you an email outlining why your account was not activated. Learn More.

We opened a ticket with them. Took them about 5 hours to reply. They needed the email address to be changed to reflect the domain, a CNAME created to show ownership of the site:

dig +short @8.8.8.8 sendgrid.domain.com CNAME
sendgrid.net.

and details on the nature of my client’s business. This as we used a different email address for initial setup as Google Mail was not set up. And as the domain hadn’t been properly propagated yet. Next time I will wait a little longer!

Somehow the initial ticket got lost so I opened a new ticket with them and added the last response with details I sent them. About 24 hrs after I set things up with them I still hadn’t had an answer from them. Then I did get a reply again to add a phone number and sent some screenshots of the web app we were building. Et voila, the day after so after about 48 hours our account got accepted!

SendGrid - Provisioning done

Also got an email with links to documentation on things setting up using SMTP or their API. Also a link to their getting starting guide which is always useful for newbies.

Trellis Mail Setup

Once things were settled with SendGrid  I could use the account and add the details in group_vars/all/mail.yml :

# Documentation: https://roots.io/trellis/docs/mail/
mail_smtp_server: smtp.example.com:587
mail_admin: admin@example.com
mail_hostname: example.com
mail_user: smtp_user
mail_password: "{{ vault_mail_password }}" # Define this variable in group_vars/all/vault.yml

Trellis would like you to use SMTP. SendGrid has the following information on that:

SMTP Relay

If you are integrating SendGrid with an existing application, setting up the application to use our SMTP relay is easiest, as it only requires modifying SMTP configuration.

  • Change your SMTP username and password to your SendGrid credentials
  • Set the server host name to smtp.sendgrid.net
  • Use ports 25 or 587 for plain/TLS connections and port 465 for SSL connections
    For most users we suggest port 587 to avoid rate limits set by some hosting companies.

With SMTP, 100 messages can be sent with each connection.

So we used smtp.sendgrid.net:587 and then added all our details. Inside group_vars/all/vault.yml I already had a password added. But as SendGrid stated I had to change the user and password to the ones used by SendGrid.

A good Discourse thread on mail here and the Roots Trellis Mail documentation here.

Re Provisioning

As the Mail credentials were changed I had to first update my repository and then re provision the server.

Git repo update:

Jaspers-Mini:trellis jasper$ pwd
/Users/jasper/webdesign/domain.com/trellis
Jaspers-Mini:trellis jasper$ git status
On branch master
Your branch is up-to-date with 'origin/master'.
Changes not staged for commit:
 (use "git add <file>..." to update what will be committed)
 (use "git checkout -- <file>..." to discard changes in working directory)

modified: group_vars/all/mail.yml
 modified: group_vars/all/vault.yml

no changes added to commit (use "git add" and/or "git commit -a")
Jaspers-Mini:trellis jasper$ git commit -a -m "changed email credentials"
[master 35672f8] changed email credentials
 2 files changed, 5 insertions(+), 5 deletions(-)
Jaspers-Mini:trellis jasper$ git push
Counting objects: 7, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (7/7), done.
Writing objects: 100% (7/7), 783 bytes | 0 bytes/s, done.
Total 7 (delta 4), reused 0 (delta 0)
remote: Resolving deltas: 100% (4/4), completed with 4 local objects.
To git@github.com:jasperf/domain.git
 abe93a8..35672f8 master -> master

Then I did the actual provisioning again:

ansible-playbook server.yml -e env=production

 Provisioning can take another 15-20 minutes as it will check all needs to make sure all has been set up well and to make sure no changes are skipped. So get another coffee or do some other work in the meanwhile. If all went well you should now see:

PLAY RECAP *********************************************************************
localhost : ok=0 changed=0 unreachable=0 failed=0 
domain.com : ok=103 changed=7 unreachable=0 failed=0

Email Failure

I tested email by creating an account and trying. It did not arrive. I then checked SendGrid documentation on working with Sstmp. I only saw another extra line

UseSTARTTLS=YES

but assumed it was already included in Trellis. When I checked the config I saw it was:

cat /etc/ssmtp/ssmtp.conf
# Ansible managed: /Users/jasper/webdesign/domain.com/trellis/roles/ssmtp/templates/ssmtp.conf.j2 modified on 2016-09-19 07:47:36 by jasper on Jaspers-Mini

root=admin@domain.com
AuthMethod=LOGIN
FromLineOverride=Yes
UseTLS=Yes
UseSTARTTLS=Yes
hostname=domain.com
mailhub=smtp.domain.net:587
AuthUser=user
AuthPass=password

Then I thought. Perhaps the from sender in WordPress settings is still not OK. But it was. It was using the existing email account attached to the domain. Then I checked the Ssmtp config again and realized the mailhub was wrong. I had used our domain, not sendgrid. Silly me. So updated repo with corrected settings and provisioned again.

Email Test Successful

So with mail settings changed I tested SendGrid / Mail again by resetting the password for the test user and it worked!

Email send succesfully

And at SendGrid it showed as well:

SendGrid Stats

I can only assume the bounce was due to the initial email with faulty settings.

Trellis Server at Digital Ocean all done

Well, we now are all done setting up a Trellis Server at Digital Ocean including SendGrid and Google Mail. The WordPress website is up and running at Digital Ocean using Trellis. The client can receive email with Google Apps Email although payment credentials still need to be added as well as some more aliases. We can also send emails from the servers for subscriptions and when payments are done using SendGrid. Great result!