Keeping your Trellis Server Updated

When you manage client servers with Trellis you will every now and then have to update the server. This to keep Ubuntu, NGINX, MariaDB and PHP updated. But also to patch security holes and update the latest Linux libraries. So keeping your Trellis server updated is pretty important. And fortunately with Trellis it is all pretty straight forward.

Local Server Update

When there is a new update of the Ubuntu Bento Box you will see that when you start your Vagrant Box from the command line:

==> default: Removing hosts
==> default: Checking if box 'bento/ubuntu-16.04' is up to date...
==> default: A newer version of the box 'bento/ubuntu-16.04' is available! You currently
==> default: have version '2.2.9'. The latest is version '2.3.0'. Run
==> default: `vagrant box update` to update.

I really love that. No need to ssh into your box to do updates unless you want specific tweaks. Just keep up with the latest Bento Box updates and you should be just fine in the majority of the cases.

You can do a local update using the following command:

vagrant box update

This will then download the latest Bento Box from the Hashicorp Atlas Server and update your Vagrant Box. Bento is a great box based on an encapsulated Hashicorp Packer template for Vagrant Boxes maintained by the Chef team. The repo is here.

When all goes well you will see a message like:

==> default: Successfully added box 'bento/ubuntu-16.04' (v2.3.0) for 'virtualbox'!

If something did go wrong you will see a warning and or errors. Often this means you have to do it all again. From my location that will mean you will be spending another 20-30 minutes. So keep your fingers crossed at all times!

Remote Staging or Production Server Update

To update your live or production server or your staging server – if you use one – you will need to fire the following command:

ansible-playbook server.yml -e env=<environment>

This command is for provisioning a server, but also for updating the Trellis server. It is an Ansible playbook as Trellis is basically a set of playbooks to manage your server locally as well as remotely. To do this, this means you have to open the terminal, go to the trellis folder and type in

ansible-playbook server.yml -e env=production

NB server.yml can be viewed here

for example to update the production box. Do remember the remote site will not be accessible or fully accessible when you do this. So make sure you do this during slow traffic hours. Sometimes the update does not go well. Due to connection issues, issues with the Let’s Encrypt certificates or other issues. Run it again and all should work out well in the end. If things do go well you will see the following:

PLAY RECAP *********************************************************************

imagewize.nl               : ok=102  changed=6    unreachable=0    failed=0   

localhost                  : ok=0    changed=0    unreachable=0    failed=0   

So as you can see it all went well and six packages were changed:

TASK [fail2ban : ensure fail2ban is configured] ********************************
changed: [imagewize.nl] => (item=jail.local)
ok: [imagewize.nl] => (item=fail2ban.local)

TASK [ferm : ensure iptables INPUT rules are added] ****************************
ok: [imagewize.nl] => (item={u'dport': [u'http', u'https'], u'type': u'dport_accept', u'filename': u'nginx_accept'})
changed: [imagewize.nl] => (item={u'dport': [u'ssh'], u'type': u'dport_accept', u'saddr': [u'89.148.54.121']})
ok: [imagewize.nl] => (item={u'dport': [u'ssh'], u'seconds': 300, u'hits': 20, u'type': u'dport_limit'})

TASK [users : Setup users] *****************************************************
changed: [imagewize.nl] => (item={u'keys': [u'ssh-rsa 
key== user@imagewize.com', u'https://github.com/jasperf.keys'], u'name': u'web', u'groups': [u'www-data']})
changed: [imagewize.nl] => (item={u'keys': [u'ssh-rsa 
keyw== user@imagewize.com', u'https://github.com/jasperf.keys'], u'name': u'dhc-user', u'groups': [u'sudo']})

TASK [users : Add SSH keys] ****************************************************
changed: [imagewize.nl] => (item=({u'name': u'web', u'groups': [u'www-data']}, u'ssh-rsa A
key== user@imagewize.com'))
changed: [imagewize.nl] => (item=({u'name': u'web', u'groups': [u'www-data']}, u'https://github.com/jasperf.keys'))
changed: [imagewize.nl] => (item=({u'name': u'dhc-user', u'groups': [u'sudo']}, u'ssh-rsa 
key== user@imagewize.com'))
changed: [imagewize.nl] => (item=({u'name': u'dhc-user', u'groups': [u'sudo']}, u'https://github.com/jasperf.keys'))

TASK [mariadb : Add MariaDB MySQL deb and deb-src] *****************************
ok: [imagewize.nl] => (item=deb http://nyc2.mirrors.digitalocean.com/mariadb/repo/10.0/ubuntu trusty main)
ok: [imagewize.nl] => (item=deb-src http://nyc2.mirrors.digitalocean.com/mariadb/repo/10.0/ubuntu trusty main)
TASK [mariadb : Restart MariaDB MySQL Server] **********************************
changed: [imagewize.nl]

RUNNING HANDLER [fail2ban : restart fail2ban] **********************************
changed: [imagewize.nl]

In this case keys were changed, MariaDB upgraded and Fail2Ban configured anew.

Ansible Package Updates

Sometimes you do need quicker updates or just want to take care of other package updates. You may want to patch security holes or update packages you need the latest version for urgently. There is a Roots Discourse thread on updates where this is mentioned here. Swalkinshaw mentions here you can do three things for manual updates:

  • Add apt upgrade as a task for a server wide upgrade – did it before myself but do test locally or on staging first!
  • manually specify version for Ansible tasks being run for provisioning – see apt Ansible Books
  • or add latest=yes to any apt action

Real Guess has some scripts that I may implement in the future for Ubuntu updates here. It uses the mentioned Ansible apt module. But the examples Ansible mentions are also very useful.

Quick Security Updates

I do ran updates manually every now and then. And I sometimes do them on the server. To deal with security issues. Not as deterministic as I would like, but it does do the trick. I will incorporate some commands into tasks as soon as I can and write another blog post on it.

Earlier I had this shown on logging into one of my servers at Digital Ocean:

Welcome to Ubuntu 16.04.1 LTS (GNU/Linux 4.4.0-34-generic x86_64)

* Documentation: https://help.ubuntu.com
 * Management: https://landscape.canonical.com
 * Support: https://ubuntu.com/advantage

System information as of Thu Oct 6 06:49:26 UTC 2016

System load: 0.09 Processes: 119
 Usage of /: 20.8% of 24.58GB Users logged in: 0
 Memory usage: 48% IP address for eth0: 208.113.134.123
 Swap usage: 1%

Graph this data and manage this system at:
 https://landscape.canonical.com/

Get cloud support with Ubuntu Advantage Cloud Guest:
 http://www.ubuntu.com/business/services/cloud

95 packages can be updated.
44 updates are security updates.

This was after a general provisioning. So to take care of the security packages I did a:

sudo unattended-upgrades -d
  • add the -d flag to get extra information

which works when the unattend-upgrades package is installed. It is on the Bento box. Running this package will update all unattended security packages – SO Thread. Sometimes including regular packages submitted by security maintainers or so it seems. And when I logged in again I saw:

59 packages can be updated.
6 updates are security updates.

Packages that were updated were:

dnsutils
libbind9-140
libdns-export162
libdns162
libisc-export160
libisc160
libisccc140
libisccfg140
liblwres141
libnss-myhostname
libpam-systemd
libssl1.0.0
libsystemd0
libudev1
linux-headers-generic
linux-headers-virtual
linux-headers-virtual-lts-xenial
linux-image-virtual
linux-libc-dev
linux-virtual
linux-virtual-lts-xenial
ntp
openssl
systemd
systemd-sysv
udev

So quite a few security packages were updated including SSL. New ones popped up and those can be dealt with the same way. This way is also pretty safe and clean. Better to build it into your Trellis workflow of course. I am working on another task for that. However, all in all a good solution to deal with security on your server in a semi automated and clean way. There are ways to run this in the background too, but so far I have preferred to do it this way, especially with the -d flag to keep everything under control.

So there you have it – keeping your Trellis server updated in a nutshell. You got feedback or questions? Leave a comment below and feel free to retweet or post to Facebook.

PS WordPress Core, plugins and theme updates are handled with

./deploy.sh production site.domain

following composer update. Another blog post on that later on.

Create WordPress Database using MySQL Client

Every now and then I need to create a MySQL or MariaDB database from the command line using the MySQL Client. And I tend to forget exactly what commands to run. Lots of things to remember as a web developer. That is why I created loads of gists at Github. Unfortunately their search is not that great and it is often hard to find stuff as you cannot write solid descriptions like you would do on a blog.

So here let me tell you what to do when you need to quickly create the database for a WordPress site from the command line. In my case I have Laravel Valet up and running on OSX with PHP7 and MariaDB running based on Homebrew. So all I need is to log into the MySQL Client and add the database. So how to proceed.

  • open the terminal
  • run the command mysql -u root -p
  • enter your password

That should show you the MySQL client prompt demonstrating you logged on. Then you need to run the following commands – with adjustments for the names and passwords:

CREATE DATABASE databasename;

And then you add privileges for a user:

GRANT ALL PRIVILEGES ON databasename.* TO "wordpressusername"@"hostname" IDENTIFIED BY "password";

Make sure you change the

  • database name to the one you picked,
  • choose a database user and
  • choose a password.

And then you flush all privileges:

FLUSH PRIVILEGES;

So you would see something like this:

mysql> CREATE DATABASE databasename;
Query OK, 1 row affected (0.00 sec)
 
mysql> GRANT ALL PRIVILEGES ON databasename.* TO "wordpressusername"@"hostname" IDENTIFIED BY "password";
Query OK, 0 rows affected (0.00 sec)
 
mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.01 sec)

mysql> EXIT
Bye

As you can see not really hard to do, but nice to have handy to quickly modify, copy and paste is it not?

NB Do not forget to note down your used credentials for adding to wp-config.php

If you would like to check if the database is there run:

SHOW DATABASES;

 

Show Databases

Your database with your name should be there.

Question? Leave a comment here below!

Roots Trellis Errors

In this post I will be collecting Roots Trellis local and remote provisioning, deployment and general setup errors with solutions I came up with. The list is growing and growing so make sure you press control or command f to search for your specific error or use the search box.

Local Setup Errors

I have had quite a few errors setting up my local box and I will gather them here for future reference.

Composer not installed properly

Sometimes vagrant provisioning just hangs on the installation of dependencies and more specifically Composer for PHP dependency management. When I then try to load the local site after the Ansible playbook stopped there I get to see an NGINX 403 access denied error locally.

 TASK: [wordpress-install | Install Dependencies with Composer] **************** 
failed: [default] => (item={'key': 'domain.com', 'value': {'site_install': True, 'permalink_structure': '/%postname%/', 'admin_user': 'admin', 'local_path': '../site', 'cache': {'duration': '30s', 'enabled': False}, 'ssl': {'enabled': False}, 'multisite': {'enabled': False, 'subdomains': False}, 'site_title': 'Example Site', 'admin_password': 'admin', 'env': {'db_name': 'domain_dev', 'db_user': 'imagewize_dbuser', 'wp_env': 'development', 'db_password': 'domain_dbpassword', 'disable_wp_cron': True, 'wp_home': 'http://domain.dev', 'wp_siteurl': 'http://domain.dev/wp'}, 'site_hosts': ['domain.dev'], 'admin_email': 'admin@domain.dev'}}) => {"changed": true, "cmd": ["composer", "install"], "delta": "0:00:00.062962", "end": "2015-12-29 10:11:13.064551", "item": {"key": "imagewize.com", "value": {"admin_email": "admin@domain.dev", "admin_password": "admin", "admin_user": "admin", "cache": {"duration": "30s", "enabled": false}, "env": {"db_name": "domain_dev", "db_password": "domain_dbpassword", "db_user": "domain_dbuser", "disable_wp_cron": true, "wp_env": "development", "wp_home": "http://domain.dev", "wp_siteurl": "http://domain.dev/wp"}, "local_path": "../site", "multisite": {"enabled": false, "subdomains": false}, "permalink_structure": "/%postname%/", "site_hosts": ["domain.dev"], "site_install": true, "site_title": "Example Site", "ssl": {"enabled": false}}}, "rc": 1, "start": "2015-12-29 10:11:13.001589", "stdout_lines": [], "warnings": []}
stderr: You are running composer with xdebug enabled. This has a major impact on runtime performance. See https://getcomposer.org/xdebug
Composer could not find a composer.json file in /srv/www/domain.com/current
To initialize a project, please create a composer.json file as described in the https://getcomposer.org/ "Getting Started" section

Best solution I found is

vagrant destroy

and

vagrant up

Just a

vagrant provision

did not do the trick

Package cannot be authenticated

Local setup errors setting up my Vagrant Box I seem to get often when I do not use my VPN are:

The following SSH command responded with a non-zero exit status.
Vagrant assumes that this means the command failed!
apt-get install -y bindfs
Stdout from the command:
Reading package lists...
Reading state information...
The following NEW packages will be installed:
bindfs
0 upgraded, 1 newly installed, 0 to remove and 1 not upgraded.
Need to get 25.1 kB of archives.
After this operation, 89.1 kB of additional disk space will be used.
WARNING: The following packages cannot be authenticated!
bindfs
Stderr from the command:
E: There are problems and -y was used without --force-yes

The package cannot be authenticated somehow. No idea why really, but seems to happen to me out here in Bahrain. Solution is using another network or a VPN and running

vagrant provision

to set all up well locally.

Unexpected Exception: TaskInclude

Installation goes well, but just before all is done you get

ERROR! Unexpected Exception: 'TaskInclude' object has no attribute 'has_triggered'
Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.

No solution yet either, besides destroying the box and starting from scratch. Googling did not dig up any dirt either. Well, one sort of related thread here, but has done me no good. Apparently the Ansible 2.1.0 version has a regression so 2.0.2 needed to be installed. See Roots Discourse thread here. Could not do that using Homebrew so had to remove Ansible, and then install it using Python’s Pip. There I had issues getting it to install at all due to OSX Mavericks and El Capitan issues. Found a solution for that here. Also needed a Python setuptools upgrade to avoid other errors  such as:

ERROR! Unexpected Exception: (setuptools 1.1.6 (/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python), Requirement.parse('setuptools>=11.3'))

SO thread here.

Dict Object has no Attribute

Setting up or provisioning the local server / Vagrant box I ran into this error as well:

TASK: [wordpress-setup | Create/assign database user to db and grant permissions] *** 
fatal: [default] => One or more undefined variables: 'dict object' has no attribute 'domain.com'Provisioning Errors

Check if vault.yml has the correct site name.

Unable to find Inventory File

ERROR: Unable to find an inventory file, specify one with -i ?

This error showed up  when I want to provision a server I did provisioning on before using a different project and I forgot about. Better to start out with an empty instance. Command leading to this error is:

ansible-playbook server.yml -e env=staging

Solution was wiping out Trellis and start from scratch and then provision again

502 Bad Gateway

After provisioning a local vagrant box I ran into this error:

502 Bad Gateway
nginx

Suspending the Vagrant box and restarting did not help. Checking the logs you may see something like:

*8 connect() to unix:/var/run/php-fpm-wordpress.sock failed (2: No such file or directory) while connecting to upstream, client: 192.168.50.1, server: domain.dev, request: "GET /favicon.ico HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm-wordpress.sock:", host: "domain.dev", referrer: "http://domain.dev/"

Meaning there was an issue with Fast CGI. The path to the socket should be

/var/run/php/php7.0-fpm.sock

So I opened

/etc/php/7.0/fpm/pool.d/wordpress.conf

and adjusted it. Then restarted the server:

sudo service nginx restart

No joy. Still was loaded from another location it seems though I could not locate it inside php.ini. So I destroyed the box:

vagrant destroy

and

 vagrant up

and re-created it. And then all was well again. If I do bump into this issue again and manage to solve it manually I will let you guys know.

sSMTP Installation Error

Another provisioning error I seem to get when I work with multiple domain names connected to one elastic IP is an issue installing sSMTP

 failed: [xx.xxx.xx.xx] => {"failed": true}
 stderr: hostname: Name or service not known
 dpkg: error processing package ssmtp (--configure):
 subprocess installed post-installation script returned error exit status 1
 Errors were encountered while processing:
 ssmtp
 E: Sub-process /usr/bin/dpkg returned an error code (1)
stdout: Reading package lists...
 Building dependency tree...
 Reading state information...
 The following NEW packages will be installed:
 ssmtp
 0 upgraded, 1 newly installed, 0 to remove and 4 not upgraded.
 Need to get 46.2 kB of archives.
 After this operation, 8192 B of additional disk space will be used.
 Get:1 http://us.archive.ubuntu.com/ubuntu/ trusty/universe ssmtp amd64 2.64-7 [46.2 kB]
 Preconfiguring packages ...
 Fetched 46.2 kB in 0s (366 kB/s)
 Selecting previously unselected package ssmtp.
 (Reading database ... 70521 files and directories currently installed.)
 Preparing to unpack .../ssmtp_2.64-7_amd64.deb ...
 Unpacking ssmtp (2.64-7) ...
 Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
 Setting up ssmtp (2.64-7) ...
msg: '/usr/bin/apt-get -y -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" install 'ssmtp'' failed: hostname: Name or service not known
 dpkg: error processing package ssmtp (--configure):
 subprocess installed post-installation script returned error exit status 1
 Errors were encountered while processing:
 ssmtp
 E: Sub-process /usr/bin/dpkg returned an error code (1)

I found an issue mentioning it here at Github https://github.com/roots/trellis/issues/148 suggesting to check /etc/hosts and /etc/hostname. Hostname had imagewize (copied from Dreamhost instance name given). Still working on this issue and adding possible FQDN . They also suggested using a domain name, but I guess I my case it was using one domain too many. In the end I moved one staging site to another server.

 

Publickey – Permission denied

Another issue I run into sometimes is an issue with the SSH public key:

GATHERING FACTS ***************************************************************
fatal: [xx.xxx.xx.xx] => SSH Error: Permission denied (publickey).
while connecting to xx.xxx.xx:22
It is sometimes useful to re-run the command using -vvvv, which prints SSH debug output to help diagnose the issue.

Often this means one of these issues:

  • your SSH public key is not OK so you need to update your remote server at Digital Ocean or DreamCompute with the latest id_rsa.pub for example
  • you switched server and need to change IP inside hosts/<environment>
  • wrong username to access server – make sure the users.yml has the correct user – For DreamCompute it should not be admin but dhc-user.

Deployment Errors

Another error trying to deploy  I got was really odd or by the looks of it initially as it mentioned paths to Ansible files I could not locate right away and stuff leading to SSH key issues in the end:

./deploy.sh staging domain.com
PLAY [Deploy WP site] *********************************************************
GATHERING FACTS ***************************************************************
ok: [xx.xxx.xx.xx]

TASK: [deploy | Initialize] ***************************************************
failed: [xx.xxx.xx.xx] => {"failed": true, "parsed": false}
Traceback (most recent call last):
File "/home/web/.ansible/tmp/ansible-tmp-1451380440.18-256782094434614/deploy_helper", line 2026, in
main()
File "/home/web/.ansible/tmp/ansible-tmp-1451380440.18-256782094434614/deploy_helper", line 382, in main
changes += deploy_helper.create_path(facts['project_path'])
File "/home/web/.ansible/tmp/ansible-tmp-1451380440.18-256782094434614/deploy_helper", line 276, in create_path
os.makedirs(path)
File "/usr/lib/python2.7/os.py", line 150, in makedirs
makedirs(head, mode)
File "/usr/lib/python2.7/os.py", line 157, in makedirs
mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/srv/www'
OpenSSH_6.9p1, LibreSSL 2.1.8
debug1: Reading configuration data /Users/jasper/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: auto-mux: Trying existing master
debug1: mux_client_request_session: master session id: 2
Shared connection to xx.xxx.xx.xx closed.

Error was related to previous provision issue and a clean slate with a clean remote server and new Trellis installation was all I could come up with to solve it all.

NGINX Error 404

I also bumped into a:

404 Not Found
 nginx

error. This was simply due to the fact that I had provisioned, but hadn’t deployed. Yet. Once that was done all was well.

Wrong Subtree

TASK: [deploy | Fail if project_subtree_path is set incorrectly] ************** 
failed: [xxx.xxx.xx.xx] => {"failed": true}
msg: subtree is set to 'site' but that path does not exist in the repo. Edit `subtree_path` for 'domain.com' in `wordpress_sites.yml`.

Make sure you have added the correct repo to the wordpress_sites.yml and that the subtree is correctly set. Last time I switched from Github to Bitbucket (free private repos!) but I did not adjust the repository.

Repo Access Error

Had an issue a few times that I suddenly had an error that the remote repository could no longer be accessed while I could push and pull from the Github repository. This made it impossible to deploy. See the errors below:

TASK [deploy : Clone project files] ********************************************
System info:
 Ansible 2.1.1.0; Darwin
 Trellis at "Fix #639 - WP 4.6 compatibility: update WP-CLI to 0.24.1"
---------------------------------------------------
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

fatal: [domain.nl]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result"}
...ignoring

TASK [deploy : Failed connection to remote repo] *******************************
System info:
 Ansible 2.1.1.0; Darwin
 Trellis at "Fix #639 - WP 4.6 compatibility: update WP-CLI to 0.24.1"
---------------------------------------------------
Git repo git@github.com:jasperf/domain.nl.git cannot be accessed. Please
verify the repository exists and you have SSH forwarding set up correctly.
More info:
> https://roots.io/trellis/docs/deploys/#ssh-keys
> https://roots.io/trellis/docs/ssh-keys/#cloning-remote-repo-using-ssh-
agent-forwarding

fatal: [domain.nl]: FAILED! => {"changed": false, "failed": true}

NO MORE HOSTS LEFT *************************************************************
 [WARNING]: Could not create retry file 'deploy.retry'. [Errno 2] No such
file or directory: ''

The issue in the end was probably related to my upgrade to OSX Sierra and some system and or keychain changes. I had to enter my passphrase earlier. And somehow the key had to be added to the keychain again as well. This as a simple:

ssh-add -K
Identity added: /Users/jasper/.ssh/id_rsa (/Users/jasper/.ssh/id_rsa)

After that I could access all again and deploy the latest changes to the site.