Ansible Nginx Let's Encrypt Automation

Automatic SSL certificate issuance and renew with Ansible and Let’s Encrypt

Here at Lenses.io, we prototype fast, and new (sub)domains are frequently added to complement our back and front-end services. Since the beginning, our specifications included “ssl everywhere”. The journey into providing fully secure and encrypted services is a long one; hence the need to adventure in the SSL land.

The tools, the needs.

We use Ansible to manage our servers. Part of our infrastructure is on Google Cloud. Ansible’s support of Google Compute Engine leaves much to be desired but otherwise we are happy with its main function.

We use nginx as a web server, proxy and SSL termination endpoint for services. There are other great servers too, but we try to keep some consistency on our configurations.

We needed an easy way to get certificates for new services. Waiting for the DevOps team to get a certificate issued and deployed was a burden to the rest of the crew. We also needed certificates to be deployed per server and to keep backups of them. It is not rare to have more than one service in each server.

As the DevOps engineer I also wanted a way to renew the SSL certificates automatically, and get all the above for free :)

The solution we came up was exactly this! Use a template to add a new ‘nginx site configuration file’ to Ansible and deploy the new configuration. Our solution automatically gets the new certificate issued, backs up and sets a cron job to renew it when needed.

The new tools.

Let’s encrypt is the new cool kid in town. It is a Certificate Authority (CA) that offers free SSL certificates through an API. I waited for them for months to go live —they entered their public beta in December 2015— and was a little too happy to use them for Lenses.io's infrastructure. They offered everything I dreamed of —except DNS validation, which now they do— and some more.

To work with let’s encrypt you also need a client. The official one is a bit too intrusive for my taste, tampering with the web server’s configuration files. Thankfully the API protocol, ACME, is well documented and various clients do exist. There is also a nice little server called caddy that supports let’s encrypt out of the box; therefore, doing everything automatically.

The client of choice in our case is acmetool, a full featured client written in Go (thus it is a statically linked executable) which supports various ways of handling let’s encrypt's challenge responses.

Challenge responses are used by the CA to verify that you indeed own the domain you ask a certificate for. Handling them is the trickiest part since you should serve them from the same server and domain that your site or service uses. Furthermore when you have a single nginx serving more than one domains, things get a bit more complicated as each challenge response should be served by the correct domain.

The solution

It is far easier and useful to give you a working Ansible role. Thus, with great joy, I present you with our Ansible examples repository on GitHub. The role you should look for is roles/nginx-lets-encrypt. It has been tested with Ansible 2.0 and is derived from our Ansible configuration. Since it is a bit different from it, it isn’t battle-tested, so your mileage may vary. Even if it doesn’t work for your case, I am positive it shouldn’t need anything more than a simple fix to work.

The role we demonstrate automatically requests missing certificates for sites and backs them up into the repository. Using a tool like git-crypt to keep the backups protected should be obvious.

How it works

The role first installs an acmetool release. If there is a backup configuration for the tool, it will restore it. If not will it configure acmetool with the files/acmetool/response-file.yml configuration. It is important to place a valid email address in there since this is the only way to revoke a certificate if needed. We also install a cronjob that checks and renews certificates as needed. Let’s encrypt’s certs are valid for 90 days.

Then nginx is installed and configured. We have ready to use settings for handling challenge responses and ssl, which is configured as suggested by Mozilla’s SSL Config Generator. It scores a solid A in SSL Labs’ Test.

You may place configurations for your sites inside files/conf.d. Please use files/conf.d/TEMPLATE.conf.sample as a template for your sites.

We chose to use proxy redirection for handling challenge responses. When acmetool asks for a new certificate, it starts a web server listening on port 402 and serving the responses. Nginx will redirect let’s encrypt's requests there. Once a certificate is acquired, acmetool exits, as it does not run permanently. The only time acmetool is running is when using Ansible to deploy a new site or via the cron-job for renewing certs.

Once your site configuration files are installed, we run our custom bash script named certs-checks-gets.sh. Let’s encrypt is careful and serious about issuing certificates and imposes a limit of 5 certs per domain per week. Thus we have to be careful not to make many experiments and have to wait for a week to get our certificate. certs-checks-gets.sh will check for missing keys from your nginx configuration files. If it finds any it will check that you have proper DNS records for them. We use the ansible_ssh_host ip address for this.

Once everything checks out, it will request your certificates. You may notice that our nginx configuration is invalid at this point since our certificate files are missing. This means that nginx can’t start. certs-checks-gets.sh knows this and replaces the site’s nginx configuration with a simple one that takes care of challenge response requests. When finished, it will restore your site configuration and nginx will reload with the proper settings.

The last step is to create an archive for acmetool’s data directory (/var/lib/acme) and download it into your repository, under files/backups/<HOSTNAME>/acme.tbz. Use git-crypt or another tool to protect these files.

That’s all for today. Again, you can find the role in our our Ansible examples repository.

Thank you for your interest !

Marios.