When setting up this website/blog I wanted to keep the maintainability of the system as low as possible, as VegerWeb is not a big company with its own IT department. So an orchestration tool was required to keep the servers up-to-date, install and configure new software, etc. Due to good experiences in the past I chose Ansible again, especially because there are lots of roles and modules available for all kinds of tasks, and it does not take a lot of time of the IT department (me) to set things up. As a web server I chose NGINX  as I just like it being light-weight (small companies do not have lots of resources to spend) while keeping its (required) functionalities.

After setting my old websites and this new one up using Ansible, I wanted to provide secure communication (HTTPS) possibilities for my websites. I chose to use Let's Encrypt to obtain the required TLS/SSL certificates. Let's Encrypt is a nice nonprofit organization that believe in free certificates to make the Internet more safe. I like this idea (and... its free)!

This post is written for Ansible, NGINX and Ubuntu 16.04. But changing for example NGINX with Apache, or Ubuntu 16.04 with 14.04 does not change the fundamentals.

Requirements

Additionally, I want to keep the web server up-and-running while renewing the certificate in order to completely make this process transparent to visitors and avoid/minimize down-time. The Let's Encrypt client provides the webroot plugin (authentication method) for this. It uses the (running) web server to authenticate the certificate renewal request, using a well known location that is used by the client to put a secret in. This secret is checked by the Let's Encrypt servers using the provided domain(s) to see if you really own it/them.

Firmly believing in the availability of Ansible roles, I searched for a role that performed these steps. Lo and behold, there is no complete role for all of these steps..! The letsencrypt roles with webroot support either have some known issues or do not provide all steps (also the roles not listed in Ansible Galaxy).

Preparation

Installing an application using Ansible is easy: add apt repository (ppa:certbot/certbot), install software and finish with providing some additional/specific configuration (letsencrypt account, work, log, etc directories). So no need to provide extensive details on this.

Obtaining the initial certificate

Obtaining the initial certificate is also fairly easy, although a properly configured and running web server (NGINX in my case) is required:

server {
    listen 443 ssl http2;
    server_name www.example.com;

    ssl_certificate /etc/letsencrypt/live/{{ site_server_name }}/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/{{ site_server_name }}/privkey.pem;
    # Rest of SSL/TLS configuration is skipped

    location ~ /.well-known {
        root /var/www/letsencrypt;
    }

    # Rest of actual server configuration, installed with Ansible
}

The web server (serving https://www.example.com) obviously requires certificates in order to serve content. This poses the first issue: there is no certificate yet, so nginx does not start. Note that Ansibles philosophy is to keep files in a certain state and thereby being idempotent, so having a different NGINX configuration for the first run is not desired. Adding a configuration specifically for Let's Encrypt might be possible, but it requires the correct domain which makes it hard(er) to have a separate configuration.

I decided to install the default snake-oil certificates when no certificates are available:

- name: Check if letsencrypt certificate is present
  stat:
    path: "/etc/letsencrypt/renewal/{{ site_server_name }}.conf"
  register: cert_installed

- name: Install snakeoil cert
  copy:
    src: "{{ item.src }}"
    dest: "{{ item.dest }}"
    remote_src: yes
  when:
  - not cert_installed.stat.exists
  with_items:
  - { src: /etc/ssl/certs/ssl-cert-snakeoil.pem, dest: "/etc/letsencrypt/live/{{ site_server_name }}/fullchain.pem" }
  - { src: /etc/ssl/private/ssl-cert-snakeoil.key, dest: "/etc/letsencrypt/live/{{ site_server_name }}/privkey.pem" }

Now NGINX is able to start, so the initial certificate can be obtained:

- name: Check if letsencrypt is active/configured
  stat:
    name: "/etc/letsencrypt/renewal/{{ site_urls[0]}}.conf"
  register: cert_installed

- name: Get rid of live directory # containing snakeoil cert
  file:
    path: "/etc/letsencrypt/live/{{ site_urls[0] }}"
    state: absent
  when: cert_installed.stat.exists

- name: Obtain initial certificate
  shell: "certbot certonly --webroot -n -w /var/www/letsencrypt/ -d {{ site_urls | join(' -d ') }} -m {{ site_email }} --agree-tos"
  when: cert_installed.stat.exists

The NGINX configuration can be reloaded at this point, so it NGINX uses the correct certificate, if required.

Update certificate periodically

The obtained certificate is (only) valid for 3 months, so it is required to check if a new certificate can be obtained periodically. The Let's Encrypt client has a renew command for this. So a simple cron job is sufficient to renew the certificate:

- name: Ensure job to renew certificates exists
  cron:
    name: Renew certificates
    job: "/bin/bash -c '{ /usr/bin/letsencrypt renew --renew-hook \"systemctl reload nginx\" | /usr/bin/logger -t letsencrypt -p syslog.info -i; } 2>&1 | /usr/bin/logger -t letsencrypt -p syslog.err -i'"
    cron_file: letsencrypt
    special_time: monthly
    user: letsencrypt

The --renew-hook is used to reload the NGINX configuration, and logger is used to get the results/attempts in syslog using the info level for stdout and the error level for stderr.

I chose the check monthly for a new certificate, why checking more often if we know the certificate is valid for 3 months? If you decide to check more other though, be sure to keep it reasonable to prevent issues with Let's Encrypt.

Bonus: Adding a new domain to the existing certificate

Not completely related, but still a nice feature to end this post with: When adding a domain to site_urls it is ignored as cert_installed shows that it is already installed. By changing the check a little and using grep, it is possible to check if all domains are part of the Let's Encrypt configuration:

- name: Check if letsencrypt has the correct configuration # i.e. all of the required (sub) domains
  shell: "grep -q {{ site_urls | join(' /etc/letsencrypt/renewal/' + site_urls[0] + '.conf && grep -q ') }} /etc/letsencrypt/renewal/{{ site_urls[0]}}.conf && echo ok || echo not found"
  changed_when: false
  register: cert_installed

- name: Get rid of old configuration # Forces to grab the initial certificate (with new sub domains) again
  file:
    path: "/etc/letsencrypt/renewal/{{ site_urls[0]}}.conf"
    state: absent
  when: cert_installed.stdout != 'ok'

Make sure to change the checks of the other tasks to when: cert_installed.stdout != 'ok'.

Note that removing the /etc/letsencrypt/live/domain/ directory (instead of just removing the snake oil certificates), fixes the known issue of one of the more complete Ansible letsencrypt role.

Conclusion

This is about everything that is needed to let NGINX and Let's Encrypt play nice using Ansible while adhering to its philosophy of impotence.

I did not create a (public) role for this, as my current implementation is (somewhat) intertwined with the rest of my playbook... Feel free to let me know if this post was enough to get you going, or you would like/need the complete role. If there is enough interest, I might be tempted to remove some dependencies and make the role public.