Nginx + Certbot on multiple servers

This article describes the solution of a following problem:

• you have multiple servers to which HTTPS traffic is passed, for load balancing and redundancy
• you want to use Let’s Encrypt issued certificates, since they are free
• you share no volumes among these machines, ie. each one has a separate filesystem
• you use Docker to deploy your things
• you use Docker Swarm to orchestrate your containers (although it’s pretty easily adaptable to Kubernetes)

This article describes the approach that I used to rectify the problem using ZooKeeper as a distributed file system, however this approach can be easily modified if you are already using a shared filesystem, by simply changing some obvious configuration.

Configuration and the solution

Let’s say that I have three machines, called 192.168.0.2, 192.168.0.3 and 192.168.0.4. They accept HTTPS connection which are load-balanced to them via a entry point router/haproxy/whatever on port 443. They are to terminate SSL and dispatch requests to our services via HTTP.

To solve this problem we’ll create a single point of entry to all /.well-known/acme-challenge/ that will receive the requests and issue certificates, and make them available to nginxes with a shared volume.

The solution: certmagic

The solution is to tell each domain in every nginx to proxy their requests on /.well-known/acme-challenge/ to a special container, further called certmagic, which will respond, issue the certificates and store them on a shared volume. This service will be tasked with:

1. Receiving requests from http://<YOUR_DOMAIN>/.well-known/acme-challenge/<TOKEN>
2. Invoking requests for this certficate
3. Storing the certificates in /etc/letsencrypt
4. Upon spawning it will try to renew the certificates, then it will wait a day and exit, trusting Docker Swarm/Kubernetes to respawn it.

I chose to respawn the container rather than wait in a loop because my ZooKeeper plugin failed sometimes to mount the volume, and the container was basically stuck.

Warning! There must be exactly one instance of this container, since if there were two one certbot would receive requests made by the other certbot, and that would kind of defeat the purpose.

Basically, it will be a nginx running with certbot. Let’s go through the Dockerfile for it:

FROM nginx

RUN apt-get update && \
    apt-get install --no-install-recommends -y python3-pip python3-setuptools rustc \
                                               python3-dev libssl-dev cargo python3-wheel

RUN pip3 install cryptography
RUN pip3 install certbot certbot-nginx

COPY certmagic.sh /docker-entrypoint.d/certmagic.sh
RUN chmod ugo+x /docker-entrypoint.d/certmagic.sh

And certmagic.sh:

#!/bin/bash

rm -rf /etc/letsencrypt/.certbot.lock

certbot renew
sleep 86400    # a day

Note that you also need to automatically reload nginxes. It is important to frequently check is /etc/letsencrypt is still mounted, as I’ve seen my plugin fail multiple times (if you catch the problem and can file a bug request, please do so). them. If it isn’t, we’ll simply suicide the container and Docker Swarm should pick it up again. The Dockerfile is:

FROM nginx

ADD reload-me.sh /docker-entrypoint.d/reload-me.sh
ADD auth_file /etc/nginx/auth_file
ADD conf.d /etc/nginx/conf.d
RUN chmod ugo+x /docker-entrypoint.d/reload-me.sh

While reload-me.sh is:

#!/bin/bash

(while :; do
  for i in {1..24}; do
    command
    sleep 3600
    if ls /etc/letsencrypt/; then
      true
    else
      echo "/etc/letsencrypt is not mounted, quitting"
      kill -9 1
      exit
    fi
  done
  echo "Reloading nginx configuration"
  kill -1 1 # reload nginx configuration
done) &

The script basically checks every hour if /etc/letsencrypt is available, and if it’s not it kills the container. Once a day it will send a signal to nginx to reload it’s configuration (and the certificates).

The solution: shared filesystem

I wrote a Docker plugin that allows you to mount ZooKeeper volumes as filesystems. Since the existing zookeeper-fuse tool did not had enough capabilities to act as a fully-fledged filesystem (it did not support renaming file or symlinking them), I issued a pull request that adds these features.

Then basing off that I wrote a Docker volume plugin called zookeeper-volume, that allows to mount ZooKeeper filesystems as a Docker volume.

After configuring ZooKeeper on each of the machines, we can deploy the following stack on the system:

version: '3.5'
services:
  certmagic:
    image: your_certmagic_image
    deploy:
      mode: replicated
      replicas: 1
      restart_policy:
        condition: any
        delay: 20s
    networks:
      - public
    volumes:
      - certs:/etc/letsencrypt
  nginx:
    volumes:
      - certs:/etc/letsencrypt
    image: your_nginx_image
    deploy:
      mode: replicated
      replicas: 3
      restart_policy:
        condition: any
        delay: 20s
    ports:
      - published: 80
        target: 80
        protocol: tcp
        mode: ingress
      - published: 443
        target: 443
        protocol: tcp
        mode: ingress
    networks:
      - public
networks:
  public:
    external: true
    name: public
volumes:
  certs:
    driver: smokserwis/zookeeper-volume
    driver_opts:
      hosts: "192.168.0.2:2181,192.168.0.3:2181,192.168.0.4:2181"
      mode: "HYBRID"

The usage

In order to use the solution you will first need to configure given domain on nginx, and to reverse proxy all calls to /.well-known/acme-challenge/ to certmagic. You can do this with a following section in nginx.conf, the server part:

    resolver 127.0.0.11 valid=30s;
    set $certmagicupstream certmagic:80;
    location /.well-known/acme-challenge/ {
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_pass http://$certmagicupstream;
    }

Setting a resolver explicitly is required for the nginx to resolve DNS using Docker Swarm’s DNS routing mesh, otherwise it will use the hosts’s resolver, which obviously doesn’t know what certmagic is.

Then you issue certbot -d yourdomain.example.com, check that certificates are available on /etc/letsencrypt, and attach them to the domain in nginx. Just put the following in your server part:

listen 80;
listen 443 ssl http2;
ssl_certificate /etc/letsencrypt/live/yourdomain.example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/yourdomain.example.com/privkey.pem;

Happy hacking!