LetsEncrypt

Moving to LetsEncrypt Server Certificates

Certbot

Install

sudo apt-get update
sudo apt-get install certbot python-certbot-apache

The interesting parts are in /etc/letsencrypt

Ubuntu 14.04

On an Ubuntu 14.04 system, need to add the certbot distro:

sudo add-apt-repository ppa:certbot/certbot
sudo apt-get update
sudo apt-get install certbot python-certbot-apache

It will also be necessary to download the LetsEncrypt intermediate certficate:

sudo curl -o /etc/letsencrypt/lets-encrypt-x3-cross-signed.pem \
  "https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem.txt"

If add-apt-repository is not found:

sudo apt-get install software-properties-common

Create

Certbot is pretty much automatic for apache, providing the config is as expected. Otherwise:

sudo certbot certonly --webroot \
-w /var/www/project/html -d project.dataone.org

The certificates are located in /etc/letsencrypt/live/DOMAIN

Renew

Called when a new certificate is issued:

/etc/letsencrypt/renew-hook.d$ cat apache-restart
#!/bin/sh

set -e

service apache2 restart >/dev/null

In /etc/cron/weekly:

#!/bin/sh
# Try renewing certificates

logger "Checking certificate renewal for all hosts"
/usr/bin/certbot renew -n --quiet \
  --renew-hook "/bin/run-parts /etc/letsencrypt/renew-hook.d/"

Multi-home Systems

On systems like search.dataone.org and the Coordinating Nodes, the certificate issuance and renewal process is a more complicated since each of these systems has two names one of which is shared across the search servers or CNs. During creation or renewal of a certificate, certbot will verify that the request is coming from the same system by placing a verification document in a well known location on the server. The letsencrypt servers will then check that the requested name resolves to a location where the verification document can be retrieved before issuing the certificate.

The generated certificate should contain the names of all participating servers and will also need to be shared with each participating node. Generating the certificate requires that the verification document be available at the expected location. Use of the certificate requires that the generated certificate is copied to the participating nodes.

The approach taken is to only generate the certificate on the machine that is the current primary. Each of the respective nodes is added to the certificate request as alternate names. For verification of the specific names, each of the respective servers proxies to the general name where the verification document can be retrieved. The resulting certificate is distributed to the other participating nodes using rsync.

Apache Configuration

<VirtualHost *:80>
  ###
  # Configuration specific to this server instance
  ###
  ServerName search-unm-1.dataone.org
  ServerAdmin administrator@dataone.org
  DocumentRoot /var/www/search.dataone.org
  AllowEncodedSlashes On
  AcceptPathInfo      On
  ErrorLog ${APACHE_LOG_DIR}/error.log
  CustomLog ${APACHE_LOG_DIR}/access.log combined

  # Route all traffic except what is used by letsencrypt certbot to
  # search.dataone.org
  <LocationMatch "^(?!/.well-known)/[^/]+">
    redirect permanent / https://search.dataone.org/
  </LocationMatch>

  ## Proxy all requests to search.dataone.org which is where certbot
  ## is being run and creating the verification files
  ProxyPass "/.well-known/" "http://search.dataone.org/.well-known/"
  ProxyPassReverse "/.well-known/" "http://search.dataone.org/.well-known/"
</VirtualHost>

<VirtualHost *:80>
  ###
  # This config only comes into play when DNS is pointing to this server
  ###
  ServerName search.dataone.org
  ServerAdmin administrator@dataone.org
  DocumentRoot /var/www/search.dataone.org
  AllowEncodedSlashes On
  AcceptPathInfo      On
  ErrorLog ${APACHE_LOG_DIR}/error.log
  CustomLog ${APACHE_LOG_DIR}/access.log combined

  # Redirect all traffic except certbot to HTTPS
  <LocationMatch "^(?!/.well-known)/[^/]+">
    redirect permanent / https://search.dataone.org/
  </LocationMatch>
</VirtualHost>

In the https configuration it was necessary to add the intermediate letsencrypt certificate in SSLCertificateChainFile otherwise ssllabs complained about an incomplete chain:

SSLEngine on
SSLCertificateKeyFile  /etc/letsencrypt/live/search.dataone.org/privkey.pem
SSLCertificateFile  /etc/letsencrypt/live/search.dataone.org/fullchain.pem
SSLCertificateChainFile /etc/letsencrypt/lets-encrypt-x3-cross-signed.pem

Certificate Generation

sudo certbot certonly --webroot \
  --cert-name search.dataone.org \
  -w /var/www/search.dataone.org -d search.dataone.org \
  -d search-ucsb-1.dataone.org \
  -d search-unm-1.dataone.org \
  -d search-orc-1.dataone.org

sudo certbot certonly --webroot \
  --cert-name cn.dataone.org \
  -w /var/www -d cn.dataone.org \
  -d cn-ucsb-1.dataone.org \
  -d cn-unm-1.dataone.org \
  -d cn-orc-1.dataone.org

Certificate Distribution

After generation, the certificates need to be distributed to other servers participating in the group. This can be done using rsync over ssh.

Setup a user to do the rsync operations:

RUSER=some_user_name_for_rsync
sudo adduser ${RUSER} --disabled-password
sudo su - ${RUSER}
mkdir .ssh
chmod 0700 .ssh
cd .ssh
ssh-keygen -N "" -f id_rsa
cp id_rsa.pub authorized_keys
chmod 0600 *
cd ~
mkdir bin
nano bin/rsync-wrapper.sh
  ...
chmod u+x bin/rsync-wrapper.sh

Allow the RUSER to use sudo rsync:

$ nano /etc/sudoers.d/certbot-sync

RUSER ALL=NOPASSWD:/usr/bin/rsync

sudo rsync -avu

--rsync-path="/home/rsync_user/bin/rsync-wrapper.sh" -e "ssh -i /home/rsync_user/.ssh/id_rsa -l rsync_user" /etc/letsencrypt/* rsync_user@cn-unm-1.dataone.org:/etc/letsencrypt/

sudo rsync -avu

--rsync-path="/home/rsync_user/bin/rsync-wrapper.sh" -e "ssh -i /home/rsync_user/.ssh/id_rsa -l rsync_user" /etc/letsencrypt/* rsync_user@cn-orc-1.dataone.org:/etc/letsencrypt/

Run rsync on the primary:

sudo rsync -avu \                                         [1]
  --rsync-path="/home/rsync_user/bin/rsync-wrapper.sh" \  [2]
  -e "ssh -i /home/rsync_user/.ssh/id_rsa -l RUSER" \     [3]
  /etc/letsencrypt/* \                                    [4]
  rsync_user@search-orc-1.dataone.org:/etc/letsencrypt/   [5]

[1]:	-a = archive mode, recursive + symlinks + preserve permissions + preserve modification times + preserve group + preserve owner + preserve device files -v = verbose -u = skip files that are newer on the receiver
[2]:	Remote path of rsync command to run
[3]:	-e = specify the remote shell to use ssh -i = ssh using specified identity file -l = user name for shell
[4]:	Source path
[5]:	Destination path

After distributing the certificates it is necessary to restart apache on each machine:

sudo service apache2 restart

This step must be done manually on the search* and cn* machines..

Apache Config

General apache config for typical server setup.

/etc/apache2/ports.conf:

Listen 80
<IfModule ssl_module>
        Listen 443
        NameVirtualHost *:443
        SSLStrictSNIVHostCheck off
</IfModule>
<IfModule mod_gnutls.c>
        Listen 443
        NameVirtualHost *:443
        SSLStrictSNIVHostCheck off
</IfModule>

Multi-vhost configuration apache with redirect to https:

<VirtualHost 64.106.84.4:80>
  ServerName examples.dataone.org
  ServerAdmin d1.sysadmin@gmail.com
  DocumentRoot /var/www/examples/html
  Header set Access-Control-Allow-Origin "*"

  #.well-known is used by lets-encrypt certbot
  <LocationMatch "^(?!/.well-known)/[^/]+">
    redirect permanent / https://examples.dataone.org/
  </LocationMatch>
  LogLevel info ssl:warn
  ErrorLog ${APACHE_LOG_DIR}/examples-error.log
  CustomLog ${APACHE_LOG_DIR}/examples-access.log combined
</VirtualHost>

<IfModule mod_ssl.c>
  <VirtualHost 64.106.84.4:443>
    ServerAdmin d1.sysadm@gmail.com
    ServerName examples.dataone.org
    DocumentRoot /var/www/examples/html

    AllowEncodedSlashes On
    AllowEncodedSlashes NoDecode

    Header set Access-Control-Allow-Origin "*"

    <Directory />
        Options Indexes FollowSymLinks
        AllowOverride All
      Require all granted
      Allow from all
    </Directory>

    ## Site config stuff here

    LogLevel info ssl:warn
    ErrorLog ${APACHE_LOG_DIR}/examples-error.log
    CustomLog ${APACHE_LOG_DIR}/examples-access.log combined

    SSLEngine on
    SSLCertificateKeyFile  /etc/letsencrypt/live/examples.dataone.org/privkey.pem
    SSLCertificateFile  /etc/letsencrypt/live/examples.dataone.org/fullchain.pem

    <FilesMatch "\.(cgi|shtml|phtml|php)$">
      SSLOptions +StdEnvVars
    </FilesMatch>
    <Directory /usr/lib/cgi-bin>
      SSLOptions +StdEnvVars
    </Directory>

    BrowserMatch "MSIE [2-6]" \
      nokeepalive ssl-unclean-shutdown \
      downgrade-1.0 force-response-1.0
    # MSIE 7 and newer should be able to use keepalive
    BrowserMatch "MSIE [17-9]" ssl-unclean-shutdown
  </VirtualHost>
</IfModule>

Alternative support redirect to https for everything except /.well-known:

RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule  ^/(?!.well-known)(.*) https://%{HTTP_HOST}%{REQUEST_URI}
Alias "/.well-known" "/var/www/html/.well-known"

ssl.conf

A- config:

<IfModule mod_ssl.c>
  SSLRandomSeed startup builtin
  SSLRandomSeed startup file:/dev/urandom 512
  SSLRandomSeed connect builtin
  SSLRandomSeed connect file:/dev/urandom 512
  AddType application/x-x509-ca-cert .crt
  AddType application/x-pkcs7-crl     .crl
  SSLPassPhraseDialog exec:/usr/share/apache2/ask-for-passphrase
  SSLSessionCache             shmcb:${APACHE_RUN_DIR}/ssl_scache(512000)
  SSLSessionCacheTimeout  300
  SSLCipherSuite HIGH:MEDIUM:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4
  SSLProtocol all -SSLv3 -SSLv2
</IfModule>

A+ config:

<IfModule mod_ssl.c>
  SSLRandomSeed startup builtin
  SSLRandomSeed startup file:/dev/urandom 512
  SSLRandomSeed connect builtin
  SSLRandomSeed connect file:/dev/urandom 512
  AddType application/x-x509-ca-cert .crt
  AddType application/x-pkcs7-crl     .crl
  SSLPassPhraseDialog exec:/usr/share/apache2/ask-for-passphrase
  SSLSessionCache             shmcb:${APACHE_RUN_DIR}/ssl_scache(512000)
  SSLSessionCacheTimeout  300
  SSLCipherSuite ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4
  SSLHonorCipherOrder on

  SSLProtocol all -SSLv2 -SSLv3
  SSLCompression off
  SSLUseStapling on
  SSLStaplingCache "shmcb:logs/stapling-cache(150000)"
</IfModule>

Entropy

Check available entropy:

sudo cat /proc/sys/kernel/random/entropy_avail