Build SSL certificate auto-renewal for Jenkins

So I try to build an auto-renewal flow to relief myself. The approach is pretty straightforward: use acme.sh to auto-renew the SSL cert periodically, and an in-house script update_cert_for_jenkins.py to do the jobs after SSL cert renewed (if any).

Situation

Many web services/apps have the function auto-renewing the SSL certificate (usually based on acme.sh). Unfortunately the local Jenkins server we have in our office for iOS building doesn’t support that. Currently we use a free SSL cert from ZeroSSL, which is a 90-days SSL. Every 90 days the admin need to

  1. Renew the cert from ZeroSSL website
  2. Download the cert fiels and
  3. Convert and apply it to the Jenkins service.

Which is very annoying.

Solution

So I try to build an auto-renewal flow to relief myself. The approach is pretty straightforward: use acme.sh to auto-renew the SSL cert periodically, and an in-house script update_cert_for_jenkins.py to do the jobs after SSL cert renewed (if any).

Both of acme.sh and check_cer_and_update.py are configured to run daily in cron (acme.sh supports cron itself).

Configure acme.sh

acme.sh is a popular tool and has tons of articles discussing about it. No need to introduce every needy-greedy details here. I will list a few pitfalls or notes:

  1. If you don’t want to (or cannot) deal with the domain validation step with the server directly like me, it’s necessary to configure for dnsapi and corresponding environments. e.g. I use AWS route53 to manage DNS record so I have to add env vars like AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. For enterprices use, also remember to create a limited policy and account for this certain usage, and use the credential of this account.
  2. On MacOS, it’s necessary to specify OpenSSL bin. Otherwise a system default LibrSSL will be used and can cause some wierd issues.

  3. The execution enviroment of crontab may not be the same as the shell you use. Add SEHLL=/bin/bash to guarantee the command in crontab can be run correctly. The whole crontab file looks like:

    SHELL=/bin/bash
43 0 * * * "/Users/xxx/.acme.sh"/acme.sh --cron --home "/Users/xxx/.acme.sh"
...
  4. The errors in crontab execution beyond a single task are difficult to be found. But usually they will be recorded in the email (crontab sends email report for each running). Even the email receiver doesn’t exist, there will be a log file containing the email content. The dir may vary. Check /var/mail or /var/log/mail.

The script checking cert update

Each time the SSL cert files are updated, they will be stored in a folder under the acme.sh working dir. The dir is named after the domain. It contains a bunch of cert files and key files (in different format).

➜  my.domain.net ll
total 88
-rw-r--r--  1 me  staff   4.3K Jul 14 00:44 ca.cer
-rw-r--r--  1 me  staff   6.5K Jul 14 00:44 fullchain.cer
-rw-r--r--  1 me  staff   2.2K Jul 14 00:44 my.domain.net.cer
-rw-r--r--  1 me  staff   577B Jul 14 00:44 my.domain.net.conf
-rw-r--r--  1 me  staff   1.0K Jul 14 00:43 my.domain.net.csr
-rw-r--r--  1 me  staff   187B Jul 14 00:43 my.domain.net.csr.conf
-rw-------  1 me  staff   1.6K Oct 17  2022 my.domain.net.key
-rw-------  1 me  staff   6.3K Jul 14 01:43 my.domain.net.pfx

I create a script update_cert_for_jenkins.py doing these jobs:

  1. Compare the MD5 checksum of the cert file under acme.sh working dir, if not changed then do nothing.
  2. Export pkcs format file from renewed cert files and private key, through acme.sh --toPkcs.
  3. Use keytool to import the key store for Jenkins.
  4. Restart Jenkins service through brew services.

The details are included in update_cert_for_jenkins.py.

Follow instruction of this repo to install. But simply speaking, you only need to 1) put the py script to a working dir, 2) duplicate config.py.example into config.py and config everything in it, then 3) add crontab line:

43 1 * * * "/Users/me/inhouse-auto-certs"/update_cert_for_jenkins.py

Some notes/pitfalls:

  • The starting options of Homebrew managed Jenkins can be found and updated in /usr/local/Cellar/jenkins/2.xx/homebrew.mxcl.jenkins.plist. If you found brew services restart jenkins not working.
  • Specify absolute path of the directories and the bins, to minimize the chance of problems.
Written on October 20, 2022