Open in app

Sign up

Sign in

Write

Sign up

Sign in

Jenkins X — TLS enabled Previews

Steve Boardwell
ITNEXT
Published in
7 min readJun 25, 2019

--

Welcome back to the mini-series on Jenkins X. This section will be dedicated to setting up wildcard certificates and TLS for preview environments.

In my earlier post I discussed how to secure both:

  • the GKE cluster (using authorised network access and dedicated IP ranges for worker nodes, pods, and services)
  • the Jenkins X default services (adding ssl certificates and an IP whitelist for access)

That done, it was time to look at adding TLS to the dynamic preview environments Jenkins X provides.

Big thanks to Pablo Loschi at this point for his original Story on Using wildcard certificates with cert-manager in Kubernetes. It was a life-saver and credit where credits due, so thank you :-).

And, as always, a big shout out to Ilya Shaisultanov who accompanied me on this little adventure.

A little information up front

Before continuing, I’d like to you give a quick explanation to a couple of technical decisions we made.

  1. We decided to use a single domain to allow for a *.domain wildcard certificate. For this, we specified "{{.Service}}-{{.Namespace}}.{{.Domain}}" in the exposecontroller so that e.g.
    https://jenkins.jx.domain.xyz — > https://jenkins-jx.domain.xyz
  2. LetsEncrypt will only issue wildcard certificates if a DNS-01 challenge is used, therefore we needed to use one of the supported DNS-01 providers.
    Both Google CloudDNS and Cloudflare were tested in this context.
From the cert-manager website

TL;DR for the impatient

This post contains quite a lot of information covering various semi-connected topics.

However, for those with little time on their hands, here are the general steps we will be performing:

  • Ensuring the correct DNS records are prepared
  • Adding the kubernetes-replicator (needed later for the previews)
  • Setting up credentials enabling DNS-01 challenge
  • Creating wildcard certificates for our domain
  • Adding the required kubernetes-replicator annotations
  • Creating a Jenkins X preview with TLS from the newly created certificate

A quick note on keeping secrets in SCM

You have probably have your own solution to this little problem but I’d like to mention it all the same, if only in the interests of security.

sops from the folks at Mozilla is a great little tool for encrypting various file formats, including YAML. From the website:

sops is an editor of encrypted files that supports YAML, JSON, ENV, INI and BINARY formats and encrypts with AWS KMS, GCP KMS, Azure Key Vault and PGP.

We use it to encrypt our kubernetes secrets, thus allow us to keep them in git with using plaintext. Using the 01-clouddns-service-account.yaml from above as an example, the encrypted file looks like this:

$ cat cert-manager-utils/01-clouddns-service-account.yaml
apiVersion: v1
data:
 devops-dns-admin_token: ENC[AES256_GCM,data:<redacted>,type:str]
kind: Secret
metadata:
 creationTimestamp: null
 name: clouddns-service-account
 namespace: cert-manager
sops:
 kms:
 — arn: <redacted>
 created_at: ‘2019–05–23T15:56:06Z’
 enc: <redacted>
 gcp_kms: []
 azure_kv: []
 lastmodified: ‘2019–05–23T18:40:17Z’
 mac: ENC[AES256_GCM,data:<redacted>,type:str]
 pgp: []
 encrypted_suffix: _token
 version: 3.2.0

Unencrypted, the file looks like:

$ sops -d cert-manager-utils/01-clouddns-service-account.yaml
apiVersion: v1
data:
 devops-dns-admin_token: <redacted>
kind: Secret
metadata:
 creationTimestamp: null
 name: clouddns-service-account
 namespace: cert-manager

One added bonus is the ability to combine this with kubectl apply and process substitution to provide on the fly decryption:

kubectl apply -f <(sops -d cert-manager-utils/01-clouddns-service-account.yaml)

Although, in the end, I really don’t mind which method is chosen as long as people are using encryption :-).

Ensure correct DNS entries exist

This was covered in my earlier post during the installation of Jenkins X (search for “You will need to create DNS records”).

Just in case you are here with a pre-installed Jenkins X platform, please ensure that you have DNS records in place for your domain. Here an excerpt from my earlier post:

You will need to create DNS records (Type A)pointing to:

  • YOUR-DOMAIN > loadbalancer ip
  • *.YOUR-DOMAIN > loadbalancer ip

You can find the loadbalancer ip with:

kubectl get svc jxing-nginx-ingress-controller -n kube-system -o'jsonpath={ .status.loadBalancer.ingress[0].ip

Installing Kubernetes Replicator

Next we’ll install the kubernetes-replicator. Unfortunately, at the time of writing there does not seem to be an official helm chart. It is, however, very easy to install so here is a quick and dirty snippet to install v1.0.0.

Just copy and paste into a shell:

Now the replicator is installed, we can add the wildcard certificates.

Creating the wildcard certificates

First we needed DNS admin account credentials which, depending on your domain registrar, can be created as follows:

TIP: You are not forced to use multiple issuers, but having both specified allows you to use the same config files for multiple domains.

Once the credentials are created, we can create the necessary kubernetes resources. The resource files have been split into 3 groups:

  • the DNS admin secrets
  • the cluster issuers
  • the certificates themselves
$ tree cert-manager-utils
cert-manager-utils
├── 01-clouddns-service-account.yaml      # cert-manager namespace
├── 01-cloudflare-api-key.yaml            # cert-manager namespace
├── 02-clusterissuer-prod.yaml            # kube-system namespace
├── 02-clusterissuer-staging.yaml         # kube-system namespace
├── 03-cluster-certificate-prod.yaml      # kube-system namespace
└── 03-cluster-certificate-staging.yaml   # kube-system namespace

There are files for both “staging” and “prod”. These use the LetsEncrypt staging and prod environments respectively.

I highly recommend using the staging environment first to make sure the configuration is correct before moving over to the LetsEncrypt prod environment.

The DNS Secrets

The names below are just a suggestion by the way. You are, of course, free to choose your own names for the secret keys used below such as “devops-dns-admin_token”, etc.

cert-manager-utils/01-clouddns-service-account.yaml
—-apiVersion: v1
data:
 devops-dns-admin_token: REPLACE_WITH_BASE64_ENCODED_JSON
kind: Secret
metadata:
 creationTimestamp: null
 name: clouddns-service-account
 namespace: cert-manager
cert-manager-utils/01-cloudflare-api-key.yaml
—-apiVersion: v1
data:
 api-key.txt: REPLACE_WITH_BASE64_ENCODED_API_KEY
kind: Secret
metadata:
 name: cloudflare-api-key
 namespace: cert-manager

The LetsEncrypt Staging Files

These resources will create a wildcard certificate, issued by the LetsEncrypt staging environment.

cert-manager-utils/02-clusterissuer-staging.yaml
---apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging-dns
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: REPLACE_WITH_your.email@your.domain.com
    privateKeySecretRef:
      name: letsencrypt-staging
    dns01:
      providers:
        - name: clouddns
          clouddns:
            serviceAccountSecretRef:
              name: clouddns-service-account
              key: devops-dns-admin_token
            project: REPLACE_WITH_your-google-cloud-project
        - name: cloudflare
          cloudflare:
            apiKeySecretRef:
              key: api-key.txt
              name: cloudflare-api-key
            email: REPLACE_WITH_your.email@your.domain.com
cert-manager-utils/03-cluster-certificate-staging.yaml
---apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: jxing-nginx-ingress-controller-wildcard-staging
  namespace: kube-system
spec:
  secretName: jxing-nginx-ingress-controller-wildcard-staging-tls
  issuerRef:
    name: letsencrypt-staging-dns
    kind: ClusterIssuer
  commonName: REPLACE_WITH_YOUR_DOMAIN
  dnsNames:
    - REPLACE_WITH_YOUR_DOMAIN
    - "*.REPLACE_WITH_YOUR_DOMAIN"
  acme:
    config:
      - dns01:
          provider: cloudflare
        domains:
          - REPLACE_WITH_YOUR_DOMAIN
          - "*.REPLACE_WITH_YOUR_DOMAIN"
      - dns01:
          provider: clouddns
        domains:
          - REPLACE_WITH_YOUR_DOMAIN
          - "*.REPLACE_WITH_YOUR_DOMAIN"

The LetsEncrypt Prod Files

The only difference to the staging files here is the value of LetsEncrypt’s “acme.server” along with some resource names (prod vs staging to keep the two resource groups apart).

cert-manager-utils/02-clusterissuer-prod.yaml
---apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod-dns
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: REPLACE_WITH_your.email@your.domain.com
    privateKeySecretRef:
      name: letsencrypt-prod
    dns01:
      providers:
        - name: clouddns
          clouddns:
            serviceAccountSecretRef:
              name: clouddns-service-account
              key: devops-dns-admin_token
            project: REPLACE_WITH_your-google-cloud-project
        - name: cloudflare
          cloudflare:
            apiKeySecretRef:
              key: api-key.txt
              name: cloudflare-api-key
            email: REPLACE_WITH_your.email@your.domain.comcert-manager-utils/03-cluster-certificate-prod.yaml
---apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: jxing-nginx-ingress-controller-wildcard-prod
  namespace: kube-system
spec:
  secretName: jxing-nginx-ingress-controller-wildcard-prod-tls
  issuerRef:
    name: letsencrypt-prod-dns
    kind: ClusterIssuer
  commonName: REPLACE_WITH_YOUR_DOMAIN
  dnsNames:
    - REPLACE_WITH_YOUR_DOMAIN
    - "*.REPLACE_WITH_YOUR_DOMAIN"
  acme:
    config:
      - dns01:
          provider: cloudflare
        domains:
          - REPLACE_WITH_YOUR_DOMAIN
          - "*.REPLACE_WITH_YOUR_DOMAIN"
      - dns01:
          provider: clouddns
        domains:
          - REPLACE_WITH_YOUR_DOMAIN
          - "*.REPLACE_WITH_YOUR_DOMAIN"

Putting it all together

Using a little bash function we can put everything together with:

which can then be used á la:

add_certificates staging
or
add_certificates prod

After a few minutes, you should see something like this:

Using wildcard certs in preview environments

Now that we have configured all the necessary components, adding TLS to a preview environment is surprisingly simple.

Check exposecontroller version

Ensure the version in your preview requirements.yaml is 2.3.111 or higher.

See this example.

Add empty secret to replicate your wildcard certificate

Add an empty secret as described here. The kubernetes-replicator will detect this secret and copy the real secret into the preview namespace.

See this example.

Edit exposecontroller config to add TLS

Changes will be made to the following attributes:

tlsacme: true
urltemplate: "\"{{.Service}}-{{.Namespace}}.{{.Domain}}\""
tlsSecretName: "jxing-nginx-ingress-controller-wildcard-prod-tls"

See this example.

Success!

And that is it! Each preview will have TLS enabled and be using our wildcard certificate.

Congratulations!

Optional Bonus Setting

nginx has another nice little option to set a “default-ssl-certificate” for a given ingress controller.

This option, when set, adds a default ssl certificate as a fallback for any ingress requests, thus removing the need to set spec.tls.secretName in the ingress configuration.

Currently, you need to set the tlsSecretName in the exposecontroller config, meaning the above trick cannot be used. However, a huge thanks to Vincent Behar for adding the option in the first place.

If you do wish to add the default ssl certificate to the nginx controller, here is a little bash snippet to do so.

So, that’s it for part two of the mini-series on Jenkins X. I hope it could help somewhat.

In part three, I’ll be going though how we manage our Jenkins server in our Jenkins X platform, adding OAuth, updating plugins and configuration, as well as keeping job configurations in git. Stay tuned…

Ssl
Jenkins X
Tls
Preview

--

--

Steve Boardwell
ITNEXT
Follow

Written by Steve Boardwell

80 Followers

Looking at all things Devops and Automation

Follow

Help

Status

About

Careers

Blog

Privacy

Terms

Text to speech

Teams