How to Implement Mutual TLS with Docker Containers

With the advent of containers and microservices, it’s likely that your services are now talking more than ever to eachother over protocols like HTTP. But when your services are crossing untrusted networks (for example, in your cloud), how can you be sure their communication is secure? One way is through mutual Transport Layer Security (mTLS) — which helps services mutually authenticate (how do we know the service is who they say they are?) and encrypt their communications. But how does mTLS work? Let’s take a deep dive.

Mutual authentication and encryption between two parties is certainly nothing new. It’s the foundation of protocols like SSH and IPSec (which powers most VPN technologies), and recently has seen adoption in service mesh projects like Istio and Linkerd.

For production use cases, service meshes are a great way to get mTLS out of the box, but before adopting a service mesh you might be curious how a simple implementation of mTLS between two docker containers might work.

Please refer to the following GitHub repository to follow along: https://github.com/blhagadorn/mutual-tls-docker

Basic Client and Server Set Up

Let’s set up a basic client and server using Go — navigate to the 01-client-server-basic directory in the GitHub repository to follow along. After we set up the basic client and server, we will add mTLS into the mix.

Here is the gist of the basic server (found here)

func helloHandler(w http.ResponseWriter, r *http.Request) {
  io.WriteString(w, "Hello, world without mutual TLS!\n")
}
func main() {
  http.HandleFunc("/hello", helloHandler)
  log.Fatal(http.ListenAndServe(":8080", nil))
}

Essentially we listen to the /hello route on port 8080 and return a string when called.

Here is the gist of the basic client (found here):

 r, err := http.Get("http://localhost:8080/hello")
 if err != nil {
   log.Fatal(err)
 }
 defer r.Body.Close()
 body, err := ioutil.ReadAll(r.Body)

Essentially, we make an HTTP GET request to http://localhost:8080/hello and then write the response out.

Let’s build and run what we have so far, all of this inside of the 01-client-server-basic/ directory.

$ docker build -t basic-server -f Dockerfile.server . && docker run -it --rm --network host basic-server

Leave the server up and running and open up a new window in the same directory and then run the client:

$ docker build -t basic-client -f Dockerfile.client . && docker run -it --rm --network host basic-client
> Hello, world without mutual TLS!

Success! We now have a client and server speaking to each other in separate Docker containers. Notice the use of the --network host which creates a shared network between the containers, so localhost is the same for both containers.

Optionally, we can use tcpdump to verify the sending of cleartext while running the client:

$ docker run -it --network host --rm dockersec/tcpdump tcpdump -i any port 8080 -c 100 -A
> Date: Sat, 03 Feb 2024 15:05:20 GMT
> Content-Length: 33
> Content-Type: text/plain; charset=utf-8
> Hello, world without mutual TLS!

We know that TLS isn’t used, simply because we can read the text (if it was encrypted, it wouldn’t be legible)

Adding Mutual TLS

To add mutual TLS, first we need to generate a private key and corresponding certificate for the connection to use. Navigate to the 02-client-server-mtls directory for the rest of these examples if you are following along with the GitHub repository.

openssl req -newkey rsa:2048 \
-nodes -x509 \
-days 3650 \
-keyout key.pem \
-out cert.pem \
-subj "/C=US/ST=Montana/L=Bozeman/O=Organization/OU=Unit/CN=localhost" \
-addext "subjectAltName = DNS:localhost"

Here we generate a private key (key.pem) and a certificate (cert.pem) that contains a corresponding public key with the CN (Common Name) and SAN (Subject Alternative Name) of localhost.

Note: CN has been deprecated and most modern TLS libraries require SAN to to be set instead, including Golang’s net/http. In this example, we set both, because some libraries still require CN to be set or use it as a fallback.

The certificate (public key) and private key do a few things here. First, the private/public key combination is used to encrypt communication for the establishment of the session. Second, the certificate information is used for authentication, and the domain name that the certificate is intended to secure is localhost (the SAN).

A note on best security practices: in this example both services are sharing the same private key. This is not the recommended trust relationship in a production environment, but for simplicity’s sake, both the client and server use the same private key. For a more thorough example of a trust relationship with intermediate certificates, check out the steps here: mTLS with Intermediate Certificates

Now let’s examine the client code (in client-mtls.go), the following function returns an HTTPS client with a given certificate and key:

func getHTTPSClientFromFile() *http.Client {
  caCert, err := ioutil.ReadFile("cert.pem")
  if err != nil {
    log.Fatal(err)
  }
  caCertPool := x509.NewCertPool()
  caCertPool.AppendCertsFromPEM(caCert)
  cert, err := tls.LoadX509KeyPair("cert.pem", "key.pem")
  if err != nil {
    log.Fatal(err)
  } 
  client := &http.Client{
  Transport: &http.Transport{
    TLSClientConfig: &tls.Config{
      RootCAs:      caCertPool,
      Certificates: []tls.Certificate{cert},
      },
    },
  }
 return client
}

A few things are happening here — first, RootCAs is being set to the certificate pool we created (which just has one certificate in it). This is the set of root certificate(s) that the client will be using to verify different certificate authorities. Since we aren’t generating intermediate certificates in our example, this doesn’t mean much, but in many transactions this defines the root of trust (and any certificate signed by the root would be valid). Second, we are passing in the certificate, cert.pem, which defines the certificate that our client will pass to the server when establishing a secure connection. As well, the certificate key pair contains the private key, key.pem, to be used for encrypting communications.

Now let’s take a look at the relevant server code:

func main() {
  http.HandleFunc("/hello", helloHandler)
  caCert, err := ioutil.ReadFile("cert.pem")
  if err != nil {
    log.Fatal(err)
  }
 
  caCertPool := x509.NewCertPool()
  caCertPool.AppendCertsFromPEM(caCert)
  tlsConfig := &tls.Config{
    ClientCAs:  caCertPool,
    ClientAuth: tls.RequireAndVerifyClientCert,
  }
  tlsConfig.BuildNameToCertificate()
  server := &http.Server{
    Addr:      ":8443",
    TLSConfig: tlsConfig,
  }
  log.Fatal(server.ListenAndServeTLS("cert.pem", "key.pem"))
}

The server configuration is quite similar to the client configuration (which makes sense, since this is mutual authentication). The root CAs are defined similarly, the TLS configuration is set, and then finally the server starts listening using the certificate and the certificate key pair. Similar to the client, the server passes its certificate to any interested parties that want to connect with it (so that a client can encrypt communications according to the certificate public key) as a part of the TLS handshake, and as well the key is used to encrypt messages and verify ownership of the public key passed in the certificate.

Now, let’s run our examples, inside of the 02-client-server-mtls directory.

First the server:

$ docker build -t mtls-server -f Dockerfile.server . && docker run -it --rm --network host mtls-server

Then the client:

$ docker build -t mtls-client -f Dockerfile.client . && docker run -it --rm --network host mtls-client
> Hello, world WITH mutual TLS!

Success again!

We can verify with tcpdump again that there is no plaintext, and that our communication between the containers is encrypted.

$ docker run -it --network host --rm dockersec/tcpdump tcpdump -i any port 8443 -c 100 -A
>..V.(.@.................................. .&............0.........
O.f........

Notice that the output is totally illegible and unable to be sniffed, meaning we are using encryption

A Quick Diagram

See the diagram below, which illustrates the mTLS interaction we just performed

Green highlights the encrypted mTLS communication

Wrapping Things Up

At this point we’ve successfully created two client-server interactions, one without mutual TLS and the other with mutual TLS. We added TLS by generating a key and certificate, with localhost as our SAN. After that we edited the client code to include TLS configuration for the root CA as well as specified the certificate and private key we wanted to encrypt communications with. Similarly with our server code, we specified the root CA and which certificate and key the server should listen with.

After this, I hope you have foundation for thinking about mTLS across microservices, particularly service mesh. In our example we generated the certificate and key once and fed them manually into our configuration, but service meshes can often take care of automatically rotating those certificates with small renewal timeframes and additionally they usually route all your traffic through a sidecar proxy — which then upgrades normal communication to mTLS and then decrypts it once it gets to its destination. Essentially the mTLS is invisible, which is the magic of a strong proxy configuration and control plane.

I hope you learned a little something from this article about securing your containerized workloads, and as always — follow me on Medium for more articles or check me out on Twitter.