Adding security layers to your App on OpenShift — Part 5: Mutual TLS with Istio

Today, securing your apps is a « must have » but it’s difficult to introduce it without modifying code if you didn’t think about it at the very beginning. Luckily, the new cloud native patterns brought by containers and platforms like Kubernetes offers simple ways to address security concerns without touching code.

As we move along our journey to securing our fruits-catalog application we came through a lot of topics: isolation and segregation, external exposure, users authentication and authorization, secrets management, database dynamic credentials lifecycle and talk about containers cloud native patterns. But wait… cloud native patterns are not only about containers!

Cloud native patterns come from the inherent highly distributed nature of applications that may be hosted on hybrid infrastructures — different cloud providers or on-premise — and architectural style like Service Oriented Architecture and its new evolution step called Microservices. However, high distribution implies the well known Fallacies of Distributed Computing we usually deal with adding resiliency and security policies in our applications…

From the security point of view, cloud native adoption also mean that there’s a shift in the security model to apply: from perimeter security dissociating untrusted and trusted zones — this model is also called castle-and-moats — to zero-trust network. This shift is necessary because the obvious man-in-middle vulnerability is even more exacerbated by the facts that:

• More and more companies have data spread across hybrid infrastructure, making it difficult to have a single security control zone,
• Financial optimisation concerns imply consolidation, sharing and elasticity of the application resources, making it even more difficult to continuously adapt and monitor the security zones.

Luckily we have got some new tools to address that! A Service Mesh is a key component of container and distributed architectures as it is all about addressing this issues and implementing zero-trust network policies. At a high level, a service mesh ensures communication between application services. It provides features such as traffic routing, load balancing and also service discovery, encryption, authentication, and authorization. This post will use Istio Service Mesh and show how it can secure and strongly increase the level of hardening of our fruits-catalog app.

Part 5 — Securing cluster internal transports with MTLS using Istio service-mesh

This post is part of a bigger series about how to add security layers to an existing application. This other parts published so far are:

• Part 1 — Deployment and TLS Ingress
• Part 2 — Authentication and Authorization with Keycloak
• Part 3 — Secret Management with Vault
• Part 4 — Dynamic secrets with Vault
• Part 5 — Mutual TLS with Istio (this post)
• Part 6 — PKI as a Service with Vault and Cert Manager

What we want to achieve in this part?

In this part, we are going to apply Istio Service Mesh features to our pods fruits-catalog and mongodb. Whilst Istio provides a large set of features related to request routing, resiliency or observability, we are going to focus here on those related to security.

We have here a particular focus of enabling the Mutual TLS feature for the communication between our pods. By using this feature, we are going to achieve 2 major points:

• The service-to-service authentication: Istio provides each service with a strong identity representing its role to enable interoperability across clusters and clouds. Thus service-to-service communication is subjected to Service naming verification preventing a malicious program to impersonate a target service (for inspecting or stealing data sent to a database for example),
• The service-to-service encryption: the Istio infrastructure provides a PKI and the automation of keys and certificates generation, distribution, and rotation among the different pods. If authenticated, a service-to-service communication is then encrypted using the keys distributed to communication ends.

Istio operates on our pods using the Sidecar Container pattern, a pattern we have already met into Part 3 and Part 4 of this series. The istio-proxy container may be injected into each pod and act as a TCP proxy that will intercept all ingress and egress traffic in the pod. It will handle the custom certificates and take care of applying the different policies we have configured for our pods. This is what is illustrated in below schema:

In order to apply policy, we will have to switch other to a new concept brought by Istio and replace the previous mongodb Kubernetes Service by what’s called a VirtualService in Istio. This way and without making any changes in the code, we will bring Mutual TLS to our app.

Let’s see how to apply this easily ;-)

How to apply it?

If you have read the first parts of this series, you may already have cloned my GitHub repository containing all the resources and assets. If not, I urge you to do so ;-) From your clone on your laptop, switch over to the master branch.

First thing first, we will need to install and activate Istio on our OpenShift cluster. When using OpenShift Container Platform, Istio is called OpenShift Service Mesh and instructions on how to deploy it can be found here. Installation is quite straightforward as it is using an Operator but take care of updating your master configuration to allow automatic sidecar injection we are using.

When deploying an application into Istio running in an OpenShift environment, it is currently necessary to relax the security constraints placed on the application by its service account to ensure the application can function correctly. Let start granting anyuid and privileged Security Context Constraints to enable the sidecars to run correctly:

$ oc adm policy add-scc-to-user anyuid -z fruits-catalog-vault -n fruits-catalog
$ oc adm policy add-scc-to-user privileged -z fruits-catalog-vault -n fruits-catalog

Then, we are going to simply add the annotation sidecar.istio.io/inject to the existing Deployments. This is the fine grained opt-in mechanisms we add in OpenShift: Istio is not enabled by default nor to a specific project, we have to enable it for each Deployment. Here are the commands:

$ oc patch dc/mongodb --type=json -p '[{"op":"add", "path":"/spec/template/metadata/annotations/sidecar.istio.io~1inject ", "value":"true"}]' -n fruits-catalog
$ oc patch dc/fruits-catalog --type=json -p '[{"op":"add", "path":"/spec/template/metadata/annotations/sidecar.istio.io~1inject", "value":"true"}]' -n fruits-catalog

Patching the two deployments triggers a new rollout of deployment because it is considered as a configuration change. Once ready, the new pods should now embed the istio-proxy container. We have also to add a new version label to the mongodb deployment in order to later define some policy:

$ oc patch dc/mongodb --type=json -p '[{"op":"add", "path":"/spec/template/metadata/labels/version", "value":"v1"}]' -n fruits-catalog

Finally, we just have to create 2 new Istio custom resources (or CRD) called DestinationRule and VirtualService. Simply put: a VirtualService allows to define an abstraction other multiple workloads hosted on pods, a DestinationRule allows to define traffic or access policies to VirtualServices. We have to create both for the mongodb application service:

$ oc apply -f k8s/mongodb-istio-destinationrule.yml -n fruits-catalog
$ oc apply -f k8s/mongodb-istio-virtualservice.yml -n fruits-catalog

And that’s it! We have just enabled Mutual TLS for securing the communication between the two pods of our application ; simply by annotating / labelling our deployments and creating Istio configuration. Nice!

Check it works as expected

Like all the previous post, the first thing to do is obviously to open a browser using the URL of the fruits-catalog route and just check the application is still working. But here are some others check if you would like/need to deep dive.

A Service Mesh is generally made of two parts: the data plane where data flows (it is made of the collection of istio-proxy in our apps) and the control plane where we define/apply/monitor policies. The Istio control plane comes with Kiali that is a console for observing, metering and controlling what’s going on in Istio Service Mesh. So we can retrieve Kiali’s route to access the console:

$ oc get route/kiali -n istio-system                                                                     NAME      HOST/PORT                                     PATH      SERVICES   PORT      TERMINATION   WILDCARD
kiali     kiali-istio-system.apps.144.76.24.92.nip.io             kiali      20001     reencrypt     None

Grab the URL from previous command result and open a browser tab. You should authenticate yourself before accessing the console. Credentials depend on your installation but the default are simply admin / admin — we should enhance secret management for this credentials too ;-)

Kiali’s console allows us to discover our application by selecting only the elements coming from the fruits-catalog namespace. Tweaking the graph options allows us seeing realtime network traffic going through the Service Mesh. You can access a whole lot of informations like response times, throughput, error rates, distributed traces and Istio custom resources configurations (more on this in a minute).

Then, we can have a look at the fruits-catalog pod deployment and check we’ve got everything in place. Here is a screenshot of the OpenShift web console that shows that:

• We’ve got an Init Container completed successfully. We can also have access to the command issued by this vault-init container,
• We’ve got a first container called vault-lease-refresher and still have access to the command issued by this later one,
• We’ve got the second and main container that is our application and that is using the application.properties configuration written by the Init Container,
• We finally got a third container that is the istio-proxy applying the traffic routing and security policies.
• And we actually got also an istio-init Init Container that fetches the proxy configuration at startup and ensures everything’s here to run the proxy.

Remember we talked about DestinationRule lately? Well you can access the definition of this rule for mongodb service using the CLI or through Kiali console. You can simply check here that the ISTIO_MUTUAL TLS policy is enabled when it comes to reach the mongodb / v1 destination:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: mongodb
spec:
  host: mongodb
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
  subsets:
  - name: v1
    labels:
      version: v1

Finnaly, you can also inspect the configuration using the istioctl CLI tool. By executing the following command we will get the real status of the communication to mongod.fruits-catalog.svc.cluster.local when data flow from the pods attached to the fruits-catalog deployment config:

$ istioctl authn tls-check $(oc get pod -l deploymentconfig=fruits-catalog -o jsonpath={.items..metadata.name}) mongodb.fruits-catalog.svc.cluster.local
HOST:PORT                                          STATUS     SERVER        CLIENT     AUTHN POLICY     DESTINATION RULE
mongodb.fruits-catalog.svc.cluster.local:27017     OK         HTTP/mTLS     mTLS       default/         mongodb/fruits-catalog

Conclusion

We have seen in this fifth part how powerful a Service Mesh could be for bringing security features other existing applications. Istio provides a service mesh implementation that is now embedded and fully supported into OpenShift Container Platform version 4.1. We have seen that using its Mutual TLS feature is just a matter of adding an annotation to our deployments to opt-in entering the mesh and declaring a destination rule!

Through this series we have added many layers of security to our existing fruits-catalog app that was developed with no security concerns in mind. Now, let me sum things up:

• First, we have isolated our components from the outer-world by containerizing them and deploying them on OpenShift. We have also secured their exposure using TLS Route for incoming calls,
• Then — in the second part — we have added an IAM layers by adding authentication and authorization on both the UI and API endpoints, by deploying and configuring Keycloak on OpenShift,
• In third and fourth parts, we tackle the problem of Secrets Management implementing a robust, dynamic and ephemeral credentials generation process using Hashicorp Vault. This process prevents from credentials leakage,
• Finally, we demonstrate how the concept of Service Mesh can be used to implement security concerns like authenticated and encrypted communications between services thus allowing to implement a real zero-trust network that is able to span other hybrid infrastructures.

And because one little picture says more than a long speech, here’s below a diagram representing this all-together architecture we achieve, completing our application with new deployment without touching the codebase!

I hope you enjoy going through this series with me and learned or discover a few techniques for enhancing security in your apps. Feedback is more than welcome through comments or claps!