Use Istio traffic mirroring for quicker debugging

Often when an error occurs, especially in production, one needs to debug the application to create a fix. Unfortunately the input that created the issue is gone. And the test data on file does not trigger the error (otherwise it would have been fixed before delivery).

Likewise if one is creating new code, one often wants to see what values a client can supply (and to be honest I have used more than once WireShark to see what is being sent).

Istio’s traffic mirroring feature can help, as it allows an application to receive real traffic, which is processed by the main version. The same request is copied and then sent to the Mirror service. The difference is though, that the reply from the mirror service is just dropped (by the Envoy proxy in the sidecar) and not returned to the caller. You do not need to modify the mirror-version — Istio does this all for you.

Diagram of mirroring setup with traffic flows

Setup with Istio rules

Let’s have a look at the rules for this setup by starting with the DestinationRule

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: a-server-dr
spec:
  host: a-service
  subsets:
    - name: prod
      labels:
        version: "v0.1"
    - name: development
      labels:
        version: "devel"

This DestinationRule defines two subsets. Prod is for the production traffic and the other, development for our debugging target. With that we can now use the following VirtualService to define the mirroring:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: a-mirror
spec:
  hosts:
    - a-service
  http:
    - route:
        - destination:
            host: a-service
            subset: prod
      mirror:
        host: a-service
        subset: development

Then in spec.http.route.destination we define the default target and in spec.http.mirror the target for mirroring.

Current display in Kiali where traffic for reviews is going to v1 and is mirrored to v2

While the display is not completely correct in Kiali, you can see in the above screenshot that the traffic for the review service is going to 100% to v1 and is at the same time mirrored to v2 (some call this mirroring feature “dark launch” — indicated here by an invisible connection; if you want to know about the issue have a look at https://issues.jboss.org/browse/KIALI-2218 and the linked discussions and issues on the Istio side. It is likely that Istio 1.2 will see changes in telemetry reporting to cater for this — see https://github.com/istio/istio/issues/11475).

Mirroring to external services

Often one does not have the luxury to run an online editor in the same Istio-cluster, but just sits in front of the laptop with an IDE open. For this case we have two options:

1. Deploy a local Envoy that is configured to talk to the Istio backplane and to forward traffic to the local process. This is very similar to the above scenario
2. Use a ServiceEntry that forwards the traffic out of Istio to the debugged service

Let’s have a look at the 2nd option:

Scenario with a ServiceEntry (blue).

We’ll start by defining the ServiceEntry:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: external-a-service
spec:
  hosts:
  - a-service.ext # this is an arbitrary string; only used in VS
  location: MESH_EXTERNAL
  ports:
  - number: 9080
    name: http-ext
    protocol: HTTP
  resolution: STATIC
  endpoints:
  - address: 1.2.3.1 # Change to IP of service under debug
    ports:
      http-ext: 9080

The spec.hosts entry is an arbitrary entry, that you just need later in the VirtualService. spec.location tells that the service is outside the mesh. There are two possibilities for resolving the target IP address. We pick ‘STATIC’ and put the IP into spec.entpoinds.address. Consult the Istio documentation for ServiceEntry to learn about more details. Now with this entry, we can look at the VirtualService:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: external-service-a
spec:
  hosts:
    - a-service
  http:
  - route:
    - destination:
        host: a-service
        subset: prod
      weight: 100
    mirror:
        host: a-service.ext

This looks like the above — the only real difference is that for the mirror destination we only provide the host, that we have set up in the ServiceEntry above.

While writing this article I also found a blog post from Christian Posta, that explains this with Istio version 0.5; the examples can not be used 1:1 as Istio routing rules have changed since.