Open source enthusiast; consultant at Red Hat in enterprise integration; trying to find things that are really cool and avoid hypes;

Jun 13

Running integration tests in Kubernetes

Linux containers changed the way that we run, build and manage applications. As more and more platforms become cloud-native, containers are playing a more important role in every enterprise’s infrastructure. Kubernetes is currently the most well-known solution for managing containers, whether they run in a private, a public or a hybrid cloud.

With a container application platform, we can dynamically create a whole environment to run a task that is discarded afterwards. In an earlier post, we covered how to run builds and unit tests in containers. Now, let’s have a look at how to run integration tests by starting multiple containers to provide a whole test environment.

Note: This post is a follow-up of Running Jenkins builds in containers, so it’s suggested to have a quick look at that one first to become familiar with the basic principles of the solution.

Let’s assume we have a back-end application that depends on other services like databases, message brokers or web services. During unit testing, we try to use embedded solutions or simply mock up these endpoints somehow to make sure no network connections are required. This requires changes in our code for the scope of the test.

The purpose of the integration test is to verify how the application behaves with other parts of the whole solution stack. Providing a service depends on more than just our code base. The overall solution is a mix of modules (e.g., databases with stored procedures, message brokers, distributed cache with server side scripts) that must be wired together the right way to provide the expected functionality. This can only be tested by actually running all these parts next to each other and not enabling “test mode” within our application.

It’s debatable whether “unit test” and “integration test” are the right terms in this case. For simplicity’s sake I’ll call tests that run “within one process” without any external dependencies “unit tests” and the ones running the app in “production mode”making network connections “integration tests”.

Maintaining a static environment for such tests can be troublesome and a waste of resources, this is where the ephemeral nature of dynamic containers comes in handy.

The code base for this post can be found at https://github.com/bszeti/kubernetes-integration-test

It contains an example Red Hat Fuse 7 application (/app-users) that takes messages from AQM, queries data from a MariaDB and calls a REST api. The repo also contains the integration test project (/integration-test) and different Jenkinsfiles explained in this post.

Versions used during putting together this example:
- Red Hat CDK v3.4
- OpenShift v3.9
- Kubernetes v1.9
- Jenkins images v3.9
- Jenkins kubernetes-plugin v1.7

Fresh start every time

We’d like to achieve the following goals with our integration test:

Start the production ready package of our app under test.
Start an instance of all the dependency systems required.
Run tests that interact only via the public service endpoints with the app.
Nothing is persisted between executions, so we don’t have to worry about restoring the initial state.
Resources are allocated only during test execution.

The solution is based on Jenkins and the jenkins-kubernetes-plugin. Jenkins can run tasks on different agent nodes while the plugin makes it possible to create these nodes dynamically on Kubernetes. An agent node is created only for the task execution and deleted afterwards.

We need to define the agent node pod template first. The jenkins-master image for OpenShift comes with predefined PodTemplates for maven and nodejs builds and admins can add such “static” pod templates to the plugin configuration.

Fortunately, it’s also possible to define the pod template for our agent node directly in our project if we use Jenkins pipeline. This is obviously a more flexible way, as the whole execution environment can be maintained in code by the development team. Let’s see an example:

This pipeline will create all the containers pulling the given Docker images running them within the same pod. This means that the containers will share the localhost interface, so the services can access each other’s ports (but we have to think about port binding collisions). This is how the running pod looks in OpenShift web console:

The images are set by their Docker url (OpenShift image streams are not supported here), so the cluster must access those registries. In this example above, we built the image of our app earlier within the same Kubernetes cluster and now pull it from the internal registry: 172.30.1.1 (docker-registry.default.svc). This image is actually our release package that may be deployed to dev, test or prod environment. It’s started with a k8sit application properties profile where the connection urls point to 127.0.0.1.

It’s important to think about memory usage for containers running java processes. Current versions of java (v1.8, v1.9) ignore the container memory limit by default and set a much higher heap size. The v3.9 jenkins-slave images support memory limits via environment variables much better than earlier versions. Setting JNLP_MAX_HEAP_UPPER_BOUND_MB=64 was enough for us to run maven tasks with 512MiB limit.

All containers within the pod have a shared empty dir volume mounted at /home/jenkins (default workingDir). This is used by the Jenkins agent to run pipeline step scripts within the container, and this is where we check out our integration test repository. This is also the current director where the steps are executed unless they are within a dir(‘relative_dir’) block. Here are the pipeline steps for the example above:

The pipeline steps are run on the jnlp container unless they are within a container(‘container_name’) block:

First, we check out the source of the integration project. In this case it’s in the integration-test sub directory within the repo.
We have the sql/setup.sh script to create tables and load test data in the database. It requires the mysql tool, so it must be run in the mariadb container.
Our application (app-users) calls a Rest API. We have no image to start this service, so we use MockServer to bring up the http endpoint. It’s configured by the mockserver/setup.sh.
The integration tests are written in Java with Junit and executed by Maven. It could be anything else — this is simply the stack we’re familiar with.

There are plenty of configuration parameters for podTemplate and containerTemplate following the Kubernetes resource api with a few differences. Environment variables, for example, can be defined at the container as well as at the pod level. Volumes can be added to the pod, but they are mounted on each container at the same mountPath:

podTemplate(...
  containers: [...], 
  volumes:[
      configMapVolume(mountPath: '/etc/myconfig', 
        configMapName: 'my-settings'),
      persistentVolumeClaim(mountPath: '/home/jenkins/myvolume', 
        claimName:'myclaim')
      ],
  envVars: [
     envVar(key: 'ENV_NAME', value: 'my-k8sit')
   ]
)

Sounds easy, but…

Running multiple containers in the same pod is a nice way to attach them, but there is an issue that we can run into if our containers have entry points with different user ids. Docker images used to run processes as root, but it’s not suggested in production environments due to security concerns, so many images switch to a non-root user. Unfortunately, different images may use different uid (USER in Dockerfile) that can cause file permission issues if they use the same volume.

In this case the source of conflict is the Jenkins workspace on the workingDir volume (/home/jenkins/workspace/). This is used for pipeline execution and saving step outputs within each container. If we have steps in a container(…) block and the uid in this image is different (non-root) than in the jnlp container, we’ll get the following error:

touch: cannot touch '/home/jenkins/workspace/k8sit-basic/integration-test@tmp/durable-aa8f5204/jenkins-log.txt': Permission denied

Let’s have a look at the USER in images used in our example:

jenkins-slave-maven: uid 1001, guid 0
mariadb: uid 27, guid 27
amq: uid 185, guid 0
mockserver: uid 0, guid 0 (requires anyuid)
fuse-java-openshift: uid 185, guid 0

The default umask in the jnlp container is 0022, so steps in containers with uid 185 and uid 27 will run into the permission issue. The workaround is to change the default umask in the jnlp container so the workspace is accessible by any uid:

See the whole Jenkinsfile that first builds the app and the Docker image before running the integration test: kubernetes-integration-test/Jenkinsfile

In these examples the integration test in run on the jnlp container because we picked Java and Maven for our test project and the jenkins-slave-maven image can execute that. This is of course not mandatory, we can use the jenkins-slave-base image as jnlp and have a a separate container to execute the test. See an example where we intentionally separate jnlp and use another container for maven: kubernetes-integration-test/Jenkinsfile-jnlp-base

Yaml template

The podTemplate and containerTemplate definitions support many configurations, but they lack a few parameters. For example:

Can’t assign environment variables from ConfigMap, only from Secret.
Can’t set readiness probe for the containers. Without them Kubernetes reports the pod running right after kicking off the containers. Jenkins will start executing the steps before the the processes are actually ready to accept requests. This can lead to failures due to racing conditions. These example pipelines typically work because checkout scm gives enough time for the containers to start. Of course a sleep helps, but defining readiness probes is the proper way.

As an ultimate solution a yaml parameter was added to the podTemplate() in kubernetes-plugin (v1.5+). It supports a complete Kubernetes Pod resource definition, so we can define any configuration for the pod:

Make sure to update the Kubernetes plugin in Jenkins (to v1.5+) otherwise the yaml parameter is silently ignored.

Yaml definition and other podTemplate parameters supposed to be merged in a way, but it’s less error-prone to only use one or the other. Defining the yaml inline in the pipeline may be difficult to read, see this example of loading it from a file: kubernetes-integration-test/Jenkinsfile-yaml

Declarative Pipeline syntax

All the example pipelines above used the Scripted Pipeline syntax which is practically a groovy script with pipeline steps. The Declarative Pipeline syntax is a new approach enforcing more structure on the script by providing less flexibility and allowing no “groovy hacks”. It results in cleaner code, but you may have to switch back to the scripted syntax in case of complex scenarios.

In declarative pipelines the kubernetes-plugin (v1.7+) supports only the yaml definition to define the pod:

It’s also possible to set a different agent for each stage as in:
kubernetes-integration-test/Jenkinsfile-declarative

Try it on Minishift

If you’d like to try the solution described above you’ll need access to a Kubernetes cluster. At Red Hat we use OpenShift, which is an enterprise ready version of k8s. There are several ways to have access to a full scale cluster:

OpenShift Container Platform on your own infrastructure
OpenShift Dedicated cluster hosted on a public cloud
OpenShift Online on-demand public environment

It’s also possible to run a small one-node cluster on your local machine, which is probably the easiest way to try things. Let’s see how to setup Red Hat CDK (or Minikube) to run our tests.

After downloading, prepare the Minishift environment:

Run setup:
minishift setup-cdk
Set the internal Docker registry as insecure:
minishift config set insecure-registry 172.30.0.0/16This is needed because the kubernetes-plugin is pulling the image directly from the internal registry which is not https.
Start the Minishift virtual machine (use your free Red Hat account):
minishift --username me@mymail.com --password ... --memory 4GB start
Note the console url or you can get it by:
minishift console --url
Add oc tool to the path:
eval $(minishift oc-env)
Login to OpenShift api (admin/admin):
oc login https://192.168.42.84:8443

Start a Jenkins master within the cluster using the template available:
oc new-app --template=jenkins-persistent -p MEMORY_LIMIT=1024Mi

Once Jenkins is up it should be available via a route created by the template, e.g.( https://jenkins-myproject.192.168.42.84.nip.io). Login is integrated with OpenShift (admin/admin).

Create a new Pipeline project that takes the Pipeline script from SCM pointing to a Git repository (e.g. https://github.com/bszeti/kubernetes-integration-test.git) having the Jenkinsfile to execute. Then simply Build Now.

The first run takes longer as images are being downloaded from the Docker registries. If everything goes well, we can see the test execution on the Jenkins build’s Console Output. The dynamically created pods can be seen on the OpenShift Console under My Project / Pods.

If something goes wrong, try to investigate by looking at:

Jenkins build output
Jenkins master pod log
Jenkins kubernetes-plugin configuration
Events of created pods (maven or integration-test)
Log of created pods

If you’d like to make the additional executions quicker, you can use a volume as local maven repository, so maven doesn’t have to download dependencies every time. Create a PersistentVolumeClaim:

# oc create -f - <<EOF
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: mavenlocalrepo
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
EOF

Add the volume to the podTemplate (and optionally the maven template in kubernetes-plugin). See kubernetes-integration-test/Jenkinsfile-mavenlocalrepo:

volumes: [ 
  persistentVolumeClaim( mountPath: '/home/jenkins/.m2/repository', 
    claimName: 'mavenlocalrepo') 
]

Note: Maven local repositories claimed to be “non-thread safe” and should not be used by multiple builds at the same time. We use a ReadWriteOnce claim here that will be mounted to one pod only at a time.

The jenkins-2-rhel7:v3.9 image has kubernetes-plugin v1.2 installed. To run the Jenkinsfile-declarative and Jenkinsfile-yaml examples you need to update the plugin in Jenkins to v1.7+.

To completely cleanup after stopping Minishift delete ~/.minishift directory.

Limitations

There are certain aspects of the solution described above that I also want to mention here. Each project is different so it’s important to understand the impact of these in your case:

Using the jenkins-kubernetes-plugin to create the test environment is independent from the integration test itself. The tests can be written using any language and executed with any test framework — which is a great power but also a great responsibility.
The whole test pod is created before the test execution and shut down afterwards. There is no solution provided here to manage the containers during test execution. It’s possible to split up your tests to different stages with different pod templates, but that adds a lot of complexity.
The containers are started before the first pipeline steps are executed. Files from the integration test project are not accessible at that point so we can’t run prepare scripts or provide configuration files for those processes.
All containers belong to the same pod so they must run on the same node. If we need many containers and the pod requires too much resource there may be no node available to run the pod.
The size and scale of the integration test environment should be kept low. Though it’s possible to start up multiple microservices and run end-to-end tests within one pod, the number of required containers can quickly increase. This environment is also not ideal to test high availability and scalability requirements.
The test pod is recreated for each execution, but obviously the state of the containers is still kept during its run. This means that the individual test cases are not independent from each other. It’s the test project’s responsibility to do some cleanup between them if needed.

Summary

Running integration tests in an environment created dynamically from code is relatively easy by using Jenkins pipeline and the kubernetes-plugin. We only need a Kubernetes cluster and some experience with containers. Fortunately, more and more platforms provide official Docker images on one of the public registries. Worst-case scenario we have to build some ourselves. The hustle of preparing the pipeline and integration tests pays back quickly, especially when we want to try different configurations or dependency version upgrades during the life-cycle of our application.

Happy Testing!

Thanks to David Johnston for the contribution.