How to Create a Kubernetes Custom Controller Using client-go
With Kubernetes custom controller, you can further develop your own custom business logic by watching events from Kubernetes API objects such as namespace, deployment or pod, or your own CRD (custom resource definitions) resource.
The next part of the article will provide a deep dive on the client-go module, following by a custom controller example.
client-go module
client-go is being used by Kubernetes as the offical API client library, providing access to Kubernetes` restful API interface served by the Kubernetes API server. Tools like kubectl or prometheus-operator use it intensively.
The library contains several important packages and utilities which can be used for accessing the API resources or facilitate a custom controller.
kubernetes package provides clientset and Kubernetes resource specific clientset methods for accessing standard Kubernetes APIs. Note you should not use it for accessing CRD resources. This package is auto generated.
discovery package provides ways to discover server-supported API groups, versions and resources.
dynamic package provides a dynamic client which can perform restful operations on arbitrary Kubernetes API resources [1]. Note that it’s discouraged to use it for access CRD resource due to it’s not type-safe [2].
transport package setups the secure TCP authorisation and connection. The default will be using HTTP2 protocol if not explicitly disabled. The underlying HTTP2 facility is provided by k8s.io/apimachinery/util/net. Due to some of the operations requires transfer of binary streams between the client and a container, such as attach, exec, portforward and logging, the transport package also establishes streaming channels. It employed SPDY and WebSocket protocols before HTTP2 became available.
plugin package provides authorisation plugins for cloud providers such as Openstack, GCP and Azure.
scale package provides a polymorphic scale client capable of fetching and updating Scale for any resource which implements the “scale” subresource, as long as that subresource operates on aversion of scale convertable to autoscaling.Scale. Note prior to Kubernetes v1.10, it doesn’t support scaling CRD[2].
util package provides convenient utilities such as work queue, buffer.
tool/cache package has many useful facilities such as SharedInformer, reflector, stores and indexers. It provides a client-side query and caching mechanism, reducing the number of request to the server and keep tracking of the events. I will be detailing workflow provided by utilities in this package because it does the plumbing when writing a custom controllers.
As shown in the above graph, there are two main parts of actions. One happens in SharedIndexInformer, the other one is in custom controller.
1. Reflector performs object (such as namespace, pod etc) ListAndWatch queries to the Kubernetes API server. Three event types Added, Updated and Deleted are being recorded along with the related objects. They are then passed onto DeltaFIFO. Note only events occurred within the past five minutes can be retrieved by default.
A given Kubernetes server will only preserve a historical list of changes for a limited time. Clusters using etcd3 preserve changes in the last 5 minutes by default. When the requested watch operations fail because the historical version of that resource is not available, clients must handle the case by recognizing the status code
410 Gone, clearing their local cache, performing a list operation, and starting the watch from theresourceVersionreturned by that new list operation. Most client libraries offer some form of standard tool for this logic. (In Go this is called aReflectorand is located in thek8s.io/client-go/cachepackage.)[3]
2. DeltaFIFO ingests the events and the objects correspondingly to the watch events, then translates them into Delta object. Those Delta objects get appended into a queue waiting for processing. For Deleted, it will check if it’s already existed in the thread safe store so it can avoid queuing up a delete action while something doesn’t exist.
3. Cache controller (Not to confused with custom controller) calls Pop() method to dequeue the DeltaFIFO queue. The Delta objects get passed onto SharedIndexInformer’s HandleDelta() method for further processing.
4. According to the Delta object’s action (event) type, the objects firstly get saved into thread safe store via indexer’s method in HandleDeltas method. It then sends those objects via sharedProcessor’s distribute() method in SharedIndexInformer to event handlers which have been registered by the custom controller through SharedInformer’s methods such as AddEventHandlerWithResyncPeriod().
5. The registered event handlers convert the object into key strings in a format of “namespace/name” or just “name” if no namespace through MetaNamespaceKeyFunc() for add and update events, DeletionHandlingMetaNamespaceKeyFunc() for delete event. The keys then get added into custom controller’s workqueue. The type of workqueues could be found in util/workqueue.
6. The custom controller pops keys from the workqueue for processing by calling custom handlers. The custom handlers will invoke indexer’s GetByKey() to retrieve the object from thread safe store. The custom handlers are where your business logic lives.
Example of a simple custom controller using workqueue
Below is an example of custom controller that watches pods in default namespace. The workqueue type is RateLimitQueue. The controller spawns out one worker in below example. You can change the number of workers when calling controller.Run().
Note that below example is using an IndexInformer which can only have one sets of the handlers to subscribe to the events. The recommendation is to use SharedIndexInformer instead. The difference is ShareIndexInformer provides methods like AddEventHandlerWithResyncPeriod() to allow you adding multiple set of handlers, consequently one event request can be distributed to different handlers at the same time, reducing the number of API requests.
If you want to hook up with your CRD, you could follow this instruction to generate clientset, informer etc for your CRD resources. You then can use the generated SharedInformer in your custom controller.
Reference
[2] Introducing client-go version 6
[4] Writting Kubernetes Custom Controller
[5] A deep dive into Kubernetes controller
Disclaimer: The views expressed here are solely those of the author in his private capacity and do not in any way represent the views of any organisation.
