Deploying Cassandra in Kubernetes with Casskop

Published in

ITNEXT

7 min readAug 4, 2019

CassKop is a Kubernetes operator for Cassandra. It can create, configure, and manage Cassandra clusters running in Kubernetes. CassKop has a number of interesting features. One of those feature is data center- and rack-aware deployments, which is the focus of this post.

CassKop introduces a new Custom Resource Definition, CassandraCluster, which allows you to specify a number things like:

The CPU and memory resources
Node anti-affinity
Resource limits
Storage class
Topology (in terms of data center and rack configuration)

This post explores how to configure some different Cassandra cluster topologies.

Install CassKop

CassKop is available is as Helm chart. Helm is the easiest way to deploy the operator. Setting up Helm is outside the scope of this post; so, I will simply refer you to the user guide for details on its installation. Be sure to also review setting up RBAC for Tiller as well.

The latest Helm chart for the operator can be found in the CassKop repository on GitHub. The first thing to do is add the repository:

$ helm repo add casskop https://Orange-OpenSource.github.io/cassandra-k8s-operator/helm

Then deploy CassKop:

$ helm install --name casskop casskop/cassandra-operator
NAME:   casskop
LAST DEPLOYED: Tue Jul 30 21:13:51 2019
NAMESPACE: default
STATUS: DEPLOYEDRESOURCES:
==> v1/Deployment
NAME                        READY  UP-TO-DATE  AVAILABLE  AGE
casskop-cassandra-operator  0/1    1           0          1s==> v1/Pod(related)
NAME                                         READY  STATUS             RESTARTS  AGE
casskop-cassandra-operator-79f77bb4c8-b4mf7  0/1    ContainerCreating  0         1s==> v1/RoleBinding
NAME                AGE
cassandra-operator  1s==> v1/ServiceAccount
NAME                SECRETS  AGE
cassandra-operator  1        1s==> v1beta1/Role
NAME                AGE
cassandra-operator  1sNOTES:
Congratulations. You have just deployed CassKop the Cassandra Operator.
Check its status by running:
kubectl --namespace default get pods -l "release=casskop"Visit https://github.com/Orange-OpenSource/cassandra-k8s-operator for instructions on hot to create & configure Cassandra clusters using the operator.

Before proceeding, make sure that the operator has deployed successfully:

$ kubectl get pods -l name=cassandra-operator
NAME                                          READY   STATUS    RESTARTS   AGE
casskop-cassandra-operator-79f77bb4c8-b4mf7   1/1     Running   0          108s

Deploy Simple Cluster

We start off with a minimal CassandraClusterthat does not define any data centers or racks.

# simple-cluster.yamlapiVersion: "db.orange.com/v1alpha1"
kind: "CassandraCluster"
metadata:
  name: simple-cluster
  labels:
    cluster: simple-cluster
spec:
  nodesPerRacks: 3
  resources:
    requests:
      cpu: 400m
      memory: 1Gi
    limits:
      cpu: 400m
      memory: 1Gi

Because neither data centers nor racks are defined, nodesPerRacksin effect specifies the cluster size.

Create the cluster with:

$ kubectl apply -f simple-cluster.yaml

Eventually we should see three pods that make up the Cassandra cluster. Let’s query for the pods:

$ kubectl get pods  -l cassandracluster=simple-cluster
NAME                         READY   STATUS    RESTARTS   AGE
simple-cluster-dc1-rack1-0   1/1     Running   0          12h
simple-cluster-dc1-rack1-1   1/1     Running   0          12h
simple-cluster-dc1-rack1-2   1/1     Running   0          12h

Note: CassKop adds the cassandraclusterlabel.

CassKop does not directly create the pods. The pods are managed by a StatefulSet which CassKop creates.

$ kubectl get statefulset -l cassandracluster=simple-cluster
NAME                       DESIRED   CURRENT   AGE
simple-cluster-dc1-rack1    3         3        12h

StatefulSets require use of a headless service to be responsible for network identify of the pods. Let’s query for the headless service:

$ kubectl get svc -l cassandracluster=simple-cluster | awk {'print $1" " $2" "$3" "$4'} | column -t
NAME                          TYPE       CLUSTER-IP  EXTERNAL-IP
simple-cluster               ClusterIP  None        <none>
simple-cluster-exporter-jmx  ClusterIP  None        <none>

We have our headless service simple-cluster. The other service, simple-cluster-exporter-jmx, is for exposing metrics to Prometheus.

Now let’s check the Cassandra cluster status with nodetool:

$ kubectl exec simple-cluster-dc1-rack1-0 -- nodetool status
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address    Load       Tokens       Owns (effective)  Rack
UN  10.0.2.18  69.94 KiB  256          69.0%             rack1
UN  10.0.5.6   87.38 KiB  256          65.0%             rack1
UN  10.0.3.7   104.86 KiB  256         65.9%             rack1

Note: The Host ID column has been removed from the output to improve readability.

All three Cassandra nodes are in the Up/Normal state. The Cassandra cluster is fully operational at this point. With a minimal spec, we are able to get a cluster up and running.

The Snitch

I want to talk briefly about the snitch which is configured in cassandra.yamlwith the endpoint_snitch property. The Cassandra out of box default is SimpleSnitch. CassKop configures Cassandra nodes to use GossipingPropertyFileSnitch.

Multiple Racks

Next we will look at a slightly more involved example that involves multiple racks:

# racks-cluster.yamlapiVersion: "db.orange.com/v1alpha1"
kind: "CassandraCluster"
metadata:
  name: racks-cluster
spec:
  nodesPerRacks: 2
  resources:
    requests:
      cpu: 400m
      memory: 1Gi
    limits:
      cpu: 400m
      memory: 1Gi
  topology:
    dc:
      - name: dc1
        rack:
          - name: rack1
          - name: rack2
          - name: rack3

Here we see the topology property for the first time. It declares a single data center and three racks. topology accepts a list of data centers. Each data center accepts a list of racks. This spec should produce a Cassandra cluster with 6 nodes spread across three racks.

Let’s query for the pods:

$ kubectl get pods -l cassandracluster=racks-cluster
NAME                        READY   STATUS    RESTARTS   AGE
racks-cluster-dc1-rack1-0   1/1     Running   0          18m
racks-cluster-dc1-rack1-1   1/1     Running   0          17m
racks-cluster-dc1-rack2-0   1/1     Running   0          15m
racks-cluster-dc1-rack2-1   1/1     Running   0          14m
racks-cluster-dc1-rack3-0   1/1     Running   0          13m
racks-cluster-dc1-rack3-1   1/1     Running   0          11m

We have six pods as expected. I previously mentioned that CassKop creates a StatefulSet which manages the pods. Let’s query for the StatefulSet:

$ kubectl get statefulset -l cassandracluster=racks-cluster
NAME                      DESIRED   CURRENT   AGE
racks-cluster-dc1-rack1   2         2         21m
racks-cluster-dc1-rack2   2         2         18m
racks-cluster-dc1-rack3   2         2         15m

This time we have three StatefulSets. CassKop creates a StatefulSet per rack.

Let’s query for the headless service:

$ kubectl get svc -l cassandracluster=racks-cluster | awk {'print $1" " $2" "$3" "$4'} | column -t
NAME                          TYPE       CLUSTER-IP  EXTERNAL-IP
racks-cluster               ClusterIP  None        <none>
racks-cluster-exporter-jmx  ClusterIP  None        <none>

It is possible to use the same headless service for multiple StatefuleSets. That is exactly what CassKop does.

Now let’s check the Cassandra cluster status with nodetool:

$ kubectl exec racks-cluster-dc1-rack1-0 -- nodetool status
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address    Load       Tokens       Owns (effective)  Rack
UN  10.0.2.19  87.34 KiB  256          30.5%             rack2
UN  10.0.4.5   92.34 KiB  256          32.8%             rack1
UN  10.0.0.6   81.63 KiB  256          31.9%             rack3
UN  10.0.5.7   69.94 KiB  256          36.0%             rack1
UN  10.0.3.8   87.38 KiB  256          33.9%             rack2
UN  10.0.1.8   69.92 KiB  256          35.0%             rack3

Note: The Host ID column has been removed from the output to improve readability.

All six Cassandra nodes are in the Up/Normal state and spread across three racks as expected.

Multiple Data Centers

Lastly, we will look at a multi-DC deployment.

# multidc-cluster.yamlapiVersion: "db.orange.com/v1alpha1"
kind: "CassandraCluster"
metadata:
  name: multidc-cluster
spec:
  nodesPerRacks: 2
  resources:
    requests:
      cpu: 400m
      memory: 1Gi
    limits:
      cpu: 400m
      memory: 1Gi
  topology:
    dc:
      - name: dc1
        rack:
          - name: rack1
          - name: rack2
          - name: rack3
      - name: dc2
        nodesPerRacks: 3
        rack:
          - name: rack1
          - name: rack2

Notice that in addition to declaring data centers dc1 and dc2 we also have set nodesPerRacks for dc2. CassKop enforces the number of nodes per rack to be the same within a data center. The number of nodes per rack can vary across data centers. The number of racks per data center can also vary. This provides a lot flexibility for configuring the cluster topology.

This spec should produce a Cassandra cluster with twelve Cassandra nodes. Let’s query for the pods:

$ kubectl get pods -l cassandracluster=multidc-cluster | awk {'print $1" " $2" "$3'} | column -t
NAME                         READY  STATUS
multidc-cluster-dc1-rack1-0  1/1    Running
multidc-cluster-dc1-rack1-1  1/1    Running
multidc-cluster-dc1-rack2-0  1/1    Running
multidc-cluster-dc1-rack2-1  1/1    Running
multidc-cluster-dc1-rack3-0  1/1    Running
multidc-cluster-dc1-rack3-1  1/1    Running
multidc-cluster-dc2-rack1-0  1/1    Running
multidc-cluster-dc2-rack1-1  1/1    Running
multidc-cluster-dc2-rack1-2  1/1    Running
multidc-cluster-dc2-rack2-0  1/1    Running
multidc-cluster-dc2-rack2-1  1/1    Running
multidc-cluster-dc2-rack2-2  1/1    Running

We have 12 pods as expected. Let’s query for the StatefulSets:

$ kubectl get statefulset -l cassandracluster=multidc-cluster
NAME                        DESIRED   CURRENT   AGE
multidc-cluster-dc1-rack1   2         2         8h
multidc-cluster-dc1-rack2   2         2         8h
multidc-cluster-dc1-rack3   2         2         7h58m
multidc-cluster-dc2-rack1   3         3         7h55m
multidc-cluster-dc2-rack2   3         3         7h51m

As expected we have five StatefulSets, one per rack.

Now let’s query for the headless service:

$ kubectl get svc -l cassandracluster=multidc-cluster | awk {'print $1" " $2" "$3" "$4'} | column -t
NAME                          TYPE       CLUSTER-IP  EXTERNAL-IP
multidc-cluster               ClusterIP  None        <none>
multidc-cluster-exporter-jmx  ClusterIP  None        <none>

Hopefully it is clear now that CassKop only creates one set of headless services, regardless of the number of data centers and racks.

Lastly, let’s check the Cassandra cluster status with nodetool:

$ kubectl exec multidc-cluster-dc1-rack1-0 -- nodetool status
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address    Load       Tokens       Owns (effective)  Rack
UN  10.0.2.2   200.27 KiB  256          14.3%            rack2
UN  10.0.4.6   299.47 KiB  256          14.4%            rack2
UN  10.0.0.7   314.06 KiB  256          15.5%            rack3
UN  10.0.5.9   294.07 KiB  256          15.0%            rack1
UN  10.0.1.9   270.8 KiB   256          16.3%            rack3
UN  10.0.3.10  289.84 KiB  256          15.6%            rack1
Datacenter: dc2
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address    Load       Tokens       Owns (effective)  Rack
UN  10.0.2.22  327.84 KiB  256          15.9%            rack1
UN  10.0.4.7   233.92 KiB  256          14.9%            rack2
UN  10.0.0.8   254.63 KiB  256          14.8%            rack2
UN  10.0.5.10  243.91 KiB  256          15.7%            rack1
UN  10.0.1.10  254.97 KiB  256          15.5%            rack2
UN  10.0.3.11  309.45 KiB  256          15.9%            rack1

Note: The Host ID column has been removed from the output to improve readability.

We have twelve Cassandra nodes spread across two data centers. dc1 has six nodes spread across three racks. dc2 has six nodes spread across two racks. Everything checks out!

Conclusion

CassKop offers a lot of flexibility in configuring the topology of Cassandra clusters running in Kubernetes. Data centers are a central feature of Cassandra, and CassKop fully exploit the feature by making every cluster that it create data center-aware.

Deploying Cassandra in Kubernetes with Casskop

Install CassKop

Deploy Simple Cluster

The Snitch

Multiple Racks

Multiple Data Centers

Conclusion

Written by John Sanda