Sveltos: Empower RBAC Management in Multitenant Environments

Published in

ITNEXT

8 min readMay 8, 2024

Introduction

In a previous post, we demonstrated how to create different virtual clusters with vCluster in a Civo environment for multitenancy. Today, we will take the next step and demonstrate how to utilise Sveltos to allow platform administrators to provide full isolation and enable tenant sharing between different environments.

The main goals of the demonstration are:

Allow platform administrators to programmatically grant permissions across various managed clusters to tenant administrators from the management cluster
Allow tenant administrators to manage Kubernetes applications from the management cluster, ensuring each one of them can deploy resources based on the defined permissions

To achieve the above, Sveltos introduced a new Customer Resource Definition (CRD) called RoleRequest. The custom resource includes the below points.

ClusterSelector: Defines which cluster will be matched based on the cluster label selector
serviceAccountName: The name of the respective service account
serviceAccountNamespace: The namespace on the management cluster where to service account will be located
RoleRefs: References one or more ConfigMaps and/or Secrets.

Note: Only the platform administrator has the permission to create, update or delete RoleRequest instances.

Diagram

Lab Setup

+-----------------------+-------------+-------------------------------------+
|    Cluster Name       |   Version   |             Comments                |
+-----------------------+-------------+-------------------------------------+
|    mgmt01             | v1.28.7+k3s1| Civo 3 Node - Medium Standard       |
|    demo01             | v1.28.7+k3s1| Civo 3 Node - Medium Standard       |
|    vcluster-prod      | v1.29.0+k3s1| Defined in the `prod` namespace     |
|    vcluster-dev       | v1.29.0+k3s1| Defined in the `dev` namespace      |
|    sveltosctl         | v0.28.0     | Download link listed below          |
+-----------------------+-------------+-------------------------------------+

Prerequisites

Helm version ≥ v3.10.0
kubectl available: Use the guide found here
sveltosctl available: Download the binary from the Release page found here

Step 1: Install Sveltos on Managament Cluster

For this demonstration, we will install Sveltos in Mode 2: Centralised Agent Mode. Effectively, we do not want any agents running on the managed clusters but rather keep all Sveltos-related resources in the management cluster with the name mgmt01. Sveltos installation details can be found here.

$ kubectl apply -f https://raw.githubusercontent.com/projectsveltos/sveltos/main/manifest/agents_in_mgmt_cluster_manifest.yaml

$ kubectl apply -f https://raw.githubusercontent.com/projectsveltos/sveltos/main/manifest/default-classifier.yaml

Validation

$ kubectl get pods -n projectsveltos
NAME                                                 READY   STATUS      RESTARTS   AGE
register-mgmt-cluster-job-q46ql                      0/1     Completed   0          5m31s
shard-controller-5f57b9dc49-mnsxq                    1/1     Running     0          5m31s
addon-controller-5cd6985f8b-n4hgd                    1/1     Running     0          5m33s
access-manager-69dfc4f67b-tjdqz                      1/1     Running     0          5m33s
sc-manager-7cd4c4f567-sx7hf                          1/1     Running     0          5m33s
hc-manager-76c5b965d8-kwsj2                          1/1     Running     0          5m33s
event-manager-78d5777b77-k29qn                       1/1     Running     0          5m32s
classifier-manager-867fd59d94-9r24c                  1/1     Running     0          5m32s
sveltos-agent-2yngbxj30qjivrp2mu2k-b6fcd5b8d-g4tt5   1/1     Running     0          5m19s

$ kubectl get sveltosclusters -A
NAMESPACE   NAME   READY   VERSION
mgmt        mgmt   true    v1.28.7+k3s1

Note: We already have the management cluster listed under the sveltosclusters Kubernetes resource. Keep in mind that all the Sveltos-related resources are created in the projectsveltos namespace.

Step 2: Register vCluster with Sveltos

As discussed in the previous post, we have prepared two vClusters, vcluster-dev and vcluster-prod. The task is to register the virtual clusters with Sveltos while assigning the correct labels.

$ sveltosctl register cluster --namespace=projectsveltos --cluster=vcluster-prod --kubeconfig=~/demo/vcluster/kubeconfig/vcluster-prod.yaml --labels=env=prod

$ sveltosctl register cluster --namespace=projectsveltos --cluster=vcluster-dev --kubeconfig=~/demo/vcluster/kubeconfig/vcluster-dev.yaml --labels=env=dev

Validation

$ kubectl get sveltoscluster -A --show-labels
NAMESPACE        NAME            READY   VERSION        LABELS
mgmt             mgmt            true    v1.28.7+k3s1   sveltos-agent=present
projectsveltos   vcluster-dev    true    v1.29.0+k3s1   env=dev,sveltos-agent=present
projectsveltos   vcluster-prod   true    v1.29.0+k3s1   env=prod,sveltos-agent=present

From the above output, we can validate that both virtual clusters created in the projectsveltos namespace with the labels “env=dev” and “env=prod” respectfully.

Step 3: Create Deployment Configuration

For the demonstration, the platform administrator responsible for creating the clusters and the virtual clusters will deploy Kyverno to the vcluster-prod environment by the use of the Sveltos ClusterProfile.

Platform Administrator — Kyverno vcluster-prod

---
apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
  name: deploy-kyverno-prod
  labels:
spec:
  clusterSelector: env=prod
  syncMode: Continuous
  helmCharts:
  - repositoryURL:    https://kyverno.github.io/kyverno/
    repositoryName:   kyverno
    chartName:        kyverno/kyverno
    chartVersion:     v3.1.4
    releaseName:      kyverno-latest
    releaseNamespace: kyverno
    helmChartAction:  Install

RoleRequest and Configmap vcluster-prod

For the vcluster-prod, a ServiceAccount with the name eng will be able to deploy Kyverno policies to the cluster alongside accessing specific resources in the default namespace. Also, the eng service account can perform changes to cluster with the label set to env=prod.

# The ConfigMap contains a Role which gives read access to the default namespace
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: shared-service-access
  namespace: default
data:
  role.yaml: |
    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      name: service-read-role
      namespace: default
    rules:
    - apiGroups: [""]
      resources: ["services"]
      verbs: ["get,list"]
# The ConfigMap contains a Role which gives full access to namespace kyverno
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: eng-full-access
  namespace: default
data:
  clusterrole.yaml: |
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: clusterrole-eng
    rules:
    - apiGroups: ["kyverno.io"]
      resources: ["clusterpolicies"]
      verbs: ["get", "list", "watch", "patch"]
  role.yaml: |
    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      name: edit-role
      namespace: kyverno
    rules:
    - apiGroups: ["*"]
      resources: ["*"]
      verbs: ["*"]
# The RoleRequest gives the service account 'eng' access to the namespace 'default' and 'kyverno'
# in all clusters matching the label selector env=prod
---
apiVersion: lib.projectsveltos.io/v1alpha1
kind: RoleRequest
metadata:
  name: eng-access
spec:
  clusterSelector: env=prod
  serviceAccountName: "eng"
  serviceAccountNamespace: "default"
  roleRefs:
  - name: shared-service-access
    namespace: default
    kind: ConfigMap
  - name: eng-full-access
    namespace: default
    kind: ConfigMap

RoleRequest and Configmap vcluster-dev

For the vcluster-dev, a ServiceAccount with the name test will be able to deploy Kubernetes applications only to the dev namespace. Also, the test service account can perform changes only to the clusters with the label set to env=dev.

# The ConfigMap contains a Role which gives full access to the namespace dev
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: staging-full-access
  namespace: default
data:
  role.yaml: |
    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      name: edit-role
      namespace: dev
    rules:
    - apiGroups: ["*"]
      resources: ["*"]
      verbs: ["*"]
# The RoleRequest gives service account 'test' access to the namespace 'dev'
# in all clusters matching the label selector env=dev
---
apiVersion: lib.projectsveltos.io/v1alpha1
kind: RoleRequest
metadata:
  name: staging-access
spec:
  clusterSelector: env=dev
  serviceAccountName: "test"
  serviceAccountNamespace: "default"
  roleRefs:
  - name: staging-full-access
    namespace: default
    kind: ConfigMap

Note: Ensure the ServiceAccounts are present in the management cluster mgmt01. When deploying resources from a ClusterProfile, Sveltos impersonates the tenant serviceAccount. The RBAC validation is then done by Kubernetes in the managed cluster.

$ kubectl get sa
NAME                                    SECRETS   AGE
default                                 0         29h
civo-marketplace-traefik2-nodeport-mg   0         29h
eng                                     0         25m
test                                    0         24m

$ kubectl apply -f ~/demo/vcluster/clusterprofile/config_dev.yaml 
configmap/staging-full-access created
rolerequest.lib.projectsveltos.io/staging-access created

$ kubectl apply -f ~/demo/vcluster/clusterprofile/config_prod.yaml 
configmap/shared-service-access created
configmap/eng-full-access created
rolerequest.lib.projectsveltos.io/eng-access created

Validation

$ sveltosctl show admin-rbac
+---------------------------------------------+--------------+-----------+------------+-----------------+-----------------+----------+
|                   CLUSTER                   |    ADMIN     | NAMESPACE | API GROUPS |    RESOURCES    | RESOURCE NAMES  |  VERBS   |
+---------------------------------------------+--------------+-----------+------------+-----------------+-----------------+----------+
| SveltosCluster:projectsveltos/vcluster-dev  | default/test | dev       | *          | *               | *               | *        |
| SveltosCluster:projectsveltos/vcluster-prod | default/eng  | default   |            | services        | services        | get,list |
| SveltosCluster:projectsveltos/vcluster-prod | default/eng  | *         | kyverno.io | clusterpolicies | clusterpolicies | *        |
| SveltosCluster:projectsveltos/vcluster-prod | default/eng  | kyverno   | *          | *               | *               | *        |
+---------------------------------------------+--------------+-----------+------------+-----------------+-----------------+----------+

Step 4: Deploy Kyverno Policies vcluster-prod

We want to test if the RoleRequests applied down the cluster do what they are supposed to do. We will use the eng service account and install different Kyverno policies stored and synchronised via Flux to the respective cluster. The action is performed from the management cluster and with the use of the Sveltos ClusterProfile.

$ cat clusterprofile-prod.yaml
---
apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
  name: deploy-kyverno-policies-prod
  labels:
    projectsveltos.io/serviceaccount-name: eng
    projectsveltos.io/serviceaccount-namespace: default
spec:
  clusterSelector: env=prod
  syncMode: Continuous
  validateHealths:
  - name: deployment-health
    featureID: Helm
    group: "apps"
    version: "v1"
    kind: "Deployment"
    namespace: kyverno
    script: |
      function evaluate()
        local hs = {healthy = false, message = "Available replicas not match requested replicas"}
        if obj.status and obj.status.availableReplicas ~= nil and obj.status.availableReplicas == obj.spec.replicas then
          hs.healthy = true
        end
        return hs
      end
  policyRefs:
  - kind: GitRepository
    name: flux-system
    namespace: flux-system
    path: kyverno-policies
    deploymentType: Remote

Expected result: The eng service account to install Kyverno policies to the kyverno namespace alongside access to the default namespace.

$ kubectl apply -f clusterprofile/clusterprofile_prod.yaml 
clusterprofile.config.projectsveltos.io/deploy-kyverno-policies-prod created

$ sveltosctl show addons
+------------------------------+--------------------------+-----------+-------------------+---------+-------------------------------+---------------------------------------------+
|           CLUSTER            |      RESOURCE TYPE       | NAMESPACE |       NAME        | VERSION |             TIME              |                  PROFILES                   |
+------------------------------+--------------------------+-----------+-------------------+---------+-------------------------------+---------------------------------------------+
| projectsveltos/vcluster-prod | helm chart               | kyverno   | kyverno-latest    | 3.1.4   | 2024-05-01 15:38:54 +0000 UTC | ClusterProfile/deploy-kyverno-prod          |
| projectsveltos/vcluster-prod | kyverno.io:ClusterPolicy |           | require-ro-rootfs | N/A     | 2024-05-01 16:23:39 +0000 UTC | ClusterProfile/deploy-kyverno-policies-prod |
+------------------------------+--------------------------+-----------+-------------------+---------+-------------------------------+---------------------------------------------+

$ kubectl get clustersummary deploy-kyverno-policies-prod-sveltos-vcluster-prod -n projectsveltos -o yaml
status:
  dependencies: no dependencies
  featureSummaries:
  - deployedGroupVersionKind:
    - ClusterPolicy.v1.kyverno.io
    featureID: Resources
    hash: riDS1DlksBGuKMyiCc03AysQHhWgjAtm3+amnwFRpbQ=
    lastAppliedTime: "2024-05-01T16:23:49Z"
    status: Provisioned

Step 5: Deploy Nginx to vcluster-dev

Now, let’s move to the vcluster-dev cluster. We can test if the RoleRequests applied down the cluster do what they are supposed to do. The eng service account will install a simple Nginx application to the dev namespace.

$ cat clusterprofile-dev.yaml
---
apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
  name: deploy-nginx-dev
  labels:
    projectsveltos.io/serviceaccount-name: dev
    projectsveltos.io/serviceaccount-namespace: default
spec:
  clusterSelector: env=dev
  syncMode: Continuous
  policyRefs:
  - kind: GitRepository
    name: flux-system
    namespace: flux-system
    path: app
    deploymentType: Remote

Expected result: The dev service account to install the Nginx application to the cluster with the label selector set to env=dev.

$ kubectl apply -f clusterprofile/clusterprofile_dev.yaml 
clusterprofile.config.projectsveltos.io/deploy-nginx-dev created

$ sveltosctl show addons
+------------------------------+-----------------+-----------+------------------+---------+-------------------------------+---------------------------------------+
|           CLUSTER            |  RESOURCE TYPE  | NAMESPACE |       NAME       | VERSION |             TIME              |               PROFILES                |
+------------------------------+-----------------+-----------+------------------+---------+-------------------------------+---------------------------------------+
| projectsveltos/vcluster-prod | helm chart      | kyverno   | kyverno-latest   | 3.1.4   | 2024-05-01 15:38:54 +0000 UTC | ClusterProfile/deploy-kyverno-prod    |
| projectsveltos/vcluster-dev  | :Service        | dev       | nginx-service    | N/A     | 2024-05-01 16:16:19 +0000 UTC | ClusterProfile/deploy-nginx-dev       |
| projectsveltos/vcluster-dev  | apps:Deployment | dev       | nginx-deployment | N/A     | 2024-05-01 16:16:19 +0000 UTC | ClusterProfile/deploy-nginx-dev       |
+------------------------------+-----------------+-----------+------------------+---------+-------------------------------+---------------------------------------+

$ kubectl get clustersummary deploy-nginx-dev-sveltos-vcluster-dev -n projectsveltos -o yaml
status:
  dependencies: no dependencies
  featureSummaries:
  - deployedGroupVersionKind:
    - Deployment.v1.apps
    - Service.v1.
    featureID: Resources
    hash: riDS1DlksBGuKMyiCc03AysQHhWgjAtm3+amnwFRpbQ=
    lastAppliedTime: "2024-05-01T16:16:20Z"
    status: Provisioned

Conclusion

The platform administrators have an easy and reliable way to deploy RBAC with confidence to a fleet of clusters, either normal or virtual. Sveltos combines straightforwardly the concept of the RoleRequest with the ClusterProfile to allow platform administrator to do their job easier and creatively.

Resources

Sveltos: https://projectsveltos.github.io/sveltos/
Sveltos Multitenancy: https://projectsveltos.github.io/sveltos/features/multi-tenancy-sharing-cluster/
vCluster Official Website: https://www.vcluster.com/
Flux: https://fluxcd.io/
Civo Cloud: https://www.civo.com/

Contact

We are here to help! Whether you have questions, issues or need assistance, our Slack channel is the perfect place for you. Click here to join us.

👏 Support this project

Every contribution counts! If you enjoyed this article, check out the Projectsveltos GitHub repo. You can star 🌟 the project if you find it helpful.

The GitHub repo is a great resource for getting started with the project. It contains the code, documentation, and many more examples.

Thanks for reading!

Sveltos: Empower RBAC Management in Multitenant Environments

Introduction

Diagram

Lab Setup

Prerequisites

Step 1: Install Sveltos on Managament Cluster

Validation

Step 2: Register vCluster with Sveltos

Validation

Step 3: Create Deployment Configuration

Platform Administrator — Kyverno vcluster-prod

RoleRequest and Configmap vcluster-prod

RoleRequest and Configmap vcluster-dev

Validation

Step 4: Deploy Kyverno Policies vcluster-prod

Step 5: Deploy Nginx to vcluster-dev

Conclusion

Resources

Contact

👏 Support this project

Written by Eleni Grosdouli