Sveltos: Empower RBAC Management in Multitenant Environments
Introduction
In a previous post, we demonstrated how to create different virtual clusters with vCluster in a Civo environment for multitenancy. Today, we will take the next step and demonstrate how to utilise Sveltos to allow platform administrators to provide full isolation and enable tenant sharing between different environments.
The main goals of the demonstration are:
- Allow platform administrators to programmatically grant permissions across various managed clusters to tenant administrators from the management cluster
- Allow tenant administrators to manage Kubernetes applications from the management cluster, ensuring each one of them can deploy resources based on the defined permissions
To achieve the above, Sveltos introduced a new Customer Resource Definition (CRD) called RoleRequest
. The custom resource includes the below points.
- ClusterSelector: Defines which cluster will be matched based on the cluster label selector
- serviceAccountName: The name of the respective service account
- serviceAccountNamespace: The namespace on the management cluster where to service account will be located
- RoleRefs: References one or more
ConfigMaps
and/orSecrets
.
Note: Only the platform administrator has the permission to create, update or delete RoleRequest instances.
Diagram
Lab Setup
+-----------------------+-------------+-------------------------------------+
| Cluster Name | Version | Comments |
+-----------------------+-------------+-------------------------------------+
| mgmt01 | v1.28.7+k3s1| Civo 3 Node - Medium Standard |
| demo01 | v1.28.7+k3s1| Civo 3 Node - Medium Standard |
| vcluster-prod | v1.29.0+k3s1| Defined in the `prod` namespace |
| vcluster-dev | v1.29.0+k3s1| Defined in the `dev` namespace |
| sveltosctl | v0.28.0 | Download link listed below |
+-----------------------+-------------+-------------------------------------+
Prerequisites
- Helm version ≥ v3.10.0
- kubectl available: Use the guide found here
- sveltosctl available: Download the binary from the Release page found here
Step 1: Install Sveltos on Managament Cluster
For this demonstration, we will install Sveltos in Mode 2: Centralised Agent Mode. Effectively, we do not want any agents running on the managed clusters but rather keep all Sveltos-related resources in the management cluster with the name mgmt01. Sveltos installation details can be found here.
$ kubectl apply -f https://raw.githubusercontent.com/projectsveltos/sveltos/main/manifest/agents_in_mgmt_cluster_manifest.yaml
$ kubectl apply -f https://raw.githubusercontent.com/projectsveltos/sveltos/main/manifest/default-classifier.yaml
Validation
$ kubectl get pods -n projectsveltos
NAME READY STATUS RESTARTS AGE
register-mgmt-cluster-job-q46ql 0/1 Completed 0 5m31s
shard-controller-5f57b9dc49-mnsxq 1/1 Running 0 5m31s
addon-controller-5cd6985f8b-n4hgd 1/1 Running 0 5m33s
access-manager-69dfc4f67b-tjdqz 1/1 Running 0 5m33s
sc-manager-7cd4c4f567-sx7hf 1/1 Running 0 5m33s
hc-manager-76c5b965d8-kwsj2 1/1 Running 0 5m33s
event-manager-78d5777b77-k29qn 1/1 Running 0 5m32s
classifier-manager-867fd59d94-9r24c 1/1 Running 0 5m32s
sveltos-agent-2yngbxj30qjivrp2mu2k-b6fcd5b8d-g4tt5 1/1 Running 0 5m19s
$ kubectl get sveltosclusters -A
NAMESPACE NAME READY VERSION
mgmt mgmt true v1.28.7+k3s1
Note: We already have the management cluster listed under the sveltosclusters
Kubernetes resource. Keep in mind that all the Sveltos-related resources are created in the projectsveltos
namespace.
Step 2: Register vCluster with Sveltos
As discussed in the previous post, we have prepared two vClusters, vcluster-dev
and vcluster-prod
. The task is to register the virtual clusters with Sveltos while assigning the correct labels.
$ sveltosctl register cluster --namespace=projectsveltos --cluster=vcluster-prod --kubeconfig=~/demo/vcluster/kubeconfig/vcluster-prod.yaml --labels=env=prod
$ sveltosctl register cluster --namespace=projectsveltos --cluster=vcluster-dev --kubeconfig=~/demo/vcluster/kubeconfig/vcluster-dev.yaml --labels=env=dev
Validation
$ kubectl get sveltoscluster -A --show-labels
NAMESPACE NAME READY VERSION LABELS
mgmt mgmt true v1.28.7+k3s1 sveltos-agent=present
projectsveltos vcluster-dev true v1.29.0+k3s1 env=dev,sveltos-agent=present
projectsveltos vcluster-prod true v1.29.0+k3s1 env=prod,sveltos-agent=present
From the above output, we can validate that both virtual clusters created in the projectsveltos
namespace with the labels “env=dev” and “env=prod” respectfully.
Step 3: Create Deployment Configuration
For the demonstration, the platform administrator responsible for creating the clusters and the virtual clusters will deploy Kyverno to the vcluster-prod environment by the use of the Sveltos ClusterProfile
.
Platform Administrator — Kyverno vcluster-prod
---
apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
name: deploy-kyverno-prod
labels:
spec:
clusterSelector: env=prod
syncMode: Continuous
helmCharts:
- repositoryURL: https://kyverno.github.io/kyverno/
repositoryName: kyverno
chartName: kyverno/kyverno
chartVersion: v3.1.4
releaseName: kyverno-latest
releaseNamespace: kyverno
helmChartAction: Install
RoleRequest and Configmap vcluster-prod
For the vcluster-prod, a ServiceAccount with the name eng
will be able to deploy Kyverno policies to the cluster alongside accessing specific resources in the default
namespace. Also, the eng
service account can perform changes to cluster with the label set to env=prod
.
# The ConfigMap contains a Role which gives read access to the default namespace
---
apiVersion: v1
kind: ConfigMap
metadata:
name: shared-service-access
namespace: default
data:
role.yaml: |
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: service-read-role
namespace: default
rules:
- apiGroups: [""]
resources: ["services"]
verbs: ["get,list"]
# The ConfigMap contains a Role which gives full access to namespace kyverno
---
apiVersion: v1
kind: ConfigMap
metadata:
name: eng-full-access
namespace: default
data:
clusterrole.yaml: |
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: clusterrole-eng
rules:
- apiGroups: ["kyverno.io"]
resources: ["clusterpolicies"]
verbs: ["get", "list", "watch", "patch"]
role.yaml: |
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: edit-role
namespace: kyverno
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["*"]
# The RoleRequest gives the service account 'eng' access to the namespace 'default' and 'kyverno'
# in all clusters matching the label selector env=prod
---
apiVersion: lib.projectsveltos.io/v1alpha1
kind: RoleRequest
metadata:
name: eng-access
spec:
clusterSelector: env=prod
serviceAccountName: "eng"
serviceAccountNamespace: "default"
roleRefs:
- name: shared-service-access
namespace: default
kind: ConfigMap
- name: eng-full-access
namespace: default
kind: ConfigMap
RoleRequest and Configmap vcluster-dev
For the vcluster-dev, a ServiceAccount with the name test
will be able to deploy Kubernetes applications only to the dev
namespace. Also, the test
service account can perform changes only to the clusters with the label set to env=dev
.
# The ConfigMap contains a Role which gives full access to the namespace dev
---
apiVersion: v1
kind: ConfigMap
metadata:
name: staging-full-access
namespace: default
data:
role.yaml: |
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: edit-role
namespace: dev
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["*"]
# The RoleRequest gives service account 'test' access to the namespace 'dev'
# in all clusters matching the label selector env=dev
---
apiVersion: lib.projectsveltos.io/v1alpha1
kind: RoleRequest
metadata:
name: staging-access
spec:
clusterSelector: env=dev
serviceAccountName: "test"
serviceAccountNamespace: "default"
roleRefs:
- name: staging-full-access
namespace: default
kind: ConfigMap
Note: Ensure the ServiceAccounts are present in the management cluster mgmt01. When deploying resources from a ClusterProfile, Sveltos impersonates the tenant serviceAccount. The RBAC validation is then done by Kubernetes in the managed cluster.
$ kubectl get sa
NAME SECRETS AGE
default 0 29h
civo-marketplace-traefik2-nodeport-mg 0 29h
eng 0 25m
test 0 24m
$ kubectl apply -f ~/demo/vcluster/clusterprofile/config_dev.yaml
configmap/staging-full-access created
rolerequest.lib.projectsveltos.io/staging-access created
$ kubectl apply -f ~/demo/vcluster/clusterprofile/config_prod.yaml
configmap/shared-service-access created
configmap/eng-full-access created
rolerequest.lib.projectsveltos.io/eng-access created
Validation
$ sveltosctl show admin-rbac
+---------------------------------------------+--------------+-----------+------------+-----------------+-----------------+----------+
| CLUSTER | ADMIN | NAMESPACE | API GROUPS | RESOURCES | RESOURCE NAMES | VERBS |
+---------------------------------------------+--------------+-----------+------------+-----------------+-----------------+----------+
| SveltosCluster:projectsveltos/vcluster-dev | default/test | dev | * | * | * | * |
| SveltosCluster:projectsveltos/vcluster-prod | default/eng | default | | services | services | get,list |
| SveltosCluster:projectsveltos/vcluster-prod | default/eng | * | kyverno.io | clusterpolicies | clusterpolicies | * |
| SveltosCluster:projectsveltos/vcluster-prod | default/eng | kyverno | * | * | * | * |
+---------------------------------------------+--------------+-----------+------------+-----------------+-----------------+----------+
Step 4: Deploy Kyverno Policies vcluster-prod
We want to test if the RoleRequests
applied down the cluster do what they are supposed to do. We will use the eng
service account and install different Kyverno policies stored and synchronised via Flux to the respective cluster. The action is performed from the management cluster and with the use of the Sveltos ClusterProfile.
$ cat clusterprofile-prod.yaml
---
apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
name: deploy-kyverno-policies-prod
labels:
projectsveltos.io/serviceaccount-name: eng
projectsveltos.io/serviceaccount-namespace: default
spec:
clusterSelector: env=prod
syncMode: Continuous
validateHealths:
- name: deployment-health
featureID: Helm
group: "apps"
version: "v1"
kind: "Deployment"
namespace: kyverno
script: |
function evaluate()
local hs = {healthy = false, message = "Available replicas not match requested replicas"}
if obj.status and obj.status.availableReplicas ~= nil and obj.status.availableReplicas == obj.spec.replicas then
hs.healthy = true
end
return hs
end
policyRefs:
- kind: GitRepository
name: flux-system
namespace: flux-system
path: kyverno-policies
deploymentType: Remote
Expected result: The eng
service account to install Kyverno policies to the kyverno
namespace alongside access to the default
namespace.
$ kubectl apply -f clusterprofile/clusterprofile_prod.yaml
clusterprofile.config.projectsveltos.io/deploy-kyverno-policies-prod created
$ sveltosctl show addons
+------------------------------+--------------------------+-----------+-------------------+---------+-------------------------------+---------------------------------------------+
| CLUSTER | RESOURCE TYPE | NAMESPACE | NAME | VERSION | TIME | PROFILES |
+------------------------------+--------------------------+-----------+-------------------+---------+-------------------------------+---------------------------------------------+
| projectsveltos/vcluster-prod | helm chart | kyverno | kyverno-latest | 3.1.4 | 2024-05-01 15:38:54 +0000 UTC | ClusterProfile/deploy-kyverno-prod |
| projectsveltos/vcluster-prod | kyverno.io:ClusterPolicy | | require-ro-rootfs | N/A | 2024-05-01 16:23:39 +0000 UTC | ClusterProfile/deploy-kyverno-policies-prod |
+------------------------------+--------------------------+-----------+-------------------+---------+-------------------------------+---------------------------------------------+
$ kubectl get clustersummary deploy-kyverno-policies-prod-sveltos-vcluster-prod -n projectsveltos -o yaml
status:
dependencies: no dependencies
featureSummaries:
- deployedGroupVersionKind:
- ClusterPolicy.v1.kyverno.io
featureID: Resources
hash: riDS1DlksBGuKMyiCc03AysQHhWgjAtm3+amnwFRpbQ=
lastAppliedTime: "2024-05-01T16:23:49Z"
status: Provisioned
Step 5: Deploy Nginx to vcluster-dev
Now, let’s move to the vcluster-dev cluster. We can test if the RoleRequests
applied down the cluster do what they are supposed to do. The eng
service account will install a simple Nginx application to the dev
namespace.
$ cat clusterprofile-dev.yaml
---
apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
name: deploy-nginx-dev
labels:
projectsveltos.io/serviceaccount-name: dev
projectsveltos.io/serviceaccount-namespace: default
spec:
clusterSelector: env=dev
syncMode: Continuous
policyRefs:
- kind: GitRepository
name: flux-system
namespace: flux-system
path: app
deploymentType: Remote
Expected result: The dev
service account to install the Nginx application to the cluster with the label selector set to env=dev
.
$ kubectl apply -f clusterprofile/clusterprofile_dev.yaml
clusterprofile.config.projectsveltos.io/deploy-nginx-dev created
$ sveltosctl show addons
+------------------------------+-----------------+-----------+------------------+---------+-------------------------------+---------------------------------------+
| CLUSTER | RESOURCE TYPE | NAMESPACE | NAME | VERSION | TIME | PROFILES |
+------------------------------+-----------------+-----------+------------------+---------+-------------------------------+---------------------------------------+
| projectsveltos/vcluster-prod | helm chart | kyverno | kyverno-latest | 3.1.4 | 2024-05-01 15:38:54 +0000 UTC | ClusterProfile/deploy-kyverno-prod |
| projectsveltos/vcluster-dev | :Service | dev | nginx-service | N/A | 2024-05-01 16:16:19 +0000 UTC | ClusterProfile/deploy-nginx-dev |
| projectsveltos/vcluster-dev | apps:Deployment | dev | nginx-deployment | N/A | 2024-05-01 16:16:19 +0000 UTC | ClusterProfile/deploy-nginx-dev |
+------------------------------+-----------------+-----------+------------------+---------+-------------------------------+---------------------------------------+
$ kubectl get clustersummary deploy-nginx-dev-sveltos-vcluster-dev -n projectsveltos -o yaml
status:
dependencies: no dependencies
featureSummaries:
- deployedGroupVersionKind:
- Deployment.v1.apps
- Service.v1.
featureID: Resources
hash: riDS1DlksBGuKMyiCc03AysQHhWgjAtm3+amnwFRpbQ=
lastAppliedTime: "2024-05-01T16:16:20Z"
status: Provisioned
Conclusion
The platform administrators have an easy and reliable way to deploy RBAC with confidence to a fleet of clusters, either normal or virtual. Sveltos combines straightforwardly the concept of the RoleRequest
with the ClusterProfile
to allow platform administrator to do their job easier and creatively.
Resources
- Sveltos: https://projectsveltos.github.io/sveltos/
- Sveltos Multitenancy: https://projectsveltos.github.io/sveltos/features/multi-tenancy-sharing-cluster/
- vCluster Official Website: https://www.vcluster.com/
- Flux: https://fluxcd.io/
- Civo Cloud: https://www.civo.com/
Contact
We are here to help! Whether you have questions, issues or need assistance, our Slack channel is the perfect place for you. Click here to join us.
👏 Support this project
Every contribution counts! If you enjoyed this article, check out the Projectsveltos GitHub repo. You can star 🌟 the project if you find it helpful.
The GitHub repo is a great resource for getting started with the project. It contains the code, documentation, and many more examples.
Thanks for reading!