Observations and thoughts after building 3 kubernetes platforms in Financial Services Industry — Part 2
In the second part of this blog series, share my observations regarding the use of workload identities and how to manage secrets based on my experience across 3 Kubernetes platforms for our clients.
Workload Identity
Kubernetes uses Service Accounts to represent workloads. These accounts are only visible within the scope of a cluster and not federated / integrated with external systems.
Workloads running on Kubernetes often have to interact with external cloud services such as Bucket/Blob storage, SQL Instances, Key stores. To allow workloads to be authenticated and authorized with the target service, an identity (service principal, service account) can be attached to your workload pod; thus bridging the kubernetes accounts to the provider’s identity back-end.
GKE excelled in this space with Workload identity as a first-class citizen. AKS had an open-source add-on named AAD Pod Identity up until January of this year. This was a very challenging component to manage. Thankfully, Microsoft has now replaced it AAD Workload Identity (still in preview).
Key Lessons:
- Use workload identities whenever a workload needs to interact with a cloud service. It avoid account keys or other shared secret that needs to be rotated.
- Workload Identity is core to GKE and becoming prime time on AKS (I’ve heard good things from coworkers using workload identity on AKS)
Secret Management / Externalization
Secret management is always a hot topic when dealing with regulated organizations. They often get hung up on “secrets are encoded in Base64” and don’t see beyond that point and immediately want to externalize the secrets….only to synchronize them to Kubernetes to be “in base64 in etcd”.
Explaining to clients about the reasoning behind base64 encoding of kubernetes secrets and encryption of that data at rest is always a fun thing to explain and dispel.
To address the “base64 debacle”, GKE provides Application-Layer encryption of data stored in etcd (BYOK or Google managed keys) as a core feature. AKS finally added KMS etcd encryption (appears to be in the last 2 months).
Now let’s talk about Secret Externalization! This area has seen tremendous growth in the past 3 years. Back then, there were no multi-tenant friendly solutions which raised eyebrows with security teams. This led organization to build custom solutions (not recommended). Fast forward to today, there are plenty of options available such as External Secrets, SOPS, HashiCorp Vault Agent Injector (VAI). My personal pick is External Secrets. It covers all common secret backends and supports multi-tenancy out of the box (each tenant configures their own secret store). Warning, there is an older project named Kubernetes External Secrets; stay away. I also found HashiCorp VAI to be awful to use as an end user (despite the nicety of run-time injection instead of synchronization).
Key Lessons:
- Be prepared to explain base64 encoding and decouple that from actual encryption at rest
- Secret externalization is now prevalent. Solutions either synchronizes the secret into kubernetes or inject them into workloads when instantiated.
- ExternalSecrets is the absolute winner in this space.
- Don’t build custom! there are many companies/projects much smarter than us that have solved this.
