HashiCorp Vault Operator Manages Kubernetes Secrets
Securing secrets — keys and passwords and so on needed to gain access to your critical applications, data and infrastructure — has never been more important or urgent. Security breaches are costly, in terms of both money and reputation; hackers are getting more creative, and more brazen.
Exhibit A: The average cost of a data breach is $4.24 million, according to a study released in 2022 by IBM and the Ponemon Institute.
Exhibit B: A whopping 99% of issued credentials go unused for 60 days, reported a study released in May 2022 by Unit 42, the security research arm of Palo Alto Networks. Unused credentials, the researchers said, are often used by hackers to access critical applications, data or infrastructure.
Trouble is, the task of locking down and monitoring the usage of your secrets is, to put it mildly, complicated. Distributed, multicloud architecture means there’s no single castle to build a moat around. And Kubernetes is complex, requiring operators to integrate properly with many services and solutions.
Furthermore, workloads on Kubernetes constantly change. Dynamic secrets handle changing service identities while minimizing the blast radius when a credential is compromised.
Kubernetes Secrets can be configured to encrypt secrets at rest, but they’re stored unencrypted by default, in the API server’s underlying data store (etcd). So they’re not, well, all that secret, at least not without additional security in place. It’s relatively easy to read, modify or retrieve secrets from Kubernetes Secrets.
HashiCorp Vault has long been a trusted way to store, manage and control secrets across complex distributed systems. Vault Secrets Operator, an open source project released alongside Vault 1.13 and now available in beta, makes it easier to use Vault with Kubernetes Secrets, automating tasks that were previously manual.
Meeting a Community Need
The demand for a Vault operator supported by HashiCorp designed to work specifically with Kubernetes Secrets came directly from the community of Vault users, according to Rosemary Wang, a developer advocate at HashiCorp.
The message the company received from the Vault community, Wang told The New Stack, was for a desire to use Kubernetes native interfaces to retrieve and manage the life cycle of secrets, rather than continue using the secure alternatives people had grown accustomed to using.
Two of the most common approaches:
Injecting a sidecar. The Vault agent injector, in which a sidecar injects secrets into an application in Kubernetes. The sidecar offloads the retrieval and life cycle of secrets from Vault and puts it into a file within a pod that the application can retrieve.
However, Wang said, the problem comes when third-party services and tools are being loaded in the Kubernetes cluster that also uses the Kubernetes Secrets interface. “The injector makes it very difficult for you to use the native Kubernetes Secrets interface if that’s what your third-party tool or service is leveraging.”
Secrets Store CSI Driver. Some users instead leveraged the Kubernetes’ Secrets Store Container Storage Interface (CSI) Driver. “The Secrets Store CSI driver had two options,” Wang said. “You could mount a volume with a file that included the secret. So, similar to [the] agent [injector], but you don’t need to run it as a sidecar.”
The other option for the Secrets Store CSI drive, she said, is where “ you could synchronize as a Kubernetes Secret. So rather than use the file volume mounting approach, you could use the sync-a-Secret approach where we create Kubernetes secret.”
Ultimately, she added, “You had a lot of different ways you could do this. But there were pitfalls, and some folks didn’t want to adopt the Secret Store CSI Driver; they had existing environments, and they weren’t planning to add the CSI Driver components into it.”
The main reason organizations didn’t want to adopt the Secret Store CSI Driver, Wang said, was that their security requirements prohibit volume mounting from the host.
The Vault Secrets Operator, Wang said, represents a third approach, one that bypasses file volume mounting and exclusively synchronizes to Kubernetes Secrets. “That often solves the gap between third-party services or other tools that are standardized on the Kubernetes Secrets interface.”
Under the Hood
The new technology implements a controller that makes use of Kubernetes custom resource definitions (CRDs) to natively synchronize Vault secrets to Kubernetes Secrets, without the need to run a sidecar container or use the Secrets Store CSI Driver.
The message from Vault community members, Wang said, was that they wanted a controller to dynamically retrieve information from Vault and pass it into their Kubernetes clusters, and then be able to read the information from that secret, she added, “because that’s a common pattern in the Kubernetes community.”
The operator sets custom resource definitions and then Vault manages the secrets used by the services, with Kubernetes orchestrating them according to their CRDs. The Vault Secrets Operator reconciles secrets from the current state to desired state, as defined by the custom resources, using Kubernetes declarative patterns.
Being able to synchronize natively to Kubernetes Secrets is especially useful when secrets change, Wang said. “When a secret changes in Vault, there’s a lot you have to either write in your application, or the process that you’re running in Kubernetes, to reload the new secret,” she said.
The Vault Secrets Operator, Wang said, will “allow you to set a target deployment in Kubernetes, so if the secret changes in Vault, the operator will pick up the difference and then roll out the deployment for you. It will restart the application on your behalf. And you don’t have to worry about that, you don’t have to control that mechanism. And that is valid for static secrets as well as dynamic secrets.”
What’s Next for Vault Secrets Operator?
While in beta period, HashiCorp Vault Operator will remain focused on the Kubernetes authentication method, though its project roadmap calls for adding other authentication methods in the future.
Also in the future: Further testing to support different platforms and secrets engines, secret templating and improvements to secure Kubernetes Secrets.
Notably, Wang said, “there’s a lot of interest in doing everything through Kubernetes, basically configuring Vault and using Kubernetes as the interface to configure HashiCorp tools. There’s certainly some community demand for that.”
As the operator is an open source project, she encouraged users to check out the code and request features. But as the Vault Secrets Operator moves toward general availability, HashiCorp’s top priorities are “the stability of the operator, as well as the ability to monitor it.”
Making Kubernetes, the orchestrator of choice for containerized applications, more user-friendly — helping to create a “golden path” for developers who build and deploy at scale — is a hot conversation in the cloud native world. The new Vault Secrets Operator, she suggested, is a step toward that path.
At HashiCorp, Wang said, “there’s always a golden path conversation, mostly because a number of our tools are focused toward helping you scale your platform. A lot of our tools are focused on addressing the platform engineering teams that need to scale platforms for a developer audience internally or even to other customers.
“And so the result is that we’re always asking the question: Is this a pattern that a development team will feel comfortable adopting?”
Vault Secrets Operator fits that need, providing a native Kubernetes Secrets experience that watches the expectations and practices of development teams rolling on deployments on Kubernetes.
