Migrating from HashiCorp Vault to Google Cloud Secret Manager

Secrets are critical to your application development and deployment life cycle. If they are not available when you need them, or they get leaked, your entire business can come to a grinding halt. Having experienced server downtimes at inopportune moments, organisations are now rapidly migrating to cloud-managed services for secret management solutions from their self-hosted variants. In this post, I'll discuss the key considerations, benefits and limitations of migrating from a self-hosted HashiCorp Vault instance to the fully managed Google Cloud Secret Manager service.

What is HashiCorp Vault?

HashiCorp Vault is an extensible, open-source solution for securely storing, managing, and distributing secrets like API keys, passwords, tokens, certificates etc. Vault integrates with trusted identity platforms and brokers access to secrets, data, and systems. Vault can be used to generate dynamic secrets, automatically rotate database passwords, create X.509 certificates on demand, and manage the key lifecycle across KMS providers. While HashiCorp has recently started offering an AWS-based Managed Vault, the vast majority of Vault customers still use a self-hosted version.

What is Google Cloud Secret Manager?

Google Cloud Secret Manager (GSM for short) is a fully managed service for securely storing and managing secrets. Secrets are versioned and automatically encrypted using Google-managed encryption keys by default. Fine-grained access controls can be applied, allowing administrators to control which users and applications have access to specific secrets. GSM provides detailed logging of all secret access and management operations, allowing for easy auditing and compliance reporting. Finally, it comes with a pay-as-you-go pricing model, you only pay for the active secret versions and the number of operations requested.

Migrating from Self-Hosted Vault to Secret Manager

Managed services can typically handle a large number of objects (here, secrets) and concurrent users/applications, scaling effortlessly to an increase in demand. They are highly available and come with built-in disaster recovery options, reducing the risk of data loss or downtime. Managed services are significantly more secure than a self-hosted instance too, and generally offer a plethora of regulatory compliance certifications out of the gate. Google Cloud Secret Manager (GSM) is no different - it offers a significant upgrade in reliability and availability to a self-managed Vault instance, but it does come with a few limitations.

Let's discuss some of the key differences and considerations for a migration below. The recommendations are largely from the Google Cloud documentation, but I've summarised them here for better readability and relevance.

What types of secrets does GSM support?

GSM allows you to store, manage, and access static secrets as binary blobs or text strings. It can be used to store configuration information such as passwords, API keys, TLS certificates or arbitrary key-value data. GSM offers labels for sorting, filtering and grouping resources, as well as annotations for storing custom metadata about a secret.

How does GSM protect unauthorised access to secrets?

Access to GSM operations and secrets is protected by IAM - you can choose a predefined role with minimal permissions or create a custom role if necessary. Follow the principle of least privilege when granting permissions to secrets. Segment applications and environments (staging/production) into separate projects, and use secret-level IAM bindings or conditional IAM access to limit access to a subset of secrets. Use IAM Recommender to identify over-privileged IAM bindings.

In addition to IAM controls, you can limit access to the GSM with network-based controls by setting up a VPC Service Controls perimeter for your organization, reducing the risk of data exfiltration by malicious insiders or compromise code.

What are the considerations for secret creation in GSM?

In GSM, secrets can be created using the Google Cloud console, gcloud CLI, REST API and gRPC API. GSM client libraries provide high-level language support for authenticating to GSM programmatically using application default credentials. Client libraries are available for C#(.NET), Go, Java, Node.js, PHP, Python and Ruby. Where possible, you should use the GSM API instead of passing secrets to your application through the filesystem or environment variables. You can find the GSM best practices guide here, while all GSM code samples are available here.

Does GSM support secret rotation / update / expiration?

Periodically rotating secrets helps to limit impact in the case of a leaked secret. A secret in GSM can have multiple versions - to rotate a secret, add a new secret version (referenced by the latest alias). GSM supports rotation schedules on secrets, and sends messages to Pub/Sub topics based on the configured rotation frequency and time. GSM automatically retries failed delivery attempts for up to seven days, after which the scheduled rotation is aborted. See this guide for recommendations on the different rotation approaches you can take. If the external system validating the secret provides an Admin API, you can set up a rotation job that runs periodically. See these steps for a sample approach with Cloud Run as the environment.

By default, GSM secrets exist until a user deletes them, but you can attach an expiration time if desired. The expiration time can be added, updated or removed from a secret at any time. You can also create alerts based on logs warning of secrets that are expiring soon. When a secret expires, it is irreversibly deleted from GSM, so a time-based conditional IAM access approach is generally preferred to expiration for production secrets. See this guide for safely using expiring secrets.

Does GSM support dynamic or short-lived secrets?

Vault supports the generation of dynamic, on-demand secrets, which are automatically assigned with a lease and destroyed when the lease expires. While GSM does not offer an equivalent capability, Google Cloud does support the ability to create short-lived service account credentials. By impersonating a service account, you can create short-lived OAuth 2.0 access tokens, OIDC ID tokens, self-signed JWTs, and self-signed binary blobs. These credentials can be used to authenticate calls to Google Cloud APIs, other Google APIs, and non-Google APIs.

How should GKE workloads access GSM secrets?

For Google Kubernetes Engine (GKE) workloads, the recommended way to access GSM secrets is to use the client libraries authenticated with workload identity. The Secrets Store GSI driver is a Kubernetes project developed and maintained by the #sig-auth Special Interest Group and is not officially supported by Google.

Where are the secrets stored? Can I choose the location?

GSM secrets have global names and globally replicated metadata, but the location of the secret payload data can be configured using the replication policy. Unless workloads have specific location requirements, you should choose the automatic replication policy during secret creation. If necessary though, you can restrict the locations where a secret can be replicated. However, note that this cannot be changed once defined.

Does GSM offer an inventory of secrets across projects?

GSM offers labels for sorting, filtering and grouping resources, as well as annotations for storing custom metadata about a secret. See example filters here. GSM is also integrated with Cloud Asset Inventory, Google Cloud's managed metadata inventory system. With this integration, you can identify and audit secrets across your organization, folder or project, and discover any configurations that aren't conformant to your organization's requirements.

Does GSM integrate with IDEs or Google Cloud services?

As mentioned earlier, one of the biggest pros of GSM is the native integration with several Google Cloud services like Cloud Build, Cloud Functions, Cloud Run, Compute Engine, Kubernetes Engine are more. In fact, if you are developing an application in an IDE with Cloud Code installation, GSM comes integrated into the extension. You can create, view, update or use secrets without leaving the IDE.

Should I use Cloud KMS (or CMEK) to encrypt the secrets?

By default, secrets stored in GSM are encrypted with Google default encryption, where the secret payloads are encrypted by keys managed by Google before they are written to persistent storage. Unless there are specific regulatory (or internal policy) requirements around key lifecycle management, this approach should be sufficient from a security perspective. For organisations that need greater control, GSM lets you configure a Cloud KMS key to encrypt secrets at rest. See this article to understand how a customer-managed encryption key (CMEK) works in GSM. Some considerations:

• GSM uses the primary CMEK key version, and cannot be configured with a specific key version.
• You may choose to store secrets and keys in separate projects if you need stronger segregation of duties.
• The CMEK key must be in the same Google Cloud location as the secret version it encrypts.
• For secrets that use the automatic replication policy, the CMEK key must be located in the global Cloud KMS multi-region.
• The user/service identity using a CMEK key must have the Cloud KMS Encryptor/Decryptor role for that key.
• If you disable or destroy the CMEK key, the secrets encrypted with it cannot be decrypted.
• Removing CMEK configuration from a secret only applies to new secret versions; existing versions are not affected.

Obviously, this is not an exhaustive list, but it does cover the top questions that arise when organisations consider such a migration. If I've missed anything, hit me up on Twitter and let me know!