DevOps
HashiCorp Vault for Production Secrets Management: Setup and Best Practices
March 202612 min read
DevOps
Before HashiCorp Vault, our secrets management at Commsult Indonesia was embarrassing: database passwords in .env files committed to private Git repos, API keys copied manually between developers, and SSL certificates stored in a shared Google Drive folder. This is more common than most teams admit. Vault changes the model fundamentally — no secrets at rest in files or Git, dynamic short-lived credentials that expire automatically, and a complete audit log of every secret access. The setup cost is a few hours; the operational improvement is permanent. This guide covers the deployment and configuration patterns I use.
Static secrets — database passwords, API keys, certificates — have a dangerous lifecycle: they're created once, shared via Slack or email, saved in .env files, and often never rotated because 'we'll do it later.' Every developer who ever had access retains the knowledge of that secret even after leaving the team. If a .env file leaks (via accidental commit, a compromised developer machine, or a cloud storage misconfiguration), the blast radius is the entire set of systems that secret granted access to. Vault introduces dynamic secrets: Vault generates a unique database credential for each application instance, with a configurable TTL (time-to-live) after which the credential expires and Vault automatically revokes it.
Vault has two operational modes. Dev mode (vault server -dev) runs entirely in memory with a pre-configured root token — useful for local development and testing but stores nothing persistently and has no security. Production mode requires: a durable storage backend (Consul, integrated Raft, or a cloud storage backend like GCS), TLS certificates for all communication, initialization with Shamir secret shares (5 key shares, 3 required to unseal is a common baseline), and an unseal mechanism. The initialization process happens once — Vault generates a root token and the Shamir key shares, and you distribute the key shares to different people (e.g., 5 senior engineers each hold one share).
The operational burden of manual Vault unsealing (requiring 3 key holders to cooperate every time the Vault process restarts) is often cited as the reason teams don't adopt Vault. Auto-unseal with a cloud KMS (Key Management Service) solves this. Configure the seal stanza in Vault's config file to use GCP KMS: Vault encrypts its master key with a KMS key. On startup, Vault automatically decrypts its master key via KMS API without human intervention. The GCP KMS key is protected by GCP's HSM-backed key management. Auto-unseal allows Vault to restart automatically (after VM patching, crashes) without requiring multiple humans to be on-call to re-enter key shares.
From my experience: start Vault with the Raft integrated storage backend rather than Consul for smaller deployments (under 5 nodes). Raft is built into Vault since version 1.4 and provides high-availability clustering without running a separate Consul cluster. A 3-node Vault cluster using Raft provides HA at lower operational complexity. Deploy each Vault node in a different availability zone for resilience. The Raft storage backend replicates all data across nodes — one node is the leader, others are followers. If the leader fails, Raft elects a new leader automatically.
The Vault database secrets engine generates dynamic credentials for PostgreSQL, MySQL, MongoDB, and other databases. Configure it with a Vault role that defines the TTL and the SQL statements to CREATE and REVOKE users. When an application needs database access, it requests credentials from Vault via the API. Vault creates a unique database user (vault-my-app-20240515-abc123) with the appropriate permissions, returns the username and password to the application, and marks the credential for expiration at the TTL. When the TTL expires or the application calls Vault to explicitly revoke the credential, Vault drops the database user. No shared, long-lived passwords.
# Configure Vault database secrets engine for PostgreSQL
vault secrets enable database
vault write database/config/mydb plugin_name=postgresql-database-plugin allowed_roles="my-app-role" connection_url="postgresql://{{username}}:{{password}}@db.internal:5432/mydb?sslmode=require" username="vault_admin" password="initial_password"
# Create a role that generates short-lived credentials
vault write database/roles/my-app-role db_name=mydb creation_statements="CREATE ROLE "{{name}}" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT, INSERT ON ALL TABLES IN SCHEMA public TO "{{name}}";" default_ttl="1h" max_ttl="24h"
# Application requests dynamic credentials
vault read database/creds/my-app-role
# Key Value
# --- -----
# username vault-my-app-20260515-abc123xyz
# password A1B2-C3D4-E5F6-G7H8
# lease_duration 1h
# Auto-unseal configuration (config.hcl)
# seal "gcpckms" {
# project = "my-gcp-project"
# region = "asia-southeast1"
# key_ring = "vault-keyring"
# crypto_key = "vault-unseal-key"
# }Applications authenticate to Vault using auth methods. AppRole is designed for machine-to-machine authentication: a role ID (public, like a username) and a secret ID (private, like a password, valid for a short time or single use). The CI/CD pipeline retrieves the secret ID from Vault at runtime using a bootstrap credential, combines it with the role ID, and authenticates to Vault to receive a short-lived token with specific policy permissions. For Kubernetes, the Kubernetes auth method is simpler: the pod's service account token is used to authenticate to Vault directly, without managing role IDs or secret IDs.
┌──────────────────────────────────────────────────────┐
│ HashiCorp Vault Architecture │
├──────────────────────────────────────────────────────┤
│ │
│ ┌────────────┐ ┌────────────┐ ┌──────────┐ │
│ │ Vault │ │ Vault │ │ Vault │ │
│ │ Node 1 │◄───►│ Node 2 │◄───►│ Node 3 │ │
│ │ (Leader) │ │ (Follower)│ │ (Follower│ │
│ └─────┬──────┘ └────────────┘ └──────────┘ │
│ │ Raft consensus (integrated storage) │
│ ↓ │
│ ┌─────────────────────────────────┐ │
│ │ GCP KMS (Auto-Unseal) │ │
│ │ HSM-backed encryption key │ │
│ └─────────────────────────────────┘ │
└──────────────────────────────────────────────────────┘Vault generates a root token during initialization that has unrestricted access to everything in Vault. I've seen teams use this root token in application configuration files because it's the first credential they have after initialization. The root token should be used exactly once — to set up the initial policies, auth methods, and secrets engines — and then revoked. Store the root token in a physical safe or HSM, not in a file or secrets manager. The root token is the master key to your entire secrets infrastructure; treating it like a regular API key is a serious security failure.
Enable Vault's audit log to file and optionally to a syslog target. The audit log records every request to Vault — who authenticated, what secret they accessed or wrote, what the response was (hashed), and when. This creates a complete paper trail for compliance (PCI-DSS, SOC 2, ISO 27001 all require audit trails for secret access). Write Vault policies (in HCL) that follow the principle of least privilege: an application should have read access only to the specific secret paths it needs, never write access to secrets it doesn't create, and never delete access at all. Policies are the primary security control in Vault — misconfigured policies are the primary way Vault deployments get compromised.
For Kubernetes workloads, the Vault Agent Injector is the standard integration pattern. It's a Kubernetes mutating webhook that automatically injects a Vault Agent sidecar into annotated pods. The sidecar authenticates to Vault using the pod's Kubernetes service account, retrieves the specified secrets, and writes them to a shared memory volume that your application container reads. When secrets rotate, the sidecar re-fetches the new values and optionally signals your application to reload. No Vault SDK integration in your application code — secrets appear as files in the container's filesystem.
GCP Secret Manager and AWS Secrets Manager are excellent managed alternatives to self-hosted Vault. They solve the core problem (no plaintext secrets in code) with zero operational overhead. I use GCP Secret Manager for simpler client projects where self-hosted Vault would be over-engineered. I use Vault for projects where: (1) we need dynamic database credentials (Secret Manager doesn't generate these); (2) we need secrets management across multiple cloud providers (Vault is provider-agnostic); (3) compliance requirements mandate an on-premises secrets store; or (4) we need the rich policy model and audit capabilities that Vault provides. For an Indonesian SME with a small DevOps team, start with the cloud-native option and graduate to Vault when you need its specific features.
Sources & Further Reading