Secret Managers
A secret manager is centralized infrastructure for storing, accessing, and rotating secrets. Instead of scattering credentials across environment variables, config files, and deployment scripts, every service retrieves secrets from a single, access-controlled, audited system. The secret manager becomes the single source of truth for every credential in your organization. It is infrastructure, not a convenience tool.
Why Use a Secret Manager
Without a secret manager:
- Secrets in .env files on developer machines
- Secrets in CI/CD environment variables
- Secrets in Kubernetes ConfigMaps (not encrypted)
- Secrets copied between environments via Slack
- No audit trail of who accessed what
- Rotation requires touching every deployment
With a secret manager:
- One place to store, access, and rotate secrets
- Access controlled by identity and policy
- Full audit log of every access
- Rotation is automated and zero-downtime
- Secrets never touch disk or version control
HashiCorp Vault
Vault is the most comprehensive secret management solution. It is open source, self-hosted (or managed through HCP Vault), and supports a wide range of secret engines, authentication methods, and integrations. It is also the most complex to operate.
Core Concepts
Concept What It Does
──────────────────────────────────────────────────────────
Secret engine Backend that stores or generates secrets
(KV, database, AWS, PKI, etc.)
Auth method How clients authenticate to Vault
(token, AppRole, Kubernetes, OIDC, etc.)
Policy What secrets an authenticated client
can access (path-based ACLs)
Lease Time limit on dynamic secrets.
Secrets expire and must be renewed.
Seal/Unseal Vault encrypts all data at rest. It must
be "unsealed" with master key shares
before it can serve requests.
Basic Operations
# Start Vault in development mode (never for production)
vault server -dev
# Write a secret
vault kv put secret/myapp/database \
username="app_user" \
password="db_password_here"
# Read a secret
vault kv get secret/myapp/database
# Output:
# Key Value
# --- -----
# username app_user
# password db_password_here
# Read a specific field
vault kv get -field=password secret/myapp/database
Dynamic Secrets
Dynamic secrets are Vault's most powerful feature. Instead of storing a static database password, Vault generates a new, temporary credential for each request. When the lease expires, Vault revokes the credential.
# Configure the database secret engine
vault secrets enable database
vault write database/config/mydb \
plugin_name=postgresql-database-plugin \
connection_url="postgresql://{{username}}:{{password}}@db.example.com:5432/mydb" \
allowed_roles="readonly" \
username="vault_admin" \
password="vault_admin_password"
vault write database/roles/readonly \
db_name=mydb \
creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
default_ttl="1h" \
max_ttl="24h"
# Request temporary credentials
vault read database/creds/readonly
# Output:
# Key Value
# --- -----
# lease_id database/creds/readonly/abc123
# lease_duration 1h
# username v-token-readonly-xyz789
# password A1B2C3D4E5F6...
Each request produces a unique username and password that expires automatically. If a credential is compromised, the blast radius is limited to the lease duration. No manual rotation needed.
Vault Complexity
Vault is powerful but operationally demanding:
- Requires high availability setup (Raft consensus or Consul backend)
- Unsealing after restarts requires key shares (or auto-unseal with a cloud KMS)
- Needs monitoring, backup, and disaster recovery planning
- Policy management becomes complex at scale
- Development teams need training on Vault APIs and patterns
For teams without dedicated infrastructure engineers, a managed alternative may be more practical.
AWS Secrets Manager
AWS Secrets Manager is a managed service that stores and rotates secrets. It integrates natively with AWS services (RDS, Redshift, DocumentDB) and provides automatic rotation through Lambda functions.
# Store a secret
aws secretsmanager create-secret \
--name myapp/database \
--secret-string '{"username":"admin","password":"db_password_here"}'
# Retrieve a secret
aws secretsmanager get-secret-value --secret-id myapp/database
# Enable automatic rotation (for RDS)
aws secretsmanager rotate-secret \
--secret-id myapp/database \
--rotation-lambda-arn arn:aws:lambda:us-east-1:123456:function:rotate-db \
--rotation-rules AutomaticallyAfterDays=30
Application Integration
# Python: retrieve secret at application startup
import boto3
import json
def get_secret(secret_name):
client = boto3.client('secretsmanager')
response = client.get_secret_value(SecretId=secret_name)
return json.loads(response['SecretString'])
db_creds = get_secret('myapp/database')
# db_creds['username'], db_creds['password']
AWS Secrets Manager costs 0.05 per 10,000 API calls. For most applications, this is negligible.
Google Secret Manager
Google Secret Manager provides similar functionality in GCP. Secrets are versioned, and you can access specific versions or always retrieve the latest.
# Create a secret
gcloud secrets create myapp-database \
--replication-policy="automatic"
# Add a version (the actual secret value)
echo -n "db_password_here" | gcloud secrets versions add myapp-database --data-file=-
# Access the latest version
gcloud secrets versions access latest --secret=myapp-database
# Access a specific version
gcloud secrets versions access 3 --secret=myapp-database
# Application code (Python)
from google.cloud import secretmanager
client = secretmanager.SecretManagerServiceClient()
name = "projects/my-project/secrets/myapp-database/versions/latest"
response = client.access_secret_version(request={"name": name})
secret_value = response.payload.data.decode("UTF-8")
Versioning is useful for rotation: add a new version, update applications, then disable the old version.
Developer-Friendly Alternatives
Doppler
Doppler is a secret management platform designed for developer experience. It syncs secrets to any environment (local development, CI/CD, production) through a CLI and integrations.
# Install and authenticate
doppler setup
# Run an application with secrets injected
doppler run -- python app.py
# Secrets are injected as environment variables
# No code changes needed
# Fetch a specific secret
doppler secrets get DATABASE_URL
Doppler manages secrets across environments (development, staging, production) and provides a dashboard for teams to manage access.
1Password Secrets Automation
1Password provides a secret management API (1Password Connect) that lets applications retrieve secrets stored in 1Password vaults. This is useful for teams already using 1Password for team password management.
# 1Password CLI: read a secret
op read "op://Production/Database/password"
# Inject secrets into environment
op run --env-file=.env.tpl -- python app.py
# .env.tpl template
DATABASE_URL=op://Production/Database/url
API_KEY=op://Production/Stripe/secret-key
Secret Rotation
Rotation replaces an existing secret with a new one. The goal is zero-downtime rotation -- applications continue operating without interruption during the transition.
Zero-Downtime Rotation Pattern
Step 1: Create new credential (old still works)
Step 2: Update secret manager with new credential
Step 3: Applications fetch new credential (on next read or restart)
Step 4: Verify all applications are using the new credential
Step 5: Revoke old credential
The key: both old and new credentials must work simultaneously
during the transition period.
Automated Rotation with AWS Secrets Manager
AWS Secrets Manager supports automatic rotation for RDS databases through Lambda functions. The rotation Lambda follows a four-step process:
1. createSecret - Generate new credentials, store as AWSPENDING
2. setSecret - Set the new credentials in the database
3. testSecret - Verify the new credentials work
4. finishSecret - Mark AWSPENDING as AWSCURRENT, old becomes AWSPREVIOUS
AWS provides rotation Lambda templates for common databases. For custom secrets (API keys, third-party services), you write your own rotation Lambda.
Why Automate Rotation
Manual rotation is error-prone and often skipped:
- Someone forgets to rotate after an employee departure
- The rotation process is undocumented and only one person knows how
- Rotation causes an outage because the process was never tested
- Compliance requires rotation every 90 days but it happens once a year
Automate rotation from the beginning. If you cannot automate, at minimum document the process and schedule regular rotation.
Real-World Incidents
CircleCI (2023)
CircleCI disclosed a security breach where an attacker gained access to customer secrets stored in CircleCI's environment. CircleCI advised all customers to rotate every secret stored in CircleCI. Organizations that used a secret manager (fetching secrets at runtime rather than storing them in CircleCI) had less exposure -- their CircleCI environment contained references to secrets, not the secrets themselves.
LastPass (2022)
LastPass, a password manager, suffered a breach where attackers accessed encrypted customer vault data. While the vaults were encrypted with user master passwords, the breach exposed the risk of centralized secret storage. The key lesson: even the secret manager can be compromised. Defense in depth means the secret manager should be one layer, not the only layer. Encrypt sensitive data at the application level in addition to using a secret manager.
Heroku/Travis CI (2022)
Stolen OAuth tokens for Heroku and Travis CI were used to access private GitHub repositories of organizations including npm. The attackers extracted secrets from CI/CD environment variables. Organizations that stored references to a secret manager (rather than raw secrets) in their CI environment were less affected.
Common Pitfalls
- Using a secret manager but caching secrets indefinitely. Applications should refresh secrets periodically or on failure. If a secret is rotated but the application never re-reads it, the rotation is ineffective.
- Not restricting access to the secret manager. A secret manager with a permissive access policy is a single point of failure. Apply least-privilege access: each service should only access the secrets it needs.
- Storing secret manager credentials in code. The secret manager's own authentication credentials (Vault tokens, AWS credentials) must also be managed securely. Use instance roles, Kubernetes service accounts, or OIDC federation -- not hardcoded tokens.
- No monitoring or alerting on secret access. Unusual secret access patterns (bulk reads, access from unexpected IPs, access to secrets a service does not normally use) should trigger alerts.
- Choosing the most complex tool. Vault is powerful but requires significant operational investment. If you are on AWS, Secrets Manager is simpler and sufficient for most use cases. Choose based on your team's operational capacity.
- Not testing rotation. Rotation that has never been tested will fail when you need it most. Run rotation in staging regularly. Test what happens when an application gets a rotated credential.
Key Takeaways
- A secret manager is infrastructure, not optional tooling. It centralizes secret storage, access control, auditing, and rotation.
- HashiCorp Vault is the most powerful option, with dynamic secrets and extensive integrations. It is also the most complex to operate.
- AWS Secrets Manager and Google Secret Manager are managed, simpler alternatives that integrate with their respective cloud platforms.
- Doppler and 1Password Secrets Automation prioritize developer experience and are good choices for smaller teams.
- Dynamic secrets (Vault) eliminate the concept of a long-lived credential. Each consumer gets a unique, temporary credential that auto-expires.
- Automate secret rotation. Manual rotation is unreliable and often causes outages when finally attempted.
- The secret manager itself must be secured. Use instance roles and OIDC federation for authentication, not hardcoded tokens.