Scrolls
4 min read
On this page

Secrets Management

A secret is any credential that grants access to something: API keys, database passwords, TLS certificates, SSH keys, tokens. Secrets are the keys to your kingdom. If an attacker gets one, they get access to everything that secret protects. Managing secrets properly is not optional -- it is foundational to security.

Where Secrets Should Not Be

Before discussing where secrets should go, it is critical to understand where they must never go.

Never in Code

# This is a real pattern found in production codebases
DB_PASSWORD = "s3cure_p4ssw0rd_2024"
AWS_SECRET_KEY = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
STRIPE_SECRET = "sk_live_4eC39HqLyjWDarjtT1zdp7dc"

Every one of these is a security incident waiting to happen.

Secrets in code get committed to version control. Once in git history, they are effectively permanent. Even if you delete the file, the secret remains in the commit history. Rotating the secret becomes urgent.

Never in Environment Variables on Disk

Environment variables set in shell profiles (~/.bashrc, ~/.zshrc) or .env files committed to repos are secrets on disk in plaintext. Any process running as your user can read them. Any log that dumps the environment exposes them.

Problematic patterns:
  - .env file committed to git (even if .gitignored, it's on disk)
  - export DB_PASSWORD="..." in ~/.bashrc
  - docker-compose.yml with hardcoded secrets
  - Kubernetes manifests with plaintext secrets in ConfigMaps

Never in Logs

Applications accidentally log secrets more often than you would think.

Common logging mistakes:
  - Logging the full HTTP request including Authorization headers
  - Logging connection strings that include passwords
  - Logging environment variables during startup for "debugging"
  - Error messages that include the secret in the stack trace

Where Secrets Should Be

Secrets belong in dedicated secrets management systems that provide encryption, access control, auditing, and rotation.

HashiCorp Vault

Vault is the most widely deployed secrets management system. It stores secrets encrypted, provides fine-grained access control, and supports automatic rotation.

# Store a secret in Vault
vault kv put secret/payment-service/db \
  username="payment_user" \
  password="generated_password_here"

# Read a secret from Vault
vault kv get secret/payment-service/db

# Application retrieves secret at runtime
# (via Vault Agent, sidecar, or API call)

Vault features:
  - Dynamic secrets: generate short-lived database credentials on demand
  - Secret rotation: automatically rotate passwords on a schedule
  - Audit logging: every access is logged (who, when, what)
  - Fine-grained policies: each service gets only the secrets it needs
  - Multiple auth methods: Kubernetes, AWS IAM, GitHub, OIDC

AWS Secrets Manager

# Store a secret
aws secretsmanager create-secret \
  --name "payment-service/db-password" \
  --secret-string "generated_password_here"

# Retrieve a secret in application code (Python example)
# import boto3
# client = boto3.client('secretsmanager')
# response = client.get_secret_value(SecretId='payment-service/db-password')
# secret = response['SecretString']

AWS Secrets Manager features:
  - Automatic rotation with Lambda functions
  - Integration with RDS, Redshift, DocumentDB
  - Cross-account sharing
  - Encryption with KMS
  - CloudTrail audit logging

Doppler

Doppler is a newer entrant focused on developer experience. It syncs secrets across environments and integrates with CI/CD, cloud providers, and local development.

# Pull secrets into your local environment
doppler run -- npm start

# Secrets are injected as environment variables
# but never stored on disk

Choosing a Secrets Manager

HashiCorp Vault:
  Best for: Large organizations, multi-cloud, complex policies
  Complexity: High (requires dedicated infrastructure)
  Cost: Free (open source) or Enterprise license

AWS Secrets Manager:
  Best for: AWS-native organizations
  Complexity: Low (managed service)
  Cost: $0.40/secret/month + $0.05/10K API calls

Google Secret Manager:
  Best for: GCP-native organizations
  Complexity: Low (managed service)
  Cost: Free for first 6 active secret versions

Doppler:
  Best for: Small to mid-size teams, multi-environment
  Complexity: Low
  Cost: Free tier, then per-seat pricing

Secret Rotation

Secrets should have a limited lifetime. If a secret never changes, a compromise may go undetected indefinitely.

Rotation schedule:
  API keys:            Every 90 days
  Database passwords:  Every 90 days
  TLS certificates:    Every 90 days (or use auto-renewal)
  Service tokens:      Every 30 days
  SSH keys:            Every 180 days
  
  After a security incident: immediately rotate all affected secrets

Automated Rotation

Manual rotation does not scale. Automate it.

# AWS Secrets Manager automatic rotation
resource "aws_secretsmanager_secret_rotation" "db_password" {
  secret_id           = aws_secretsmanager_secret.db_password.id
  rotation_lambda_arn = aws_lambda_function.rotate_secret.arn

  rotation_rules {
    automatically_after_days = 90
  }
}

Rotation strategy for database passwords:
  1. Generate new password
  2. Create new database user with new password (or update existing)
  3. Update secret in secrets manager
  4. Applications pick up new secret on next connection
  5. After grace period, revoke old password
  
  The key: applications must handle credential refresh gracefully.
  Hard-coded connection pools that never refresh will break.

Least Privilege

Each service should get only the secrets it needs. A frontend service should not have access to the database password. A reporting service should not have access to the payment API key.

Bad: shared secrets
  All services use the same database password.
  If one service is compromised, the attacker has full
  database access.

Good: per-service secrets
  payment-service: has payment-db-password, stripe-api-key
  user-service:    has user-db-password, auth0-api-key
  reporting:       has reporting-db-readonly-password
  
  If reporting is compromised, the attacker gets read-only
  access to the reporting database. Not the payment database.
  Not the Stripe API key.

Vault Policies

# Vault policy for payment-service
path "secret/data/payment-service/*" {
  capabilities = ["read"]
}

# payment-service can read its own secrets
# It cannot read secrets for user-service or any other service
# It cannot write or delete secrets

Secrets in CI/CD

CI/CD pipelines need secrets to deploy, push to registries, and access cloud resources. These secrets must be managed carefully.

GitHub Encrypted Secrets

# Secrets are referenced but never exposed in workflow files
name: Deploy
on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Login to container registry
        run: |
          echo "${{ secrets.REGISTRY_PASSWORD }}" | \
            docker login -u "${{ secrets.REGISTRY_USERNAME }}" --password-stdin
      
      - name: Deploy
        run: ./deploy.sh
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

CI/CD secrets best practices:
  - Use the platform's secret storage (GitHub secrets, GitLab CI variables)
  - Mark secrets as masked (hidden in logs)
  - Use OIDC federation instead of long-lived credentials when possible
  - Scope secrets to the narrowest environment (production secrets
    only available in production deploy jobs)
  - Never echo or print secrets in pipeline output

OIDC Federation (No Long-Lived Secrets)

The best secret is no secret at all. OIDC federation allows CI/CD pipelines to authenticate to cloud providers using short-lived tokens instead of long-lived API keys.

# GitHub Actions with OIDC to AWS (no AWS secret keys needed)
jobs:
  deploy:
    permissions:
      id-token: write
      contents: read
    runs-on: ubuntu-latest
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789:role/deploy-role
          aws-region: us-east-1
      
      # No AWS_SECRET_ACCESS_KEY needed
      # GitHub provides a short-lived OIDC token
      # AWS trusts GitHub as an identity provider
      - run: aws ecs update-service --cluster prod --service myapp

What to Do When a Secret Leaks

Secrets leak. It happens. What matters is how fast you respond.

Incident response for a leaked secret:

1. IMMEDIATE (within minutes):
   - Rotate the compromised secret
   - Revoke the old secret
   - Do not wait to investigate — rotate first

2. WITHIN 1 HOUR:
   - Identify all systems that used the compromised secret
   - Update all systems with the new secret
   - Check logs for unauthorized access using the old secret

3. WITHIN 24 HOURS:
   - Determine how the secret leaked (committed to git, logged,
     shared in Slack, insider threat)
   - Check for lateral movement (did the attacker use the secret
     to access other secrets or systems?)
   - Write an incident report

4. WITHIN 1 WEEK:
   - Implement controls to prevent the same type of leak
   - Add secret scanning to prevent commits containing secrets
   - Review access policies for the affected system

Preventing Leaks

# Pre-commit hook to scan for secrets
# Using gitleaks
gitleaks detect --source . --verbose

# GitHub secret scanning (automatic for public repos)
# Alerts when secrets are pushed to the repo
# Partners (AWS, Stripe, etc.) are notified and can auto-revoke

# Example .gitleaks.toml configuration
# title = "Gitleaks config"
# [[rules]]
# id = "aws-access-key"
# description = "AWS Access Key"
# regex = '''AKIA[0-9A-Z]{16}'''
# tags = ["aws", "credentials"]

Real-World Example

A startup's developer accidentally committed an AWS root account access key to a public GitHub repository. Within 15 minutes, automated scanners found the key and began spinning up cryptocurrency mining instances. By the time the team noticed the next morning, the AWS bill was $12,000.

They implemented a secrets management program:

  1. Rotated all AWS credentials immediately
  2. Enabled MFA on the root account and stopped using root keys entirely
  3. Deployed gitleaks as a pre-commit hook on every developer machine
  4. Moved all secrets to AWS Secrets Manager
  5. Replaced long-lived IAM keys with OIDC federation for CI/CD
  6. Added GitHub's secret scanning alerts to all repositories

The total cost of the incident (AWS bill + engineering time + security audit): roughly 25,000.Thecostofthepreventionmeasures:roughly25,000. The cost of the prevention measures: roughly 2,000 per year. The math is clear.

Common Pitfalls

  • Secrets in git history -- Even if you delete the file, the secret is in the commit history; you must rotate the secret, not just remove the file
  • Shared secrets -- One database password used by ten services means compromising any one service compromises the database; use per-service credentials
  • No rotation -- Secrets that never change accumulate risk; automate rotation on a schedule
  • Over-privileged CI/CD secrets -- A deploy pipeline with admin-level cloud credentials can destroy everything; scope CI/CD credentials to the minimum required permissions
  • Trusting .gitignore -- .gitignore prevents committing a file, but the file still exists on disk and in backups; use a secrets manager, not files
  • No audit logging -- If you do not log who accessed which secret and when, you cannot detect unauthorized access; enable audit logging on your secrets manager

Key Takeaways

  • Never store secrets in code, on disk in plaintext, or in environment variable files committed to version control
  • Use a dedicated secrets manager: Vault, AWS Secrets Manager, Google Secret Manager, or Doppler
  • Rotate secrets on a schedule and immediately after any suspected compromise
  • Apply least privilege: each service gets only the secrets it needs, nothing more
  • In CI/CD, use platform-native secret storage and prefer OIDC federation over long-lived credentials
  • When a secret leaks, rotate first, investigate second; speed matters more than understanding