Scrolls
7 min read
On this page

Key Management

Encryption is only as strong as your key management. You can use AES-256-GCM with a perfectly random key, and it means nothing if that key is hardcoded in your source code, committed to Git, or stored alongside the encrypted data. Key management is consistently identified as the hardest part of applied cryptography -- not because the concepts are complex, but because the operational discipline is demanding.

The Key Management Lifecycle

Every cryptographic key goes through a lifecycle. Understanding this lifecycle prevents the most common key management failures.

Generation → Storage → Distribution → Use → Rotation → Retirement → Destruction
  • Generation: Keys must be generated using cryptographically secure random number generators. Never derive keys from passwords without a proper KDF (Key Derivation Function). Never use math/rand or Math.random().
  • Storage: Keys must be stored in a system designed for key storage (HSM, KMS). Never in source code, config files, or databases.
  • Distribution: Keys must be delivered securely. Never over email, Slack, or unencrypted channels.
  • Use: Keys should have a single purpose. Do not use the same key for encryption and signing.
  • Rotation: Keys must be rotated on a regular schedule and immediately when compromise is suspected.
  • Retirement: Old keys are kept in read-only mode to decrypt existing data during a migration period.
  • Destruction: After all data encrypted with a key has been re-encrypted or deleted, the key is securely destroyed.

Key Storage

Hardware Security Modules (HSMs)

An HSM is a dedicated hardware device for key storage and cryptographic operations. The key never leaves the HSM -- all encryption and decryption happens inside the device. HSMs are tamper-resistant and tamper-evident.

Application → API call → HSM performs encryption → Returns ciphertext
                          (key never leaves HSM)

HSMs are used for:

  • Root CA private keys
  • Payment processing (PCI DSS often requires HSMs)
  • Government and military applications (FIPS 140-2 Level 3+)

Cloud providers offer HSM services:

  • AWS CloudHSM
  • Google Cloud HSM
  • Azure Dedicated HSM

HSMs are expensive and operationally complex. Most applications do not need dedicated HSMs -- a managed KMS is sufficient.

Key Management Services (KMS)

A KMS is a managed service for key storage and cryptographic operations. It handles generation, storage, rotation, and access control. The cloud provider manages the underlying HSMs.

# AWS KMS: create a key
aws kms create-key --description "Application data encryption key"

# Encrypt data
aws kms encrypt \
  --key-id alias/my-app-key \
  --plaintext fileb://secret.txt \
  --output text --query CiphertextBlob | base64 --decode > encrypted.bin

# Decrypt data
aws kms decrypt \
  --ciphertext-blob fileb://encrypted.bin \
  --output text --query Plaintext | base64 --decode > decrypted.txt

# Google Cloud KMS: create a key ring and key
gcloud kms keyrings create my-keyring --location global
gcloud kms keys create my-key \
  --location global \
  --keyring my-keyring \
  --purpose encryption

# Encrypt
gcloud kms encrypt \
  --location global \
  --keyring my-keyring \
  --key my-key \
  --plaintext-file secret.txt \
  --ciphertext-file encrypted.bin

KMS advantages over managing keys yourself:

  • Access control through IAM policies
  • Automatic key rotation
  • Audit logging of every key usage
  • High availability and durability
  • No need to handle raw key material

Envelope Encryption

Envelope encryption is the standard pattern for encrypting data with a KMS. You do not send your data to the KMS for encryption (that would be slow and have size limits). Instead, you use two layers of keys.

1. Generate a Data Encryption Key (DEK) locally
2. Encrypt your data with the DEK (fast, local, AES-256-GCM)
3. Encrypt the DEK with the Master Key in KMS (small operation, one API call)
4. Store the encrypted DEK alongside the encrypted data
5. Delete the plaintext DEK from memory

┌─────────────────────────────────────────────┐
│ Stored Together:                            │
│   - Encrypted Data (encrypted with DEK)     │
│   - Encrypted DEK (encrypted with KMS key)  │
└─────────────────────────────────────────────┘

To decrypt:
1. Send encrypted DEK to KMS → get plaintext DEK
2. Decrypt data with plaintext DEK
3. Delete plaintext DEK from memory

Envelope encryption advantages:

  • Data stays local. You never send large payloads to the KMS.
  • Each piece of data can have its own DEK, limiting blast radius.
  • Rotating the master key does not require re-encrypting all data. You re-encrypt only the DEKs.
  • KMS rate limits and size limits are not a bottleneck.

AWS S3 server-side encryption, Google Cloud Storage encryption, and Azure Storage encryption all use envelope encryption internally.

Key Rotation

Key rotation is replacing an existing key with a new key on a regular schedule. It limits the amount of data encrypted under any single key and reduces the impact of a key compromise.

Rotation Strategies

Strategy              How It Works                     When to Use
─────────────────────────────────────────────────────────────────────
Automatic rotation    KMS rotates the key material     Default for KMS keys
                      on a schedule (e.g., yearly).
                      Old versions kept for decryption.

Re-encryption         Generate a new key, decrypt       When you need to
                      all data with the old key,        eliminate old key
                      re-encrypt with the new key.      material entirely.

Envelope re-wrap      Keep data encrypted with same     Most common approach.
                      DEK. Decrypt DEK with old         Fast, no data
                      master key, re-encrypt DEK        re-encryption needed.
                      with new master key.

# AWS KMS automatic key rotation (rotates key material annually)
aws kms enable-key-rotation --key-id alias/my-app-key

# Check rotation status
aws kms get-key-rotation-status --key-id alias/my-app-key

AWS KMS automatic rotation keeps old key material indefinitely, so data encrypted with previous versions can still be decrypted. The key ID and alias remain the same. This is transparent to your application.

When to Rotate Immediately

  • A key is suspected or confirmed to be compromised
  • An employee with key access leaves the organization
  • A security audit finds the key was exposed (logs, error messages, backups)
  • The key has been in use beyond its cryptoperiod (check your compliance requirements)

Never Hardcode Keys

This is the most violated rule in key management. Engineers hardcode keys because it is easy and because the key is "just for development." Then the development key ends up in production.

# NEVER do this
API_KEY = "AKIAIOSFODNN7EXAMPLE"
DB_PASSWORD = "super_secret_password_123"
ENCRYPTION_KEY = "0123456789abcdef0123456789abcdef"

Never Commit Keys to Git

Git history is permanent. Even if you delete the file in a later commit, the key is still in the history. Rewriting Git history is painful and unreliable when the repository has been cloned.

# .gitignore: prevent common secret files from being committed
.env
.env.local
*.pem
*.key
credentials.json
service-account.json

Use pre-commit hooks to catch secrets before they enter the repository:

# Install git-secrets (by AWS)
git secrets --install
git secrets --register-aws

# Or use truffleHog, gitleaks, or detect-secrets
gitleaks detect --source . --verbose

GitHub's secret scanning automatically detects exposed credentials from major providers (AWS, GCP, Azure, Stripe, etc.) and alerts the provider, who may revoke the key automatically. In 2023, GitHub reported that it detected over 1 million exposed secrets in public repositories in a single month.

Real-World Key Management Failures

Uber (2016)

Attackers accessed Uber's private GitHub repository and found AWS credentials hardcoded in the source code. They used those credentials to access an S3 bucket containing personal data of 57 million users and 600,000 drivers. Uber paid the attackers $100,000 to delete the data and did not disclose the breach for over a year. Using a secret manager and restricting S3 access through IAM roles would have prevented this.

Capital One (2019)

A misconfigured WAF allowed an attacker to obtain temporary AWS credentials through the instance metadata service (IMDS). The credentials had excessive permissions, granting access to S3 buckets containing 106 million customer records. The root cause was not key management alone but the combination of overly permissive IAM roles and the ability to reach IMDS. AWS later released IMDSv2 to mitigate this class of attack.

SolarWinds (2020)

During the investigation of the SolarWinds supply chain attack, researchers discovered that a SolarWinds intern had set a critical server password to "solarwinds123" and it was exposed on a public GitHub repository. While this was not the primary attack vector, it illustrated the organizational culture around credential management. Strong key management requires organizational discipline, not just tooling.

The Secret Hierarchy

Not all secret storage approaches are equal. Choose the highest level your infrastructure supports.

Level     Method                   Risk
──────────────────────────────────────────────────────
Worst     Hardcoded in source      Permanent exposure in Git history
Bad       Config files on disk     Readable by anyone with server access
Okay      Environment variables    Visible in process listings, crash dumps
Better    Encrypted config files   Key management for the config key
Best      Secret manager (Vault,   Centralized, audited, rotatable,
          AWS Secrets Manager)     access-controlled

Common Pitfalls

  • Hardcoding keys in source code. The most common and most preventable key management failure. Use pre-commit hooks and secret scanning to catch this.
  • Using the same key for everything. A single compromised key should not unlock all your data. Use separate keys for separate services and data classifications.
  • Not rotating keys. Keys should be rotated on a schedule and immediately when compromise is suspected. Automate rotation so it is not a manual, error-prone process.
  • Storing the encryption key next to the encrypted data. If an attacker gets your database dump and the key is in the same database or the same server, encryption provided zero protection.
  • Logging keys or including them in error messages. Audit your logging pipeline. Ensure keys, tokens, and passwords are never written to log files, error tracking services, or monitoring dashboards.
  • No access control on key material. Every key access should be authenticated, authorized, and logged. KMS services provide this by default. DIY key storage usually does not.
  • Ignoring the blast radius. When a key is compromised, you need to know exactly what data it protects, who had access, and how to rotate it. Document this before you need it.

Key Takeaways

  • Key management is harder than encryption itself. The algorithm is rarely the weak point -- key handling is.
  • Use a KMS (AWS KMS, Google Cloud KMS) for key storage and operations. The key material never leaves the managed environment.
  • Envelope encryption is the standard pattern: encrypt data with a local DEK, encrypt the DEK with a KMS master key.
  • Rotate keys on a regular schedule and immediately on suspected compromise. Automate rotation.
  • Never hardcode keys in source code. Never commit keys to Git. Use pre-commit hooks and secret scanning.
  • The hierarchy of secret storage, from worst to best: hardcoded, config files, environment variables, encrypted config, secret manager.
  • Document which keys protect which data. When a key is compromised, you need to know the blast radius immediately.