Scrolls
6 min read
On this page

Never in Code

The most common security mistake in software engineering is putting secrets in source code. API keys, database passwords, encryption keys, service account credentials -- committed to Git, pushed to a repository, and forgotten. Attackers know this. Automated scanners continuously scrape public repositories for credentials. Exposed AWS keys are exploited within minutes, often before the developer even realizes the commit happened. This is not a theoretical risk. It is the single most frequent cause of cloud account compromise.

Why Secrets End Up in Code

Developers put secrets in code because it is the path of least resistance. The application needs a database password to start. Hardcoding it takes five seconds. Setting up a secret manager takes an hour. So the secret goes into a config file, gets committed, and stays there.

# How secrets typically end up in source code

1. Developer hardcodes a secret during local development
2. Forgets to remove it before committing
3. Secret is pushed to the repository
4. Even if deleted in a later commit, it remains in Git history forever
5. Repository is cloned, forked, or made public
6. Automated scanners find the secret within minutes

The Speed of Exploitation

In 2023, researchers from Truffle Security published a study where they deliberately committed AWS keys to public GitHub repositories. The keys were discovered and used by attackers in an average of 2 minutes. Some keys were exploited in under 30 seconds. The attackers used the credentials to spin up cryptocurrency mining instances, and the charges began accumulating immediately.

GitHub reports detecting over 1 million exposed secrets per month in public repositories through its secret scanning program. This is just what is detectable through pattern matching -- the actual number is higher.

The Secret Hierarchy

Not all secret storage approaches are equal. Each level improves on the last.

Level         Method                    Risk Profile
──────────────────────────────────────────────────────────────────────
Never         Hardcoded in source       Permanent. In Git history forever.
              code                      Visible to anyone with repo access.
                                        Automated scanners find it in minutes.

Never         Committed config files    Same as hardcoded. .json, .yaml, .ini
              (database.yml,            files with passwords are committed
              config.json)              constantly.

Bad           .env files committed      Developers create .env, forget to add
              to Git                    it to .gitignore, commit it.

Acceptable    .env files (not           Stays on the developer's machine.
              committed) or             Visible in process listings and
              environment variables     crash dumps. Acceptable for simple
                                        setups and local development.

Better        Encrypted config files    The config is encrypted, but you
              (SOPS, git-crypt)         still need to manage the encryption
                                        key securely.

Best          Secret manager            Centralized, access-controlled,
              (Vault, AWS Secrets       audited, rotatable. Secrets are
              Manager)                  never on disk or in environment
                                        variables.

.env Files: Better But Still Risky

.env files keep secrets out of source code but introduce their own risks.

# .env file
DATABASE_URL=postgres://admin:password123@db.example.com:5432/production
AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
STRIPE_SECRET_KEY=sk_live_4eC39HqLyjWDarjtT1zdp7dc

The critical rule: never commit .env files to Git.

# .gitignore: always include these
.env
.env.local
.env.production
.env.*.local

Risks of .env files:

  • Developers copy them between machines over Slack, email, or shared drives
  • They exist as plaintext on the filesystem, readable by any process running as the same user
  • Container orchestrators may mount them as volumes, making them visible to other containers
  • Backup systems may include them in snapshots

Environment Variables

Environment variables are the standard way to pass secrets to applications in many deployment systems (Heroku, Docker, Kubernetes).

# Setting environment variables
export DATABASE_URL="postgres://admin:password@db.example.com/prod"

# Docker
docker run -e DATABASE_URL="postgres://..." myapp

# Kubernetes (from a Secret object)
env:
  - name: DATABASE_URL
    valueFrom:
      secretKeyRef:
        name: db-credentials
        key: url

Environment variables are acceptable but not ideal:

  • They appear in /proc/<pid>/environ on Linux (readable by root and the process owner)
  • They are included in crash dumps and core files
  • Child processes inherit all environment variables from the parent
  • docker inspect shows environment variables in plaintext
  • Logging frameworks sometimes log the full environment on startup

For production systems handling sensitive data, a secret manager is preferable.

What to Do When a Secret Leaks

Secrets leak. It is not a matter of if, but when. The response must be immediate and systematic.

Step 1: Rotate Immediately

The moment you discover a leaked secret, rotate it. Do not assess impact first. Do not file a ticket. Rotate the secret now, then investigate.

# AWS: rotate access keys
aws iam create-access-key --user-name compromised-user
aws iam delete-access-key --user-name compromised-user \
  --access-key-id AKIAIOSFODNN7EXAMPLE

# Database: change the password
ALTER USER app_user WITH PASSWORD 'new_secure_password_here';

# API keys: revoke and regenerate in the provider's console
# (Stripe, Twilio, SendGrid, etc.)

Step 2: Audit Usage

After rotating, determine whether the secret was exploited.

# Check questions to answer:
- When was the secret committed? (git log)
- When was the repository made public? (GitHub audit log)
- Was the secret accessed? (CloudTrail for AWS, audit logs for other services)
- What resources were accessed with the compromised credentials?
- Were new resources created? (crypto miners, backdoors)
- Were other secrets or data exfiltrated?

# AWS CloudTrail: search for usage of a specific access key
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=AKIAIOSFODNN7EXAMPLE \
  --start-time 2024-01-01 \
  --end-time 2024-01-31

Step 3: Understand the Blast Radius

A single leaked credential can cascade.

Leaked AWS key
  → Access to S3 bucket with customer data
  → Access to RDS database
  → Access to Parameter Store with other secrets
  → Those secrets grant access to third-party APIs
  → Those APIs have access to customer PII

Map out every resource the compromised credential could access. Rotate all downstream secrets if the compromised credential had access to them.

Step 4: Remove from Git History

Deleting the file in a new commit does not remove it from history. Use one of these tools:

# git-filter-repo (recommended, replaces BFG and filter-branch)
git filter-repo --path .env --invert-paths

# BFG Repo Cleaner (simpler for single files)
bfg --delete-files .env
git reflog expire --expire=now --all
git gc --prune=now --aggressive

After rewriting history, force push and notify all collaborators to re-clone. Anyone who pulled the old history still has the secret locally.

Important: rewriting history is disruptive. If the repository has been forked or cloned by others, the secret must be considered permanently compromised regardless. Always rotate first.

Preventing Secret Leaks

Pre-Commit Hooks

Install tools that scan for secrets before they enter the repository.

# git-secrets (by AWS Labs)
git secrets --install
git secrets --register-aws
# Blocks commits containing AWS key patterns

# detect-secrets (by Yelp)
detect-secrets scan > .secrets.baseline
detect-secrets-hook --baseline .secrets.baseline
# Blocks commits containing any detected secrets

# gitleaks
gitleaks protect --staged
# Scans staged changes for secrets

GitHub Secret Scanning

GitHub automatically scans public repositories (and private repositories on Enterprise plans) for known secret formats from over 200 service providers. When a secret is found, GitHub notifies the provider, who may automatically revoke the credential.

Supported providers include:
- AWS (access keys)
- Google Cloud (service account keys, API keys)
- Azure (storage keys, connection strings)
- Stripe (API keys)
- Twilio (API keys)
- SendGrid, Mailgun, Slack, npm, PyPI, and 200+ more

GitHub Push Protection goes further: it blocks the push before the secret enters the repository.

Real-World Incidents

Uber (2016)

Two attackers accessed Uber's private GitHub repository and found AWS credentials. They used those credentials to access an S3 bucket containing names, email addresses, and phone numbers of 57 million Uber users, plus driver's license numbers of 600,000 drivers. Instead of disclosing the breach, Uber paid the attackers $100,000 through the bug bounty program to delete the data and keep quiet. Uber's CISO was later convicted of federal charges for covering up the breach.

Samsung (2022)

Samsung accidentally exposed internal source code in a public GitHub repository, including secret keys, credentials, and internal API endpoints. The Lapsus$ group exploited this exposure as part of a broader attack that resulted in the theft of 190 GB of Samsung source code, including Galaxy device source code and proprietary algorithms.

Codecov (2021)

Attackers modified Codecov's Bash Uploader script to exfiltrate environment variables from CI/CD environments. Every organization using Codecov in CI had their environment variables (often containing secrets) sent to the attackers. The attack persisted for two months before discovery. This demonstrated that secrets in environment variables are vulnerable to supply chain attacks on CI tooling.

Common Pitfalls

  • "It's a private repository." Private repositories become public through acquisitions, open-sourcing, access control mistakes, or contractor access. Treat private repositories as if they will eventually be public.
  • "I'll remove it in the next commit." Git history is permanent. The secret is in the history forever unless you rewrite it, and rewriting history is painful and incomplete.
  • "It's just a development key." Development keys often have the same access as production keys, or the same key is used in both environments. Attackers do not distinguish between development and production credentials.
  • Forgetting .env in .gitignore. Add .env to .gitignore before creating the file, not after. If the file is already tracked, .gitignore will not help.
  • Passing secrets as command-line arguments. Command-line arguments are visible to all users via ps aux. Use environment variables or files instead.
  • Not rotating after a leak. Some teams discover a leaked secret and simply delete it from the code without rotating. The secret is compromised. Deleting it from the code does not un-compromise it.
  • Sharing secrets over Slack or email. These channels are logged, searchable, and backed up. Use a secret manager or a dedicated sharing tool (1Password, Vault).

Key Takeaways

  • Never put secrets in source code. Not in variables, not in config files, not in comments. Never.
  • Exposed credentials are exploited within minutes on public repositories. Automated scanners run continuously.
  • The secret hierarchy: code (never), .env files (avoid committing), environment variables (acceptable), secret managers (best).
  • When a secret leaks: rotate immediately, audit usage, assess blast radius, remove from history. In that order.
  • Use pre-commit hooks (git-secrets, detect-secrets, gitleaks) to catch secrets before they are committed.
  • Enable GitHub secret scanning and push protection on all repositories.
  • Assume every leaked secret was exploited. Rotate first, investigate second.