Secrets in CI/CD
CI/CD pipelines are high-value targets. They have access to production credentials, deployment keys, cloud provider accounts, and code signing certificates. A compromised pipeline can deploy malicious code to production, exfiltrate secrets, or modify infrastructure. Yet CI/CD environments are often treated with less rigor than production servers. Secrets are stored in plaintext environment variables, logged in build output, and protected by minimal access controls. The principle is straightforward: treat CI/CD like production.
GitHub Encrypted Secrets
GitHub Actions provides encrypted secrets that are stored encrypted at rest and made available to workflows as environment variables. They are masked in logs by default.
Setting Secrets
# Via GitHub CLI
gh secret set DATABASE_URL --body "postgres://user:pass@host/db"
gh secret set AWS_ACCESS_KEY_ID --body "AKIAIOSFODNN7EXAMPLE"
gh secret set AWS_SECRET_ACCESS_KEY --body "wJalrXUtnFEMI..."
# Organization-level secrets (shared across repos)
gh secret set DEPLOY_KEY --org my-org --visibility selected \
--repos repo1,repo2
# Environment-level secrets (only available in specific environments)
gh secret set PROD_DATABASE_URL --env production
Using Secrets in Workflows
# .github/workflows/deploy.yml
name: Deploy
on:
push:
branches: [main]
jobs:
deploy:
runs-on: ubuntu-latest
environment: production # Gates deployment, loads env secrets
steps:
- uses: actions/checkout@v4
- name: Deploy to production
env:
DATABASE_URL: ${{ secrets.PROD_DATABASE_URL }}
API_KEY: ${{ secrets.API_KEY }}
run: |
./deploy.sh
Secret Scoping
Scope Visibility Use Case
────────────────────────────────────────────────────────────────
Repository Single repository Repo-specific API keys
Environment Specific environment Production vs staging
within a repo credentials
Organization All or selected repos Shared deployment keys,
in an organization registry credentials
Environment secrets are the most secure option. They can require manual approval before a workflow can access them, limiting who can trigger production deployments.
# GitHub environment protection rules
- Required reviewers: 2 approvals before deployment
- Wait timer: 15-minute delay before deployment proceeds
- Branch restrictions: only main branch can deploy to production
Never Echo Secrets in Logs
The most common CI/CD secret leak is accidentally printing a secret in build logs. Build logs are often accessible to anyone with repository access, stored indefinitely, and forwarded to log aggregation systems.
# WRONG: secret will appear in logs
- name: Debug
run: echo "The API key is ${{ secrets.API_KEY }}"
# WRONG: curl verbose mode shows headers including auth tokens
- name: Call API
run: curl -v -H "Authorization: Bearer ${{ secrets.API_TOKEN }}" https://api.example.com
# WRONG: environment dump exposes everything
- name: Debug environment
run: env | sort
# WRONG: set -x traces every command with expanded variables
- name: Deploy
run: |
set -x
export TOKEN=${{ secrets.DEPLOY_TOKEN }}
./deploy.sh --token=$TOKEN
GitHub Actions automatically masks secrets that match known values, but this masking is imperfect. If a secret is transformed (base64-encoded, URL-encoded, split across lines), the masking fails.
# Masking bypass examples (the secret leaks):
- echo "${{ secrets.API_KEY }}" | base64 # Encoded form is not masked
- echo "${{ secrets.API_KEY }}" | rev # Reversed form is not masked
Mask Custom Values
# Manually mask a value in GitHub Actions
- name: Generate token
id: gen
run: |
TOKEN=$(./generate-token.sh)
echo "::add-mask::$TOKEN"
echo "token=$TOKEN" >> "$GITHUB_OUTPUT"
- name: Use token
run: ./deploy.sh --token=${{ steps.gen.outputs.token }}
# TOKEN value is now masked in all subsequent log output
OIDC Federation: No Long-Lived Secrets
The best secret is the one that does not exist. OIDC (OpenID Connect) federation allows CI/CD pipelines to authenticate to cloud providers without any stored secrets. The CI platform issues a short-lived JWT token, and the cloud provider exchanges it for temporary credentials.
How OIDC Federation Works
1. GitHub Actions workflow starts
2. GitHub issues a signed OIDC token (JWT) for the workflow
3. Workflow presents the JWT to AWS STS (or GCP/Azure equivalent)
4. Cloud provider verifies the JWT signature and claims
5. Cloud provider issues temporary credentials (15 min - 1 hour)
6. Workflow uses temporary credentials to access cloud resources
7. Credentials expire automatically -- nothing to rotate or revoke
GitHub OIDC to AWS
# Step 1: Create an OIDC identity provider in AWS (one-time setup)
aws iam create-open-id-connect-provider \
--url https://token.actions.githubusercontent.com \
--client-id-list sts.amazonaws.com \
--thumbprint-list 6938fd4d98bab03faadb97b34396831e3780aea1
# Step 2: Create an IAM role with a trust policy
# trust-policy.json
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::123456789:oidc-provider/token.actions.githubusercontent.com"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
},
"StringLike": {
"token.actions.githubusercontent.com:sub": "repo:my-org/my-repo:ref:refs/heads/main"
}
}
}]
}
# Step 3: Use in GitHub Actions workflow
jobs:
deploy:
runs-on: ubuntu-latest
permissions:
id-token: write # Required for OIDC
contents: read
steps:
- uses: actions/checkout@v4
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::123456789:role/github-actions-deploy
aws-region: us-east-1
- name: Deploy
run: aws s3 sync ./dist s3://my-bucket
Key security properties of OIDC federation:
- No stored secrets. There are no AWS access keys in GitHub. Nothing to rotate, nothing to leak.
- Scoped by repository and branch. The trust policy restricts which repositories and branches can assume the role. A workflow in a fork or a different branch cannot assume the role.
- Temporary credentials. The STS credentials expire in 1 hour by default. Even if intercepted, they are short-lived.
- Auditable. Every role assumption appears in CloudTrail with the GitHub repository and workflow as the identity.
OIDC for Other Providers
Provider OIDC Support
──────────────────────────────────────────────────────
AWS STS AssumeRoleWithWebIdentity
Google Cloud Workload Identity Federation
Azure Federated Identity Credentials
HashiCorp Vault JWT/OIDC auth method
OIDC federation should be the default for all CI/CD-to-cloud authentication. If you are storing cloud provider credentials as CI secrets, you should migrate to OIDC.
Secret Scanning in CI
Even with good practices, secrets can accidentally appear in code, test fixtures, or build artifacts. Run secret scanning as part of the CI pipeline to catch leaks before they reach production.
# GitHub Actions: run gitleaks on every pull request
name: Secret Scan
on: [pull_request]
jobs:
scan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0 # Full history for scanning
- name: Run gitleaks
uses: gitleaks/gitleaks-action@v2
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
# Alternative: trufflehog
- name: Run trufflehog
uses: trufflesecurity/trufflehog@main
with:
extra_args: --only-verified
What to Scan
Scan Target Why
──────────────────────────────────────────────────────────────
Source code (git diff) Catch new secrets in pull requests
Full git history Find previously committed secrets
Docker images Secrets baked into image layers
Build artifacts Secrets in compiled output or bundles
CI configuration Secrets in workflow files or scripts
Infrastructure as code Hardcoded credentials in Terraform,
CloudFormation, Kubernetes manifests
CI/CD Is a High-Value Target
CI/CD systems have privileged access by necessity. They deploy code, modify infrastructure, and access production data. This makes them attractive targets.
The Attack Surface
Attack Vector Impact
──────────────────────────────────────────────────────────────
Compromised dependency Malicious code runs in pipeline
with access to all secrets
Malicious pull request Workflow triggered by PR may
access secrets (depends on config)
Stolen CI credentials Attacker can trigger deployments,
access secrets, modify infrastructure
Supply chain attack on CI Compromised CI platform exposes
platform all customers (Codecov, CircleCI)
Modified workflow files Attacker adds steps to exfiltrate
secrets or deploy backdoors
Hardening CI/CD
# Security checklist for CI/CD
1. Use OIDC federation instead of stored cloud credentials
2. Restrict secret access to specific environments and branches
3. Require approvals for production deployments
4. Pin action versions to specific SHA hashes, not tags
5. Do not allow workflows to run on PRs from forks with secret access
6. Audit workflow changes (who modified .github/workflows/)
7. Use read-only tokens where possible (GITHUB_TOKEN permissions)
8. Scan for secrets in every PR
9. Limit CI runner network access (no unrestricted outbound)
10. Monitor for unusual CI activity (unexpected deployments, bulk secret reads)
Pin Actions to SHA
# BAD: tag can be moved to point to malicious code
- uses: actions/checkout@v4
# GOOD: SHA is immutable
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
A compromised action maintainer (or a compromised maintainer account) can move a tag to point to malicious code. Pinning to a SHA ensures you always run the same code. Use Dependabot or Renovate to update pinned SHAs automatically.
Real-World CI/CD Breaches
Codecov (2021)
Attackers modified Codecov's Bash Uploader script, adding a line that exfiltrated all environment variables to an attacker-controlled server. The modified script ran inside thousands of CI/CD pipelines for two months before discovery. Affected organizations included HashiCorp, Twitch, and dozens of others. Every environment variable in every CI pipeline that used Codecov was compromised.
CircleCI (2023)
An engineer's laptop was compromised through malware, giving attackers access to CircleCI's internal systems. The attacker accessed customer environment variables and secrets stored in CircleCI. CircleCI advised all customers to rotate all secrets stored in CircleCI immediately. Organizations using OIDC federation for cloud access and a secret manager for other credentials had significantly less exposure.
SolarWinds Build System (2020)
Attackers compromised SolarWinds' build system and injected malicious code into the Orion software during the build process. The compromised build produced trojanized updates that were distributed to 18,000 organizations, including US government agencies. The build system had been a target because it had access to code signing certificates and the distribution infrastructure.
Common Pitfalls
- Storing long-lived cloud credentials in CI secrets. Use OIDC federation. There should be no AWS access keys, GCP service account keys, or Azure credentials stored in your CI platform.
- Using
set -xin CI scripts. This traces every command with expanded variables, printing secrets to the log. Remove debug flags before merging. - Running workflows on fork PRs with secret access. By default, GitHub Actions does not provide secrets to workflows triggered by PRs from forks. Do not override this behavior.
- Not pinning action versions. Using
@v4instead of a specific SHA means a compromised upstream action can inject code into your pipeline. - Treating CI logs as private. Anyone with repository access can typically read CI logs. Assume logs are semi-public and never let secrets appear in them.
- No separation between environments. Production and staging should use different secrets, different IAM roles, and different deployment pipelines. A compromised staging pipeline should not grant access to production.
- Ignoring CI platform security advisories. When your CI platform discloses a breach (Codecov, CircleCI), rotate all secrets immediately. Do not wait for confirmation that your specific secrets were accessed.
Key Takeaways
- CI/CD pipelines are high-value targets with privileged access to production infrastructure. Secure them with the same rigor as production servers.
- Use GitHub encrypted secrets with environment scoping and approval requirements for production deployments.
- Never echo, log, or print secrets in CI output. Masking is imperfect -- do not rely on it as the only defense.
- OIDC federation eliminates the need for stored cloud credentials in CI. Use it for all CI-to-cloud authentication.
- Run secret scanning (gitleaks, trufflehog) on every pull request to catch accidental credential commits.
- Pin GitHub Actions to specific commit SHAs, not version tags. Tags can be moved maliciously.
- When a CI platform discloses a security incident, rotate all secrets stored in that platform immediately.