Scrolls
6 min read
On this page

Deploy Confidence

If deploying is scary, you deploy less often. If you deploy less often, each deploy contains more changes. If each deploy contains more changes, it is harder to debug when something breaks. If it is harder to debug, deploying gets scarier. This is the fear cycle, and it is one of the most common ways teams slow themselves down.

The fix is not "be braver." The fix is to make deploying safe. When you can deploy with confidence -- knowing you can detect problems quickly and roll back in seconds -- you deploy more often. When you deploy more often, each deploy is smaller. When deploys are smaller, they are safer. The virtuous cycle replaces the fear cycle.

Feature Flags

Feature flags decouple deployment from release. You deploy code to production without exposing it to users. When you are ready, you flip the flag. If something goes wrong, you flip it back. No rollback, no redeploy, no downtime.

# Simple feature flag check
if feature_flags.enabled("new-checkout-flow", user_id=user.id):
    return new_checkout_flow(cart)
else:
    return legacy_checkout_flow(cart)

This is not just for big features. Use flags for:

- Database migrations:     Read from new table, fall back to old
- API changes:             Route to new endpoint, fall back to old
- UI experiments:          Show new design to 10% of users
- Performance changes:     Enable new caching strategy gradually
- Kill switches:           Disable a feature instantly if it causes problems

The key insight is that a feature flag turns a binary deploy (works or doesn't) into a gradual rollout with an instant off switch. Instead of "deploy and pray," you get "deploy, observe, and adjust."

Flag Lifecycle

Feature flags are not permanent. A flag that lives forever is tech debt. Every flag has a lifecycle:

1. Create:     Add flag, default off
2. Develop:    Use flag to gate incomplete work
3. Test:       Enable for internal users, then beta users
4. Roll out:   Enable for 10%, 50%, 100% of users
5. Clean up:   Remove flag and dead code path within 2 weeks

The cleanup step is critical. A codebase with 200 active feature flags is unmaintainable. Set a policy: every flag gets a cleanup ticket when it is created. If a flag has been at 100% for more than 2 weeks, it should be removed.

Flag Management

For small teams, a config file or environment variable works fine. For larger teams, use a proper flag management system.

Hosted services:       LaunchDarkly, Split, Flagsmith, Statsig
Self-hosted:           Unleash, Flipt, OpenFeature
Simple/DIY:            Environment variables, database-backed config

The decision point is how often you need to change flags without deploying. If the answer is "rarely," environment variables are sufficient. If the answer is "multiple times per day" or "different values per user," use a proper service.

Canary Deploys

A canary deploy sends new code to a small percentage of traffic before rolling it out to everyone. If the canary is healthy, you proceed. If it is not, you pull it back, and 95% of your users never saw the problem.

Canary deploy progression:
1. Deploy to 1 canary instance (5% of traffic)
2. Monitor error rate, latency, CPU for 10 minutes
3. If healthy: expand to 25%, monitor again
4. If healthy: expand to 50%, monitor again
5. If healthy: expand to 100%
6. If unhealthy at any stage: route all traffic back to old version

The monitoring part is what makes canary deploys work. Without automated health checks, a canary deploy is just a slow deploy. You need:

Key metrics to monitor during canary:
- Error rate (HTTP 5xx, exceptions)
- Latency (p50, p95, p99)
- Resource usage (CPU, memory)
- Business metrics (conversion rate, checkout completions)
- Saturation (queue depth, connection pool usage)

Compare canary metrics to the baseline (the old version running alongside it). If the canary's error rate is 2x the baseline, something is wrong, even if the absolute error rate looks acceptable.

Automated Canary Analysis

Manual canary monitoring does not scale. If someone has to watch dashboards for 30 minutes per deploy, deploys become a chore. Automate the analysis.

# Argo Rollouts canary strategy (Kubernetes)
spec:
  strategy:
    canary:
      steps:
        - setWeight: 5
        - pause: {duration: 10m}
        - analysis:
            templates:
              - templateName: error-rate-check
        - setWeight: 25
        - pause: {duration: 10m}
        - analysis:
            templates:
              - templateName: error-rate-check
        - setWeight: 100

Tools like Argo Rollouts, Flagger, or Spinnaker can automate the entire progression: deploy the canary, wait, check metrics, proceed or roll back. The developer pushes code and walks away. The system handles the rest.

Rollback in One Command

The most important property of your deploy system is not how fast it deploys. It is how fast it rolls back. When something goes wrong in production, the time between "we detected a problem" and "the problem is gone" is all that matters.

Rollback should be:

  • One command (or one button click)
  • Under 60 seconds to take effect
  • Tested regularly, not just theoretically possible
# Good: explicit rollback commands
kubectl rollout undo deployment/api-server
heroku rollback
aws deploy stop-deployment --deployment-id <id> --auto-rollback-enabled

# Bad: rollback means "revert the commit, push, wait for CI, redeploy"
# That is a 20-minute rollback, which is not a rollback. It is a new deploy.

If your rollback procedure involves running CI, it is too slow. Rollback should use a previously built and tested artifact. You are not building new code -- you are switching back to the old code that was already proven to work.

Immutable Artifacts

The key to fast rollback is immutable deploy artifacts. Every deploy creates a versioned artifact (Docker image, compiled binary, bundled assets) that is stored and can be re-deployed at any time.

Deploy flow with immutable artifacts:
1. CI builds artifact: api-server:v1.2.3
2. Artifact is stored in registry
3. Deploy points traffic to api-server:v1.2.3
4. Problem detected
5. Rollback points traffic to api-server:v1.2.2 (already built, already stored)

This is why container-based deployments have such fast rollbacks. The old image is still in the registry. Rolling back means changing which image tag the deployment points to. No build step, no CI, no waiting.

Blue-Green Deploys

Blue-green deploys maintain two identical production environments. One (blue) serves live traffic. The other (green) receives the new deploy. Once the green environment is verified, traffic switches from blue to green. If something goes wrong, switch back to blue.

Blue-green deploy:
1. Blue serves all traffic (current version)
2. Deploy new version to green
3. Run smoke tests against green
4. Switch load balancer from blue to green
5. Green now serves all traffic (new version)
6. Blue remains ready as instant rollback target

The advantage over canary deploys is simplicity: there is no gradual rollout to manage. The disadvantage is cost: you need two full production environments. For many teams, the cost is worth the simplicity.

Blue-green works especially well for database-backed applications where you need to verify that the new code works with the current data before exposing it to users. Run the smoke tests against the real database (not a copy) to catch data-dependent bugs.

Building the Pipeline

A deploy pipeline that inspires confidence has these stages:

Push to main
  -> CI runs (under 5 minutes)
  -> Artifact is built and stored
  -> Deploy to staging (automatic)
  -> Smoke tests on staging (automatic)
  -> Deploy to production canary (automatic or one-click)
  -> Canary analysis (automatic, 10-15 minutes)
  -> Full production rollout (automatic if canary passes)
  -> Post-deploy verification (automatic)

Every stage has an automatic rollback path. If smoke tests fail on staging, the deploy stops. If the canary shows elevated errors, it rolls back. If post-deploy verification catches a problem, one command rolls back.

The goal is to make the happy path fully automatic. A developer merges to main and goes to lunch. When they come back, their code is in production. If something went wrong, the system already rolled it back and notified them.

Common Pitfalls

  • Feature flags without cleanup. Every flag needs an expiration. A codebase with hundreds of stale flags is harder to reason about than one with no flags at all.
  • Canary deploys without automated analysis. If you rely on someone watching dashboards, canaries become a bottleneck. Automate the health checks.
  • Rollback that requires a new build. If rolling back means reverting a commit and running CI again, you do not have rollback. You have a slow forward-fix process.
  • No staging environment. Deploying directly to production canary without a staging step means every canary is an experiment on real users. Staging catches the obvious problems cheaply.
  • Deploying on Fridays without rollback confidence. The "no Friday deploys" rule exists because teams do not trust their rollback. Fix the rollback and you can deploy any day.
  • Blue-green without database compatibility. If the new version requires a database migration that breaks the old version, you cannot switch back to blue. Always make migrations backward-compatible.

Key Takeaways

  • The fear cycle (scary deploys lead to infrequent deploys lead to bigger deploys lead to scarier deploys) is the root cause of deploy anxiety. Break it by making deploys safe, not by being brave.
  • Feature flags decouple deployment from release. Deploy code without exposing it. Flip a flag to release. Flip it back if something goes wrong.
  • Canary deploys catch problems with real traffic while limiting the blast radius. Automate the analysis so it does not become a bottleneck.
  • Rollback must be one command and under 60 seconds. If it requires a new build or CI run, it is too slow.
  • Immutable, versioned deploy artifacts make fast rollback possible. Store every build. Rollback means pointing to the previous artifact.
  • The goal is a fully automatic deploy pipeline where merging to main results in production deployment, with automatic rollback at every stage if something goes wrong.