Scrolls
6 min read
On this page

Fast Feedback Loops

A CI pipeline that takes 30 minutes is not a safety net. It is a productivity killer. By the time it finishes, you have context-switched to something else. When it fails, you have to reload the original problem into your brain, figure out what went wrong, fix it, push again, and wait another 30 minutes. Two failures and you have lost half a day to a change that should have taken an hour.

The goal is simple: know if your code works before you context-switch. That means CI under 5 minutes. Not 10. Not "under 15." Five minutes. That is the threshold where a developer can push, grab coffee, and come back to a green or red result while the problem is still fresh.

Why Speed Matters More Than Coverage

Teams often justify slow CI by pointing to comprehensive test suites. "We test everything" sounds responsible. But a 30-minute pipeline that catches 5% more bugs than a 5-minute pipeline is a terrible trade. Those extra bugs will be caught in code review or QA anyway. The 25 minutes of developer wait time, multiplied by every push, multiplied by every developer, adds up to weeks of lost productivity per quarter.

Fast CI changes behavior. When CI is fast, developers push smaller changes more frequently. Smaller changes are easier to review, easier to debug when they fail, and easier to roll back. When CI is slow, developers batch changes to avoid the wait. Batched changes are harder to review, harder to debug, and riskier to deploy.

Parallel Jobs

The single biggest win for CI speed is parallelization. Most pipelines run steps sequentially by default because that is how the config file reads top to bottom. But linting, unit tests, integration tests, and build steps rarely depend on each other.

# Bad: sequential pipeline (12 minutes total)
steps:
  - lint          # 2 min
  - type-check    # 2 min
  - unit-tests    # 4 min
  - integration   # 4 min

# Good: parallel pipeline (4 minutes total)
jobs:
  lint:           # 2 min (parallel)
  type-check:     # 2 min (parallel)
  unit-tests:     # 4 min (parallel)
  integration:    # 4 min (parallel)

In GitHub Actions, this means separate jobs instead of sequential steps. In GitLab CI, it means stages with parallel jobs. In CircleCI, it means workflows with fan-out. The syntax differs, the principle is identical: if two things do not depend on each other, run them at the same time.

The constraint is your CI provider's concurrency limit. If you are on a free tier with 2 concurrent jobs, parallelizing 8 jobs still bottlenecks at 2. Budget for CI concurrency like you budget for developer tools. The ROI is immediate.

Caching

Every CI run that installs dependencies from scratch is wasting time. Node modules, Python packages, Go modules, Docker layers -- these do not change on most commits. Cache them.

# GitHub Actions example: cache node_modules
- uses: actions/cache@v4
  with:
    path: node_modules
    key: node-${{ hashFiles('package-lock.json') }}
    restore-keys: node-

# The key is based on the lockfile hash.
# If dependencies haven't changed, skip the install entirely.

Effective caching strategies by language:

JavaScript/TypeScript:  Cache node_modules, keyed on lockfile hash
Python:                 Cache .venv or pip cache directory
Go:                     Cache GOMODCACHE and GOCACHE
Rust:                   Cache target/ directory and cargo registry
Docker:                 Use BuildKit cache mounts and layer caching

A common mistake is caching too aggressively. If your cache key is too broad (e.g., just the branch name), you will restore stale dependencies and get mysterious failures. Always include the lockfile or dependency manifest hash in the key.

Test Splitting

If your test suite takes 8 minutes on one machine, split it across 4 machines and it takes 2 minutes. This is not theoretical -- it is basic arithmetic, and every major CI provider supports it.

# CircleCI parallelism example
jobs:
  test:
    parallelism: 4
    steps:
      - run:
          command: |
            circleci tests glob "tests/**/*.test.js" |
            circleci tests split --split-by=timings |
            xargs jest

The key detail is --split-by=timings. Naive splitting (alphabetical, round-robin) creates uneven shards where one machine finishes in 30 seconds and another takes 4 minutes. Timing-based splitting uses historical data to distribute tests evenly.

For pytest, use pytest-split or pytest-xdist. For RSpec, use parallel_tests or Knapsack. For Jest, use --shard. For Go, split by package.

Fail Fast

If linting fails in 30 seconds, there is no reason to wait for an 8-minute test suite to also fail. Configure your pipeline to cancel downstream jobs when an early job fails.

# GitHub Actions: cancel in-progress runs on new pushes
concurrency:
  group: ${{ github.ref }}
  cancel-in-progress: true

Beyond canceling on new pushes, structure your pipeline so cheap checks run first:

Stage 1 (30 seconds):  Lint, format check, type check
Stage 2 (2 minutes):   Unit tests
Stage 3 (4 minutes):   Integration tests, build

If Stage 1 fails, Stage 2 and 3 never start. This is not just faster -- it gives better feedback. "Your code has a syntax error" is more useful than "47 tests failed" when the root cause is a missing semicolon.

Selective Testing

Not every change needs every test. A change to the README should not trigger the full integration suite. A change to a CSS file should not run backend tests.

# Run backend tests only when backend code changes
on:
  push:
    paths:
      - 'src/api/**'
      - 'src/models/**'
      - 'tests/backend/**'

# Run frontend tests only when frontend code changes
on:
  push:
    paths:
      - 'src/components/**'
      - 'src/hooks/**'
      - 'tests/frontend/**'

This requires some discipline in project structure. If your code is a tangled monolith where everything depends on everything, selective testing is risky. But if you have clear module boundaries, it can cut CI time dramatically for most changes.

For monorepos, tools like Turborepo, Nx, or Bazel can determine which packages are affected by a change and only test those. This is more sophisticated than path-based triggers but far more accurate.

Measuring & Maintaining CI Speed

Track your CI times. Not just the p50 -- track the p90 and p99. A pipeline that usually takes 3 minutes but occasionally takes 20 is still a productivity problem.

Metrics to track:
- Median CI duration (target: under 5 minutes)
- p90 CI duration (target: under 8 minutes)
- Flaky test rate (target: under 1%)
- Cache hit rate (target: above 80%)
- Queue wait time (target: under 30 seconds)

When CI starts creeping slower, treat it like tech debt. A 10-second regression per week becomes 8 minutes over a year. Set alerts when median duration exceeds your threshold. Make "CI speed" a first-class metric on your team dashboard.

Flaky tests deserve special mention. A test that fails 5% of the time means 1 in 20 CI runs fails for no reason. Developers learn to just re-run the pipeline, which doubles the effective CI time. Quarantine flaky tests immediately. Fix them or delete them. A flaky test is worse than no test because it erodes trust in the entire suite.

Common Pitfalls

  • Optimizing the wrong thing. Your CI is slow because of a 6-minute integration test, but you spend a week shaving 10 seconds off the lint step. Profile first.
  • Not caching across branches. Many teams cache only on the same branch, so every new feature branch starts cold. Use fallback keys that restore from the main branch cache.
  • Running full suites on draft PRs. If a PR is marked as draft or WIP, run only the fast checks. Save the full suite for when it is ready for review.
  • Ignoring queue time. Your pipeline takes 3 minutes to run but 5 minutes to start because all runners are busy. Queue time is invisible in most dashboards but very visible to developers.
  • Over-testing in CI. End-to-end browser tests are slow and flaky. Run a small smoke suite in CI and save the full E2E suite for a nightly or pre-deploy pipeline.

Key Takeaways

  • CI under 5 minutes is the target. It keeps developers in flow and encourages small, frequent pushes.
  • Parallelize everything that does not have a dependency. This is the single biggest lever.
  • Cache dependencies aggressively, keyed on lockfile hashes. Never install from scratch when nothing changed.
  • Split large test suites across machines using timing-based distribution.
  • Fail fast: run cheap checks first, cancel downstream jobs on failure, cancel stale runs on new pushes.
  • Track CI duration as a first-class metric. Regressions accumulate silently if you do not measure.
  • Quarantine flaky tests immediately. They erode trust and waste everyone's time.