Pipeline Design

A CI/CD pipeline is the automated path your code takes from commit to production. A well-designed pipeline gives you fast feedback, catches problems early, and becomes the single source of truth for whether code is ready to ship.

The Canonical Pipeline

The stages that appear in nearly every production pipeline follow this order:

lint -> test -> build -> deploy

Each stage answers a specific question:

Lint: Does the code follow standards?
Test: Does the code behave correctly?
Build: Does the code compile and package?
Deploy: Can the code run in the target environment?

This ordering is deliberate. Linting is cheap and fast. If your code has a syntax error or style violation, there is no reason to spend minutes running a full test suite to find out.

A Real-World Example

stages:
  - lint
  - test
  - build
  - deploy-staging
  - deploy-production

lint:
  stage: lint
  script:
    - npm run lint
    - npm run typecheck
  timeout: 2m

unit-tests:
  stage: test
  script:
    - npm run test:unit -- --coverage
  timeout: 5m

integration-tests:
  stage: test
  script:
    - npm run test:integration
  timeout: 8m

build:
  stage: build
  script:
    - npm run build
    - docker build -t myapp:$CI_COMMIT_SHA .
  artifacts:
    paths:
      - dist/
    expire_in: 1 hour

deploy-staging:
  stage: deploy-staging
  script:
    - ./deploy.sh staging $CI_COMMIT_SHA
  environment:
    name: staging

deploy-production:
  stage: deploy-production
  script:
    - ./deploy.sh production $CI_COMMIT_SHA
  environment:
    name: production
  when: manual

Fail Fast

The fail-fast principle is simple: run the cheapest checks first. If linting takes 15 seconds and your integration tests take 8 minutes, you do not want developers waiting 8 minutes to discover a missing semicolon.

Ordering by cost:

Static analysis and linting (seconds) -- syntax errors, formatting, type checks
Unit tests (seconds to low minutes) -- isolated function behavior
Integration tests (minutes) -- service interactions, database queries
End-to-end tests (many minutes) -- full user workflows
Build and package (minutes) -- container images, artifacts
Deploy (minutes) -- ship to an environment

If step 1 fails, steps 2 through 6 never run. That is the point.

Fast Feedback

A good pipeline returns results in under 10 minutes for the common case. Once a pipeline exceeds 15 minutes, developers stop waiting for it and start stacking pull requests. Once it exceeds 30 minutes, they stop trusting it entirely.

Strategies to keep pipelines fast:

Parallel Stages

Stages that do not depend on each other should run simultaneously. Unit tests and integration tests can often run in parallel if they share the same base image.

test:
  parallel:
    matrix:
      - SUITE: [unit, integration, e2e]
  script:
    - npm run test:$SUITE

Caching Dependencies

Downloading dependencies on every run is wasteful. Cache them between pipeline runs.

cache:
  key:
    files:
      - package-lock.json
  paths:
    - node_modules/

For Docker builds, layer caching prevents rebuilding unchanged layers:

docker build --cache-from myapp:latest -t myapp:$SHA .

Selective Testing

Not every commit needs every test. If only documentation changed, skip the test suite. If only the frontend changed, skip backend tests.

unit-tests:
  rules:
    - changes:
        - "src/**/*"
        - "package.json"

Artifacts Between Stages

Build once, deploy many times. The build stage produces an artifact -- a Docker image, a compiled binary, a zip file -- and subsequent stages use that exact artifact. You never rebuild for staging and again for production. That defeats the purpose.

build:
  stage: build
  script:
    - docker build -t myapp:$CI_COMMIT_SHA .
    - docker push registry.example.com/myapp:$CI_COMMIT_SHA
  artifacts:
    reports:
      dotenv: build.env

deploy-staging:
  stage: deploy-staging
  needs: [build]
  script:
    - kubectl set image deployment/myapp myapp=registry.example.com/myapp:$CI_COMMIT_SHA

The commit SHA tags the artifact. Every deployment is traceable back to exactly one commit.

The Pipeline as Source of Truth

The pipeline should be the definitive answer to "is this code ready?" If the pipeline is green, the code can ship. If it is red, it cannot. No exceptions, no "well it failed but it's probably fine."

This means:

No skipping stages. If a stage is in the pipeline, it runs.
No manual workarounds. If you regularly bypass the pipeline, the pipeline is wrong -- fix it.
Branch protection. Merging to main requires a green pipeline. No human override.
Consistent environments. The pipeline runs in the same environment every time. No "works on my CI machine."

# Branch protection in practice
merge_request:
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
  allow_failure: false

# Profile which stages take the longest
grep -E "duration|stage" pipeline-report.json | sort -t: -k2 -n

Common Pitfalls

Not caching dependencies. Every run downloads 500MB of packages. Cache them.
Running all tests on every change. Use change-based rules to skip irrelevant suites.
Deploying a different artifact than you tested. Build once, use that artifact everywhere.
Allowing pipeline failures on protected branches. If main can merge with a red pipeline, the pipeline is decorative.
Hardcoding secrets in pipeline files. Use your CI system's secret management. Never commit credentials.
Ignoring pipeline duration. Measure it. Set a time budget. Treat regressions in pipeline speed like regressions in product performance.
No timeout on stages. A hung test can block a pipeline for hours. Set explicit timeouts.

Key Takeaways

Order stages by cost: lint, test, build, deploy. The cheapest checks run first.
Target under 10 minutes for feedback on the common path.
Parallelize stages that do not depend on each other.
Cache dependencies and Docker layers aggressively.
Build artifacts once and promote them through environments.
The pipeline is the single source of truth for code readiness. If it is green, ship it. If it is red, fix it.
Treat pipeline maintenance as seriously as application code maintenance.