Scrolls
4 min read
On this page

Automate the Boring Stuff

The Third Time Rule

The first time you do something, you just do it. The second time, you notice you're repeating yourself. The third time, you automate it.

This isn't a rigid law — some tasks aren't worth automating even after ten repetitions, and some are worth automating the first time because they're error-prone. But the third-time heuristic is a good default because it balances the cost of automation against the cost of manual repetition.

The engineers who are most productive over a career are not the ones who type the fastest or know the most algorithms. They're the ones who systematically eliminate manual toil. Every hour spent on automation returns tenfold because the automation keeps saving time long after you've forgotten you wrote it.

Identifying Automation Candidates

Not everything should be automated. The best candidates share these characteristics:

High frequency:    You do it daily or weekly
Low complexity:    The steps are mechanical, not creative
Error-prone:       Manual execution occasionally goes wrong
Time-consuming:    Each instance takes more than a minute
Well-defined:      The inputs and outputs are predictable

Common automation targets

File operations. Renaming batches of files, converting between formats, organizing directories, cleaning up build artifacts.

# Rename all .jpeg files to .jpg
for f in *.jpeg; do
    mv "$f" "${f%.jpeg}.jpg"
done

# Organize files by date
for f in *.log; do
    date_dir=$(date -r "$f" +%Y-%m)
    mkdir -p "$date_dir"
    mv "$f" "$date_dir/"
done

Data transformation. Converting CSV to JSON, extracting fields from logs, reformatting configuration files, generating reports from raw data.

# Extract unique error types from a log file and count them
grep "ERROR" app.log | awk -F'[][]' '{print $2}' | sort | uniq -c | sort -rn

# Convert a list of values to a SQL IN clause
cat user_ids.txt | tr '\n' ',' | sed 's/,$//' | xargs -I{} echo "WHERE id IN ({})"

Environment setup. Installing dependencies, configuring services, setting environment variables, starting development servers.

#!/usr/bin/env bash
set -euo pipefail

# dev-setup.sh: Get a new developer from zero to running in one command

echo "Installing dependencies..."
npm install

echo "Setting up local database..."
docker compose up -d postgres
sleep 2
npm run db:migrate
npm run db:seed

echo "Creating local environment file..."
cp .env.example .env.local
echo "DATABASE_URL=postgres://localhost:5432/myapp_dev" >> .env.local

echo "Starting development server..."
npm run dev

Deployment steps. Building, tagging, pushing images, running migrations, verifying health checks.

#!/usr/bin/env bash
set -euo pipefail

# deploy.sh: Deploy to staging

VERSION=$(git describe --tags --always)
IMAGE="myapp:$VERSION"

echo "Building $IMAGE..."
docker build -t "$IMAGE" .

echo "Pushing to registry..."
docker tag "$IMAGE" registry.example.com/"$IMAGE"
docker push registry.example.com/"$IMAGE"

echo "Deploying to staging..."
kubectl set image deployment/myapp myapp=registry.example.com/"$IMAGE" -n staging

echo "Waiting for rollout..."
kubectl rollout status deployment/myapp -n staging --timeout=120s

echo "Deployed $VERSION to staging."

Shell Scripts

For most automation tasks, a bash script is the right tool. It's available everywhere, handles file operations natively, and composes well with existing CLI tools.

Script essentials

Every script should start with:

#!/usr/bin/env bash
set -euo pipefail

set -e: Exit immediately if a command fails. Without this, the script keeps running after errors, potentially causing cascading damage.

set -u: Treat unset variables as errors. Without this, $UNDEFINED_VAR silently expands to an empty string, which causes bugs like rm -rf /$UNDEFINED_VAR deleting your root filesystem.

set -o pipefail: A pipeline fails if any command in it fails. Without this, broken_command | sort succeeds because sort succeeded, even though broken_command failed.

Argument handling

#!/usr/bin/env bash
set -euo pipefail

usage() {
    echo "Usage: $0 <environment> [--dry-run]"
    echo "  environment: staging or production"
    exit 1
}

if [ $# -lt 1 ]; then
    usage
fi

ENV="$1"
DRY_RUN=false

if [ "${2:-}" = "--dry-run" ]; then
    DRY_RUN=true
fi

if [ "$ENV" != "staging" ] && [ "$ENV" != "production" ]; then
    echo "Error: environment must be 'staging' or 'production'"
    exit 1
fi

if [ "$DRY_RUN" = true ]; then
    echo "[DRY RUN] Would deploy to $ENV"
else
    echo "Deploying to $ENV..."
fi

Logging and output

# Use functions for consistent output
info() { echo "[INFO] $*"; }
warn() { echo "[WARN] $*" >&2; }
error() { echo "[ERROR] $*" >&2; exit 1; }

info "Starting deployment..."
warn "This will affect production traffic"
error "Could not connect to database"  # This exits

Cleanup traps

# Clean up temporary files even if the script fails
TMPDIR=$(mktemp -d)
trap "rm -rf $TMPDIR" EXIT

# Do work with temporary files
cp important_data.csv "$TMPDIR/working.csv"
# ... process ...

Makefiles

Makefiles are underrated for project automation. Despite being associated with C compilation, they're excellent as a universal task runner for any project.

Why Makefiles

- Available on every Unix system without installation
- Self-documenting (make with no target can list available targets)
- Dependency tracking (only run what's needed)
- Language-agnostic (works for any project type)
- Your team already knows how to use them (or can learn in 5 minutes)

A practical Makefile

.PHONY: help dev test lint build clean deploy

help: ## Show this help
	@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | \
		awk 'BEGIN {FS = ":.*?## "}; {printf "  %-15s %s\n", $$1, $$2}'

dev: ## Start development environment
	docker compose up -d
	npm run dev

test: ## Run tests
	npm test

test-watch: ## Run tests in watch mode
	npm test -- --watch

lint: ## Run linter
	npm run lint

lint-fix: ## Run linter with auto-fix
	npm run lint -- --fix

build: ## Build for production
	npm run build

clean: ## Remove build artifacts
	rm -rf dist node_modules/.cache

deploy-staging: build ## Deploy to staging
	./scripts/deploy.sh staging

deploy-prod: build ## Deploy to production (requires confirmation)
	@read -p "Deploy to production? [y/N] " confirm; \
	if [ "$$confirm" = "y" ]; then ./scripts/deploy.sh production; fi

db-migrate: ## Run database migrations
	npm run db:migrate

db-seed: ## Seed database with test data
	npm run db:seed

db-reset: db-migrate db-seed ## Reset database (migrate + seed)

The help target auto-generates documentation from the ## comments. Run make or make help and you get a clean list of available commands.

Makefile vs npm scripts vs task runners

npm scripts (package.json):  Good for JavaScript projects. Limited
                              to what npm supports. No dependency
                              tracking between tasks.

Makefile:                     Universal. Works for any language.
                              Dependency tracking. Available everywhere.
                              Syntax is awkward for complex logic.

Just (justfile):              Modern alternative to Make. Nicer syntax.
                              No dependency tracking. Requires installation.

Task (taskfile.yml):          YAML-based. Good for complex workflows.
                              Requires installation.

For most projects, a Makefile with npm/pip/cargo scripts underneath is the sweet spot. The Makefile provides the uniform interface; the language-specific tools do the actual work.

Automating Repetitive Development Tasks

Pre-commit hooks

Instead of remembering to lint and format before committing, let git do it:

# Install pre-commit hooks
# .pre-commit-config.yaml
repos:
  - repo: local
    hooks:
      - id: lint
        name: lint
        entry: npm run lint
        language: system
        pass_filenames: false
      - id: format-check
        name: format check
        entry: npx prettier --check
        language: system
        types: [javascript, typescript]

Database management scripts

Wrap common database operations (reset, dump, restore) in a single script with subcommands. This eliminates the need to remember individual commands and ensures they're run with the right flags every time.

Log analysis one-liners

Save these as aliases for when you need them:

# Find the slowest API requests from today's logs
grep "$(date +%Y-%m-%d)" api.log | grep "duration_ms" | \
    sort -t= -k2 -rn | head -20

# Count requests per endpoint per hour
awk '{print $4, $7}' access.log | \
    sed 's/\[//' | cut -d: -f1-2 | sort | uniq -c | sort -rn

Common Pitfalls

Automating too early. If you've done something once, you don't know enough about the variations to automate it well. Wait until you've done it at least a few times and understand the edge cases.

Automating too late. If you've manually deployed 50 times and made a mistake on deploy 47, you should have automated this 45 deploys ago. The third-time rule exists to prevent this.

Over-engineering automation. A 200-line bash script with comprehensive error handling, logging, and retry logic for a task you do weekly is over-engineering. A 15-line script that handles the happy path is usually enough.

Not testing automation scripts. Run your scripts in a safe environment before pointing them at production data. A typo in a cleanup script can delete things that should not be deleted.

Writing automation that only you understand. If your deployment script is 500 lines of undocumented bash, it's a liability, not an asset. Document what your scripts do, or keep them simple enough that documentation is unnecessary.

Ignoring the team. Automating your personal workflow is good. Automating team processes requires buy-in. If you write a deployment script and nobody uses it because they don't trust it, you've wasted your time.

Key Takeaways

Automate tasks that are frequent, mechanical, error-prone, and well-defined. The third-time rule is a useful heuristic for when to start.

Shell scripts with set -euo pipefail are the right tool for most automation tasks. They're portable, composable, and require no additional dependencies.

Makefiles provide a universal, self-documenting task runner interface. Use them as the entry point for project commands regardless of language.

Start simple. A 10-line script that solves 90% of the problem is better than a 200-line script that solves 100%. You can always add complexity when you need it.

Automation compounds. Every script you write saves time on every future execution. The engineers who consistently automate toil end up spending their time on problems that actually require human judgment.