Emergent Behavior & Side Effects

Sometimes a system behaves in ways that none of its individual parts would predict. Traffic jams appear without any single cause. A simple social media feature changes human behavior on a massive scale. A collection of reliable components creates an unreliable system. This is emergence — and it is one of the hardest things to anticipate.

The Everyday Version

Traffic Jams

Nobody decides to create a traffic jam. Yet they appear reliably.

What happens:
1. A driver taps their brakes slightly (maybe a bird flew by)
2. The driver behind them brakes a little harder (caution)
3. The driver behind them brakes even harder
4. Five cars back, someone stops completely
5. Twenty cars back, traffic is at a standstill
6. The original driver is long gone, driving normally

No single driver caused the jam.
No single driver can fix it.
The jam is an emergent property of many drivers
interacting through simple rules:
- Follow the car ahead
- Brake when they brake
- Brake a little harder to be safe

These simple rules, followed by thousands of individuals, produce complex behavior that nobody intended.

Viral Content

Why does a video go viral? It is rarely because of the video itself.

The components:
- A person posts a video
- A few friends share it
- Each share reaches new people
- Some of those people share it too
- A celebrity happens to repost it
- A news outlet covers "the viral video"
- Millions of views

Could anyone predict this? Almost never.
The same video posted on a different day might get 50 views.

Emergent behavior:
- The virality is not a property of the video
- It is a property of the network, timing, and
  thousands of individual sharing decisions
- It emerges from the interaction of all these factors

Ant Colonies

No single ant knows how to build a colony, find the shortest path to food, or organize a defense. Yet colonies do all of these things.

Individual ant behavior:
- Follow pheromone trails
- Drop pheromones when carrying food
- Wander randomly when no trail is detected
- Return to nest when carrying food

Emergent colony behavior:
- Efficient food-gathering routes appear
- The colony "decides" which food source to prioritize
- Defense is coordinated against threats
- Complex tunnel architectures are built

No ant is in charge. No ant has a plan.
The colony's intelligence emerges from simple
individual rules and interactions.

Side Effects: The Unintended Consequences

A side effect is something that happens as a consequence of an action, but was not the goal.

Building a highway through a city:
Intended: Faster commute between suburbs and downtown
Side effects:
- Neighborhoods split in two
- Property values near the highway drop
- Businesses on old routes lose customers
- Air quality decreases for nearby residents
- Suburban sprawl increases

Every one of these side effects was predictable in
hindsight, but none was the goal.

Introducing a new grading policy at school:
Intended: Motivate students to study harder
Side effects:
- Students optimize for grades, not learning
- Cheating increases
- Student anxiety rises
- Students avoid challenging courses to protect their GPA
- Teachers feel pressure to inflate grades

Why Emergence Surprises Us

We think linearly:
- "If I add one more server, I get a little more capacity"
- True up to a point, then coordination overhead dominates

We think in isolation:
- "This feature works perfectly in testing"
- But in production, it interacts with features we did not test

We think about first-order effects:
- "This policy will reduce costs" (first order)
- "But it will also reduce quality" (second order)
- "Which will reduce customer trust" (third order)
- "Which will reduce revenue more than we saved" (fourth order)

Emergence lives in the second, third, and fourth order effects.

Connecting to Technology

Distributed System Bugs

Individual services work perfectly. Together, they produce bugs that no single service contains.

Scenario: Two services update the same user record

Service A: Update user's email address
Service B: Update user's phone number

Each service works correctly in isolation.

What happens when both run at the same time:
1. Service A reads user record: {email: "old", phone: "old"}
2. Service B reads user record: {email: "old", phone: "old"}
3. Service A writes: {email: "new", phone: "old"}
4. Service B writes: {email: "old", phone: "new"}

Result: The email update is lost.
Neither service has a bug.
The bug emerges from their interaction.

This is not a flaw in either component — it is a property of the system that only appears when parts interact under specific timing conditions.

Race Conditions

A race condition is when system behavior depends on the unpredictable timing of events.

Everyday analogy:
Two people walking toward the same parking spot
from different directions. Who gets it depends on
who arrives first — and neither can see the other
coming around the corner.

In tech:
Two users both see "1 item left in stock"
Both click "Buy" at the same time
Both transactions try to decrement the inventory
Result: -1 items in stock (should be impossible)

The system was designed for one buyer at a time.
The race condition emerges when two buyers act
simultaneously — a scenario the individual purchase
logic does not account for.

Common race condition patterns:

Check-then-act:
1. Check: Is the file available? → Yes
2. (Another process deletes the file)
3. Act: Open the file → Error: file not found

Read-modify-write:
1. Read: Counter = 10
2. (Another process reads Counter = 10)
3. Modify: Counter = 10 + 1 = 11
4. Write: Counter = 11
5. (Other process writes: Counter = 11)
6. Expected: 12. Actual: 11.

Unintended API Usage

When you build a system, people will use it in ways you never imagined.

Example: A public API for weather data

Intended use:
- Apps check the weather a few times per day per user

Actual use:
- A company builds a service that checks every weather
  station every 30 seconds
- They sell this aggregated data to hedge funds
- Your API is handling 100x the expected load
- Your infrastructure costs spike
- Your other users experience slow responses

The API works exactly as designed.
The emergent behavior (being used as a real-time data
pipeline for financial trading) was never intended.

Another example: Social media "Like" button

Intended: Let people express appreciation for content
Emergent effects:
- Dopamine-driven engagement loops
- Content optimized for likes rather than quality
- Social comparison and anxiety
- Political polarization (divisive content gets more likes)
- An entire influencer economy

None of these were the goal of adding a thumbs-up button.
All of them emerged from millions of people interacting
with a simple feature.

Cascading Emergent Failures

A real-world pattern from cloud services:

1. A minor bug causes one server to restart slowly
2. Load shifts to other servers (normal behavior)
3. Other servers are now at higher load (expected)
4. The monitoring system flags high load (working correctly)
5. Autoscaler launches new servers (correct response)
6. New servers need to sync data from existing servers
7. This syncing adds more load to already-stressed servers
8. More servers become unhealthy
9. More restarts, more syncing, more load
10. Entire region goes down

Every component behaved correctly according to its rules.
The outage emerged from the interaction of correct behaviors
under unusual conditions.

Anticipating Emergent Behavior

You cannot predict every emergent behavior, but you can prepare for it.

Strategies:

1. Think about interactions, not just components
   - What happens when A and B act at the same time?
   - What if C fails while D is in the middle of something?

2. Test at the system level, not just the unit level
   - Unit tests: "Does this function work?"
   - Integration tests: "Do these parts work together?"
   - Chaos testing: "What happens when things break randomly?"

3. Design for graceful degradation
   - If one part behaves unexpectedly, the others should cope
   - Timeouts, circuit breakers, rate limits

4. Expect unintended usage
   - If an API can be called 1000 times per second,
     someone will call it 1000 times per second
   - Build rate limiting from the start

5. Monitor the system, not just the components
   - Individual health checks say "everything is fine"
   - System-level metrics say "total throughput dropped 50%"
   - Both views are needed

6. Think about second and third order effects
   - First order: What does this change do?
   - Second order: What will people do in response?
   - Third order: What happens when those responses interact?

Common Pitfalls

• Assuming predictability from simple rules. Simple rules governing many interacting agents produce complex, unpredictable behavior. Do not assume you can predict what will emerge.
• Testing only individual components. A system where every part passes its unit test can still fail spectacularly when assembled. Integration testing is essential.
• Ignoring second-order effects. Asking "what will this change do?" is necessary but insufficient. Ask "what will people do in response to this change?" and "what happens after that?"
• Designing for intended use only. If a system can be used in an unintended way, it will be. Rate limits, input validation, and usage monitoring are not optional.
• Blaming individual components for system failures. When an outage happens, the root cause is often not a single bug but an interaction between multiple correct behaviors under unexpected conditions.
• Assuming more parts means proportionally more capability. Adding more servers, more people, or more features creates coordination overhead. Past a certain point, adding more can make the system slower.

Key Takeaways

• Emergent behavior is when a system's behavior cannot be predicted from its individual parts alone. It arises from interactions.
• Traffic jams, viral content, and ant colonies are all examples of emergence in everyday life.
• In technology, race conditions, distributed system bugs, and cascading failures are emergent behaviors.
• Side effects are the unintended consequences of intentional actions. They are almost always present and often more impactful than the intended effect.
• You cannot eliminate emergence, but you can prepare for it: test at the system level, monitor the whole system, and think about second-order effects.
• The question to always ask: "What happens when all these parts interact under conditions we have not tested?"