Scrolls
5 min read
On this page

Detection & Containment

The first hour of a security incident determines its outcome. Fast detection limits the attacker's access. Effective containment prevents the compromise from spreading. Slow detection and poor containment turn a minor breach into a catastrophic one. The 2020 SolarWinds attack went undetected for nine months — by the time it was discovered, attackers had accessed systems across thousands of organizations including US government agencies.

Detection Methods

Security incidents are discovered through multiple channels. Relying on a single detection method leaves dangerous blind spots.

Automated Monitoring & Alerts

The primary detection mechanism for mature organizations. Security monitoring systems continuously analyze logs, network traffic, and system behavior for indicators of compromise.

# Types of automated detection
SIEM (Security Information & Event Management)
  - Aggregates logs from all systems
  - Correlates events across sources
  - Triggers alerts on suspicious patterns
  - Examples: Splunk, Elastic Security, Sentinel

EDR (Endpoint Detection & Response)
  - Monitors individual machines for malicious behavior
  - Detects malware, fileless attacks, lateral movement
  - Examples: CrowdStrike, SentinelOne, Carbon Black

IDS/IPS (Intrusion Detection/Prevention Systems)
  - Monitors network traffic for attack signatures
  - Can block known attack patterns automatically
  - Examples: Suricata, Snort, Palo Alto

Effective alerting requires tuning. An alert system that fires 500 times per day becomes background noise — analysts stop investigating, and real incidents are missed. This is alert fatigue, and it is one of the biggest challenges in security operations.

User Reports

Users often detect incidents before automated systems do. An employee noticing unexpected password reset emails, a customer reporting unauthorized transactions, or a developer seeing unfamiliar code in a repository — these human observations catch what automated systems miss.

# Make reporting easy
- Provide a clear reporting channel (email: security@company.com)
- Respond to every report, even false alarms
- Never punish reporters, even if they caused the issue
- Include security reporting in employee onboarding
- Acknowledge reports within 1 hour during business hours

Third-Party Notification

Sometimes you learn about your breach from someone else. Security researchers discover your data on paste sites. A competitor's breach investigation reveals shared infrastructure was compromised. Law enforcement contacts you about your systems being used in an attack.

The "Have I Been Pwned" service has notified numerous organizations about breaches they were unaware of. Being notified externally is embarrassing but still better than remaining unaware.

Log Analysis

Logs are the forensic evidence of a security incident. Knowing what to look for separates rapid detection from months of undetected compromise.

Unusual Access Patterns

# Indicators to monitor
- Login from unexpected geographic locations
- Access at unusual hours for the user's timezone
- Multiple failed login attempts followed by success
- Session activity from multiple IP addresses simultaneously
- Accessing resources outside the user's normal pattern

A developer who typically accesses three repositories suddenly downloading every repository in the organization is suspicious. A marketing account querying the production database at 3 AM warrants investigation.

Privilege Escalation

# Watch for these events
- User added to admin or privileged groups
- New service accounts created outside normal provisioning
- Permission changes on sensitive resources
- Sudo or administrator command execution by non-admin users
- Role assignments that bypass approval workflows

Attackers who gain initial access through a low-privilege account immediately attempt to escalate privileges. Monitoring privilege changes catches this lateral movement.

Data Exfiltration Indicators

# Potential exfiltration signals
- Unusually large database queries
- Bulk API calls to data endpoints
- Large file uploads to external services
- DNS queries to unusual domains (DNS tunneling)
- Encrypted traffic to unknown endpoints
- Compressed archive creation on servers

Data exfiltration is often the attacker's primary objective. They may move slowly, exfiltrating small amounts over time to avoid detection. Baseline normal data access patterns so anomalies stand out.

Authentication Anomalies

# Authentication events to alert on
- Password spray attacks (many accounts, few passwords)
- Credential stuffing (known breach credentials tested)
- MFA bypass attempts
- Token reuse after session invalidation
- OAuth token granted to unfamiliar application
- Service account authentication from unexpected source

Containment Strategies

Containment stops the bleeding. The goal is to prevent the incident from spreading while preserving evidence for investigation. Every containment action involves trade-offs between business disruption and security.

Isolate Affected Systems

Remove compromised systems from the network without shutting them down. Shutting down destroys volatile memory that contains forensic evidence.

# Network isolation approaches
- Move affected systems to an isolated VLAN
- Apply firewall rules blocking all traffic except forensic access
- Disable network interfaces on compromised hosts
- Revoke VPN access for compromised accounts
- Isolate affected cloud resources using security groups

For cloud environments, modify security groups to restrict traffic rather than terminating instances. The running instance contains evidence.

Revoke Compromised Credentials

When credentials are compromised, revocation must be immediate and comprehensive.

# Credential revocation checklist
1. Reset the compromised password
2. Revoke all active sessions and tokens
3. Rotate API keys associated with the account
4. Revoke OAuth grants to third-party applications
5. Rotate any shared secrets the account had access to
6. Disable the account until investigation completes
7. Check for credential reuse on other systems
8. Rotate any secrets stored in systems the account accessed

If a service account is compromised, every secret that service account could access must be treated as compromised. This cascade effect is why limiting service account permissions matters.

Block Attacker Infrastructure

# Blocking actions
- Block attacker IP addresses at the firewall
- Add malicious domains to DNS blocklists
- Block command-and-control (C2) server addresses
- Update WAF rules to block attack patterns
- Add indicators of compromise to EDR blocklists

Be cautious with IP blocking — attackers use shared infrastructure (cloud providers, CDNs, VPNs). Blocking a cloud provider's IP range can disrupt legitimate services.

Preserving Evidence

Evidence preservation is critical and often overlooked in the rush to contain and remediate.

Do Not Wipe Before Forensics

The instinct to "clean up" a compromised system by reimaging or deleting files destroys the evidence needed to understand what happened, what was accessed, and whether the attacker left backdoors.

# Evidence preservation steps
1. Create forensic disk images before any changes
2. Capture memory dumps from running systems
3. Export all relevant logs to secure storage
4. Record network traffic captures if available
5. Screenshot any visible indicators of compromise
6. Document the exact state of affected systems
7. Preserve timestamps - do not modify files

Chain of Custody

If the incident may involve law enforcement or legal proceedings, maintain a chain of custody for all evidence. Document who accessed the evidence, when, and what they did with it. Store evidence on write-once media with cryptographic hashes to prove integrity.

Log Retention

Logs are only useful if they exist when you need them. Many organizations discover during an incident that their logs were rotated or deleted weeks ago.

# Recommended log retention
Security logs:        1 year minimum
Authentication logs:  1 year minimum
Application logs:     90 days minimum
Network flow data:    90 days minimum
Database audit logs:  1 year minimum

Store logs in a centralized, tamper-evident system. Attackers who compromise a system often attempt to delete logs to cover their tracks. Shipping logs to a separate system in real time prevents this.

The First Hour Checklist

When a potential incident is detected, the first hour sets the trajectory for the entire response.

# First hour actions
00:00  Alert received - assign initial responder
00:05  Confirm whether alert is a true positive
00:10  Classify severity (P1-P4)
00:15  Notify Incident Commander and assemble team
00:20  Begin evidence preservation (logs, memory, disk)
00:25  Identify scope: what systems are affected?
00:30  Initiate containment for confirmed compromise
00:40  Begin log analysis for indicators of compromise
00:50  Assess data exposure - what could the attacker access?
00:60  First status update to stakeholders

Speed matters, but accuracy matters more. Containment actions that are too broad (shutting down all production systems) cause more damage than the incident itself. Containment that is too narrow (blocking one IP when the attacker has ten) gives a false sense of security.

Common Pitfalls

  • Alert fatigue. Too many false positives cause analysts to ignore real alerts. Tune detection rules to minimize noise while maintaining coverage.
  • Wiping compromised systems immediately. Reimaging before forensic analysis destroys evidence of what happened and what the attacker accessed.
  • Incomplete credential revocation. Resetting a password but not revoking active sessions leaves the attacker logged in. Revoke everything.
  • Containing too slowly. Every minute of delay extends the attacker's access. Once an incident is confirmed, containment should begin within minutes, not hours.
  • Not monitoring during containment. Attackers who detect containment efforts may accelerate their attack or activate backup access methods. Increase monitoring during and after containment.
  • Single-channel detection. Relying only on automated monitoring misses incidents that automated tools cannot detect. Combine automated, human, and third-party detection.

Key Takeaways

  • Detection comes from multiple sources: automated monitoring, user reports, and third-party notifications.
  • Log analysis should focus on unusual access patterns, privilege escalation, data exfiltration, and authentication anomalies.
  • Containment isolates affected systems, revokes compromised credentials, and blocks attacker infrastructure.
  • Never wipe systems before forensic investigation — preserve evidence first.
  • The first hour determines the incident's trajectory. Have a checklist ready.
  • Alert fatigue is a real threat — tune detection systems to minimize false positives.
  • Ship logs to a centralized, tamper-evident system with sufficient retention periods.