7 min read
On this page

Malware Analysis

Malware analysis determines the functionality, origin, and impact of malicious software. Analysts use static and dynamic techniques to understand malware behavior without falling victim to it, enabling detection, containment, and remediation.

Malware Types

Virus

Self-replicating code that attaches to a host program. Executes when the host runs. Spreads through infected files, boot sectors, or macros.

File infector: Modifies executable files by prepending, appending, or cavity-injecting code.

Boot sector virus: Infects the Master Boot Record (MBR) or Volume Boot Record (VBR). Executes before the OS loads.

Macro virus: Embedded in document formats (Word, Excel). Exploits scripting capabilities (VBA).

Worm

Self-propagating malware that spreads without user interaction, typically over networks.

Propagation vectors: Exploiting vulnerabilities (EternalBlue/WannaCry), brute-forcing credentials, email attachments, USB drives.

Key difference from virus: Worms are standalone — they do not need a host program.

Trojan

Disguised as legitimate software. Does not self-replicate. Relies on social engineering for distribution.

Subtypes: Remote Access Trojan (RAT), banking trojan, dropper (downloads additional payloads), downloader, spyware.

Ransomware

Encrypts victim files and demands payment for the decryption key.

Modern ransomware operations:

  1. Initial access (phishing, exploits, RDP brute force).
  2. Lateral movement and privilege escalation.
  3. Data exfiltration (double extortion — threaten to publish data).
  4. Encryption of files across the network.
  5. Ransom note with payment instructions (usually cryptocurrency).

Encryption: Typically uses a symmetric key (AES) per file, then encrypts each file key with an RSA public key. Only the attacker holds the RSA private key.

Rootkit

Hides its presence and other malware from the operating system and security tools.

User-mode rootkit: Hooks API functions (IAT hooking, inline hooking) to filter results. Hides processes, files, and registry keys from user-mode tools.

Kernel-mode rootkit: Modifies kernel data structures (DKOM — Direct Kernel Object Manipulation) or hooks system call tables. More powerful and harder to detect.

Bootkits: Infect the boot process (MBR, VBR, UEFI firmware). Execute before the OS loads, gaining control before any security software.

Detection: Cross-view analysis (compare multiple methods of listing processes/files), integrity checking of kernel structures, boot process verification (Secure Boot, Measured Boot).

Static Analysis

Examining malware without executing it. Safe but limited by obfuscation.

Triage

Initial assessment to classify the sample and prioritize analysis.

# File type identification
file suspicious.exe
# PE32 executable (GUI) Intel 80386, for MS Windows

# Hash for lookup in threat intelligence databases
sha256sum suspicious.exe
# Search hash on VirusTotal, MalwareBazaar, etc.

# String extraction
strings -n 8 suspicious.exe | grep -i "http\|cmd\|reg\|password"
# Reveals URLs, commands, registry keys, hardcoded credentials

# Entropy analysis (high entropy suggests packing/encryption)
binwalk -E suspicious.exe

PE (Portable Executable) Analysis

Windows executables use the PE format. Key areas for analysis:

PE Structure:
┌─────────────────┐
│ DOS Header      │ MZ signature
│ DOS Stub        │
├─────────────────┤
│ PE Signature    │ PE\0\0
│ COFF Header     │ Machine type, number of sections, timestamp
│ Optional Header │ Entry point, image base, subsystem
├─────────────────┤
│ Section Table   │
│ .text           │ Code — should be executable, not writable
│ .rdata          │ Read-only data, imports, exports
│ .data           │ Read-write initialized data
│ .rsrc           │ Resources (icons, strings, embedded files)
│ .reloc          │ Relocation table
└─────────────────┘

Suspicious indicators:

  • Section with both writable and executable flags (self-modifying code).
  • Very high entropy in a section (packed/encrypted content).
  • Small import table (imports resolved dynamically at runtime to evade analysis).
  • Mismatched compile timestamp (backdated or zeroed).
  • Non-standard section names (.UPX0, .packed).

Tools: PE-bear, pestudio, CFF Explorer, pefile (Python library).

ELF Analysis

Linux executables use ELF format. Analysis approach is similar.

# ELF header information
readelf -h suspicious_elf

# Section headers
readelf -S suspicious_elf

# Symbol table (if not stripped)
readelf -s suspicious_elf

# Dynamic linking information (imported libraries and functions)
readelf -d suspicious_elf
ldd suspicious_elf  # WARNING: can execute code — use on trusted files only

Disassembly

Converting machine code to assembly for detailed code analysis.

Tools: Ghidra (free, includes decompiler), IDA Pro (commercial, industry standard), radare2/Cutter (open source).

Key analysis targets: Entry point, main function, network communication functions, file I/O, registry/configuration access, cryptographic routines, anti-analysis checks.

Dynamic Analysis

Executing malware in a controlled environment to observe its behavior.

Sandboxing

Automated malware execution and behavior recording.

Automated sandboxes: Cuckoo Sandbox (open source), ANY.RUN (interactive), Joe Sandbox, VMRay.

Environment setup:

  • Isolated virtual machine with snapshots for clean restoration.
  • Simulated network services (INetSim, FakeNet-NG) to capture C2 traffic without real connectivity.
  • Host-only or isolated network to prevent accidental spread.

What sandboxes capture: File system changes, registry modifications, process creation, network traffic, API calls, screenshots.

API Monitoring

Monitoring system calls reveals malware behavior regardless of obfuscation.

Windows API monitoring tools: API Monitor, x64dbg with plugins, Process Monitor (Sysinternals).

Key API categories to monitor:

| Category | APIs | Indicates | |----------|------|-----------| | File I/O | CreateFile, WriteFile, DeleteFile | File dropping, encryption | | Registry | RegSetValue, RegCreateKey | Persistence, configuration | | Process | CreateProcess, VirtualAllocEx, WriteProcessMemory | Process injection | | Network | connect, send, recv, InternetOpen | C2 communication | | Crypto | CryptEncrypt, CryptDecrypt | Ransomware, data protection | | Anti-debug | IsDebuggerPresent, NtQueryInformationProcess | Evasion |

Injection detection pattern: VirtualAllocEx followed by WriteProcessMemory to the same remote process strongly indicates process injection. Sequences of NtCreateSection + NtMapViewOfSection indicate a variant called process hollowing.

Obfuscation Techniques

Packing

Compressing or encrypting the executable payload. At runtime, a small stub decompresses/decrypts the original code in memory.

Common packers: UPX (easily unpacked), Themida, VMProtect, custom packers.

Detection: High entropy, small import table, VirtualAlloc + VirtualProtect calls to allocate executable memory at runtime.

Unpacking: Set breakpoints on VirtualProtect (when changing page to executable) or at the original entry point (OEP). Dump memory after unpacking.

Polymorphism

Each copy of the malware has a different binary signature while maintaining the same functionality.

Techniques: Variable instruction substitution (e.g., xor eax,eax vs sub eax,eax vs mov eax,0), register reassignment, instruction reordering (where order-independent), junk code insertion.

Metamorphic malware: Goes further — rewrites its own code engine each generation. No constant signature exists.

Other Obfuscation Methods

  • String encryption: Strings decrypted at runtime. Defeats static string extraction.
  • Control flow flattening: Replaces structured control flow with a dispatcher loop, making decompilation difficult.
  • Opaque predicates: Conditional branches whose outcome is predetermined but hard to analyze statically.
  • API hashing: Instead of importing functions by name, the malware resolves them at runtime by comparing hash values.

Persistence Mechanisms

Techniques malware uses to survive system reboots.

Windows Persistence

| Mechanism | Location/Method | |-----------|----------------| | Registry Run keys | HKLM\...\Run, HKCU\...\Run | | Scheduled Tasks | schtasks /create | | Services | sc create or direct registry modification | | WMI event subscriptions | Permanent event consumers | | DLL hijacking | Placing a malicious DLL in a search-path directory | | COM object hijacking | Registering a malicious COM server | | Boot/logon scripts | Group Policy scripts | | Startup folder | %AppData%\...\Startup\ |

Linux Persistence

| Mechanism | Location/Method | |-----------|----------------| | Cron jobs | /etc/crontab, /var/spool/cron/ | | Systemd services | /etc/systemd/system/ | | Shell profiles | .bashrc, .profile, /etc/profile.d/ | | SSH authorized_keys | ~/.ssh/authorized_keys | | LD_PRELOAD | /etc/ld.so.preload (library injection) | | Kernel modules | insmod / /etc/modules | | Init scripts | /etc/init.d/ |

Indicators of Compromise (IoCs)

Observable artifacts that indicate a system has been compromised.

IoC Types

| Type | Example | Use | |------|---------|-----| | File hash | SHA-256 of malware binary | Exact match detection | | IP address | C2 server IP | Network blocking/detection | | Domain | evil-c2.example.com | DNS blocking | | URL | Full URL path for payload download | Proxy/WAF rules | | Email address | Phishing sender | Email filtering | | Mutex name | Named mutex created by malware | Host-based detection | | Registry key | Persistence registry path | Host-based detection | | YARA rule | Pattern-based detection signature | Flexible file scanning |

YARA Rules

Pattern matching language for malware classification.

rule Ransomware_Note {
    meta:
        description = "Detects common ransomware note patterns"
        severity = "high"

    strings:
        $s1 = "Your files have been encrypted" ascii wide nocase
        $s2 = "bitcoin" ascii wide nocase
        $s3 = "decrypt" ascii wide nocase
        $s4 = /[13][a-km-zA-HJ-NP-Z1-9]{25,34}/ // Bitcoin address pattern
        $ext = ".encrypted" ascii
        $ext2 = ".locked" ascii

    condition:
        2 of ($s1, $s2, $s3) and 1 of ($s4, $ext, $ext2)
}

Pyramid of Pain

Ranks IoC types by how hard they are for attackers to change (bottom = trivial, top = very hard):

| Level | IoC Type | Difficulty to Change | |-------|----------|---------------------| | Top | TTPs (behavior patterns) | Very hard — requires new techniques | | | Tools | Hard — custom tools take effort | | | Network/host artifacts | Moderate — new infrastructure | | | IP addresses / domains | Easy — register new ones | | Bottom | Hash values | Trivial — recompile with one byte changed |

Detecting TTPs — how the attacker operates — is the most durable form of detection. The MITRE ATT&CK framework catalogs known adversary TTPs and maps them to detection strategies.

Effective malware analysis combines static and dynamic techniques, using static analysis to guide dynamic investigation and dynamic behavior to contextualize code-level findings. The output — IoCs, YARA rules, and behavioral descriptions — feeds directly into detection, response, and threat intelligence sharing.