4 min read
On this page

Protection and Security

OS protection mechanisms control access to resources. Security prevents unauthorized access, modification, and denial of service.

Protection Domains

A protection domain defines the set of resources a process can access and the operations it can perform.

Each process runs in a domain that specifies: which files it can read/write, which memory regions it can access, which devices it can use, which system calls it can make.

Access Control

Access Matrix

Conceptual model: rows = domains (processes/users), columns = resources (files/devices).

         File1    File2    Printer  Process2
User A:  RW       R        -        -
User B:  R        RW       W        Signal
Root:    RWX      RWX      W        Kill

The matrix is usually sparse — implemented as ACLs or capability lists.

Access Control Lists (ACLs)

Store permissions per resource (column of the access matrix).

File: /etc/passwd
  Owner: root  → RW
  Group: root  → R
  Others:      → R
  User alice:  → RW  (extended ACL)

UNIX permissions (simplified ACL):

-rwxr-xr-- 1 alice developers 4096 Jan 1 00:00 script.sh
 │││ │││ │││
 │││ │││ └── Others: read only
 │││ └── Group: read + execute
 └── Owner: read + write + execute

Three permission classes (owner, group, others) × three permission types (read, write, execute).

Extended ACLs (POSIX): Additional named users/groups beyond the basic three. setfacl / getfacl.

Capability Lists

Store permissions per domain (row of the access matrix). Each process holds a set of capabilities (unforgeable tokens granting specific access).

Advantages: Easy to delegate access (pass capability). Easy to determine a process's full access. Fine-grained.

Disadvantages: Hard to revoke access (must find all copies of a capability). Hard to determine who has access to a resource.

Used in: seL4 (capability-based security), Capsicum (FreeBSD), WASI (WebAssembly capability model).

Mandatory vs Discretionary Access Control

DAC (Discretionary Access Control)

Resource owners control access. Standard UNIX permissions.

Problem: A process running as a user has ALL of that user's permissions. A compromised web browser can read all the user's files.

MAC (Mandatory Access Control)

System-wide policy controls access. Users/processes cannot change the policy.

Bell-LaPadula model (confidentiality): "No read up, no write down." Prevents information leakage from higher to lower security levels.

Biba model (integrity): "No read down, no write up." Prevents corruption of higher-integrity data from lower levels.

RBAC (Role-Based Access Control)

Assign permissions to roles (admin, developer, reader), then assign roles to users.

Role "developer": read/write code repos, read databases, deploy to staging
User "alice": role "developer"
User "bob": roles "developer", "admin"

Simplifies management: change a role's permissions → all users with that role are updated.

Used in: Cloud IAM (AWS, GCP, Azure), databases, enterprise systems.

OS Security Mechanisms

ASLR (Address Space Layout Randomization)

Randomize the base addresses of the stack, heap, libraries, and executable in the virtual address space.

Purpose: Make exploitation harder. Attacker can't predict where code/data is located. Defeats many buffer overflow and ROP attacks.

Bypass: Information leaks (format string bugs, partial overwrites) can reveal addresses. Brute force on 32-bit systems (limited entropy).

DEP / NX (No-Execute)

Mark memory pages as either writable or executable, but not both (W⊕X).

Purpose: Prevent code injection. Attacker writes shellcode to the stack → can't execute it (page is writable, not executable).

Bypass: Return-to-libc, ROP (use existing executable code gadgets instead of injecting new code).

Stack Canaries

Place a random value (canary) on the stack between local variables and the return address. On function return, check if the canary is intact.

Stack:
[local vars] [CANARY] [saved BP] [return addr]

If buffer overflow overwrites the return address, it must also overwrite the canary → detected → abort.

Bypass: Information leak to read the canary, or overwrite function pointers instead of return addresses.

Seccomp (Secure Computing)

Restrict which system calls a process can make.

// Allow only read, write, exit, sigreturn
seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(read), 0);
seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(write), 0);
seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(exit), 0);
// All other syscalls → KILL

seccomp-bpf: Programmable filter using BPF programs. Can inspect syscall arguments.

Used by: Docker, Chrome (renderer processes), Firefox, systemd, OpenSSH.

SELinux and AppArmor

SELinux (Security-Enhanced Linux): MAC system. Every file, process, socket has a security context (label). Policy rules specify which contexts can access which.

AppArmor: Path-based MAC. Simpler than SELinux (uses file paths instead of labels). Per-program profiles.

Sandboxing

Restrict a process's access to the minimum needed.

Techniques: seccomp, namespaces, chroot, capabilities, MAC (SELinux/AppArmor).

Landlock (Linux 5.13): Unprivileged sandboxing. Process can restrict its own access to files and network.

Pledge/Unveil (OpenBSD): Process promises to only use certain syscalls (pledge) and access certain paths (unveil).

Applications in CS

  • Container security: Namespaces + cgroups + seccomp + AppArmor/SELinux provide defense in depth.
  • Browser security: Each tab/process sandboxed with restricted syscalls and file access.
  • Server hardening: Drop unnecessary capabilities, enable MAC, restrict syscalls.
  • Cloud IAM: Role-based access to cloud resources. Principle of least privilege.
  • Mobile security: App sandboxing (Android uses SELinux, iOS uses mandatory code signing + sandboxing).
  • Secure boot: Chain of trust from firmware → bootloader → kernel → userspace.