Operating System Overview
An operating system is the software layer between applications and hardware. It manages resources, provides abstractions, and enforces protection.
OS Roles
Resource Manager
The OS allocates and manages:
- CPU: Scheduling processes/threads
- Memory: Virtual memory, allocation, protection
- Storage: File systems, I/O scheduling
- I/O devices: Drivers, buffering, access control
- Network: Protocol stacks, sockets, routing
Abstraction Provider
The OS hides hardware complexity behind clean abstractions:
| Hardware | OS Abstraction |
|---|---|
| CPU | Process / Thread |
| Physical memory | Virtual address space |
| Disk blocks | Files and directories |
| Network hardware | Sockets and connections |
| Display hardware | Windows and graphics contexts |
| Timer hardware | sleep(), alarm(), scheduling |
Protection Enforcer
The OS isolates processes from each other and from the OS itself:
- Memory protection: Each process has its own address space
- CPU protection: Preemptive scheduling prevents monopolization
- I/O protection: Only the OS can access hardware directly
- File protection: Access control lists, permissions
Kernel Types
Monolithic Kernel
All OS services (scheduling, memory management, file systems, drivers, networking) run in kernel mode in a single address space.
┌─────────────────────────────┐
│ User Space │
│ App1 App2 App3 Shell │
├─────────────────────────────┤ ← Syscall interface
│ Kernel Space │
│ Scheduler │ Memory Mgr │
│ File Sys │ Network Stack │
│ Device Drivers │ IPC │
└─────────────────────────────┘
│ Hardware │
Advantages: Fast (no IPC overhead between kernel components). Direct function calls between subsystems.
Disadvantages: A bug in any component crashes the entire system. Large, complex codebase. Hard to maintain.
Examples: Linux, FreeBSD, Windows NT (hybrid but mostly monolithic), Solaris.
Linux: ~30M lines of code. Modular monolithic — loadable kernel modules (LKMs) can be inserted/removed at runtime without rebooting.
Microkernel
Minimal kernel: only essential services (IPC, basic scheduling, memory management). Everything else (file systems, drivers, networking) runs as user-space servers.
┌──────────────────────────────────────┐
│ User Space │
│ App1 │ FS Server │ Net Server │ Driver│
├──────────────────────────────────────┤
│ Microkernel │
│ IPC │ Scheduling │ Memory │
└──────────────────────────────────────┘
Advantages: Isolation (driver crash doesn't crash kernel). Smaller trusted computing base. Easier formal verification.
Disadvantages: IPC overhead for every system service call. Historically slower than monolithic (gap has narrowed).
Examples: MINIX 3, QNX, seL4 (formally verified), L4 family, Fuchsia (Zircon), GNU Hurd.
Hybrid Kernel
Monolithic core with some services in user space. Pragmatic compromise.
Examples: Windows NT, macOS (XNU = Mach microkernel + BSD monolithic layer).
Exokernel
Minimal kernel that only multiplexes hardware. Applications manage their own abstractions via library operating systems (libOS).
Philosophy: "Don't abstract — just protect." Applications get near-direct hardware access for maximum performance.
Examples: MIT Exokernel (research), Unikernel concept derives from this.
Unikernel
Application + minimal OS compiled into a single address space image. Runs directly on a hypervisor.
Advantages: Tiny attack surface. Fast boot (milliseconds). Minimal overhead.
Disadvantages: Single application. No shell, no debugging tools. Hard to develop.
Examples: MirageOS (OCaml), IncludeOS (C++), Unikraft.
System Calls
The system call interface is the boundary between user space and kernel space.
Mechanism
- User program invokes a trap instruction (syscall, svc, int 0x80).
- CPU switches to kernel mode (privileged).
- Kernel validates arguments and performs the operation.
- Kernel returns result and switches back to user mode.
Common System Calls
| Category | System Calls |
|---|---|
| Process | fork, exec, exit, wait, kill, getpid |
| File | open, close, read, write, lseek, stat |
| Directory | mkdir, rmdir, readdir, chdir |
| Memory | mmap, munmap, brk, mprotect |
| I/O | ioctl, poll, select, epoll |
| Network | socket, bind, listen, accept, connect, send, recv |
| Time | time, clock_gettime, nanosleep |
| Signals | signal, sigaction, kill, sigprocmask |
User Mode vs Kernel Mode
| Aspect | User Mode | Kernel Mode |
|---|---|---|
| Privilege | Restricted | Full hardware access |
| Memory | Own address space | All memory accessible |
| Instructions | Normal instructions only | All instructions (I/O, MMU, interrupts) |
| Failure | Process crash | System crash (kernel panic) |
| Transition | syscall, exception, interrupt | Return from syscall/interrupt |
Hardware support: CPU privilege rings (x86: Ring 0 = kernel, Ring 3 = user). ARM: EL0 (user), EL1 (kernel), EL2 (hypervisor), EL3 (secure monitor).
Trap/Interrupt Mechanism
Interrupts
Hardware interrupts: External events (keyboard, network, disk, timer). Asynchronous — can happen at any time.
Software interrupts (traps): Triggered by the program (syscall, breakpoint, illegal instruction). Synchronous.
Interrupt Handling
- CPU saves state (PC, registers) to kernel stack.
- CPU looks up the interrupt vector table for the handler address.
- Jump to the interrupt handler (ISR).
- Handler processes the event.
- Restore state and return to the interrupted code.
Interrupt vector table: Array of handler addresses, one per interrupt type. Initialized at boot.
OS History and Evolution
| Era | Systems | Key Innovations |
|---|---|---|
| 1950s | Batch systems | Single job at a time. Operator-driven. |
| 1960s | Multiprogramming | Multiple jobs in memory. Timesharing (CTSS, Multics). |
| 1970s | UNIX | Simple, elegant. "Everything is a file." C language. |
| 1980s | PC operating systems | MS-DOS (single-tasking). Mac OS. |
| 1990s | Modern OS | Windows NT, Linux. Preemptive multitasking. TCP/IP. GUI. |
| 2000s | Mobile / Cloud | iOS, Android. Virtualization (VMware, Xen). Containers. |
| 2010s | Containers / Serverless | Docker, Kubernetes. Lambda/Cloud Functions. |
| 2020s | Rust-based OS, unikernels | Redox OS, Theseus, Hubris. eBPF. io_uring. |
UNIX Philosophy
- Make each program do one thing well.
- Expect the output of every program to become the input of another.
- Design and build software to be tried early.
- Use tools rather than unskilled help to lighten a programming task.
"Everything is a file" — files, devices, pipes, sockets all accessed through file descriptors with read/write.
Applications in CS
- Systems programming: Writing OS code requires deep understanding of hardware, concurrency, and resource management.
- Application development: Understanding the OS helps write efficient applications (avoiding unnecessary syscalls, using appropriate I/O mechanisms).
- Security: OS is the foundation of system security. Understanding the kernel helps identify and fix vulnerabilities.
- Cloud computing: VMs, containers, and orchestration rely on OS mechanisms (namespaces, cgroups, hypervisors).
- Embedded systems: Real-time OS choices (FreeRTOS, Zephyr) depend on understanding OS tradeoffs.
- Performance engineering: Understanding scheduling, memory management, and I/O helps diagnose and fix performance issues.