Differential Equations

Differential equations describe relationships between functions and their derivatives. They model dynamic systems — from population growth to neural network training dynamics.

First-Order ODEs

Separable Equations

Form: dy/dx = f(x)g(y). Separate variables and integrate:

∫ dy/g(y) = ∫ f(x) dx

Example: dy/dx = xy → ∫ dy/y = ∫ x dx → ln|y| = x²/2 + C → y = Ae^(x²/2).

First-Order Linear

Form: dy/dx + P(x)y = Q(x).

Integrating factor: μ(x) = e^(∫P(x)dx).

Multiply both sides by μ: d/dx[μy] = μQ. Integrate:

y = (1/μ) ∫ μQ dx

Example: dy/dx + 2y = e⁻ˣ. μ = e^(2x). y = e⁻²ˣ ∫ e²ˣ · e⁻ˣ dx = e⁻²ˣ(eˣ + C) = e⁻ˣ + Ce⁻²ˣ.

Exact Equations

Form: M(x,y)dx + N(x,y)dy = 0 is exact if ∂M/∂y = ∂N/∂x.

Then there exists F(x,y) with ∂F/∂x = M and ∂F/∂y = N. Solution: F(x,y) = C.

Bernoulli Equations

Form: dy/dx + P(x)y = Q(x)yⁿ.

Substitution v = y^(1-n) converts to a linear equation.

Second-Order Linear ODEs

Homogeneous with Constant Coefficients

ay'' + by' + cy = 0

Characteristic equation: ar² + br + c = 0.

| Roots | General Solution | |---|---| | Real distinct r₁, r₂ | y = c₁e^(r₁x) + c₂e^(r₂x) | | Repeated r | y = (c₁ + c₂x)e^(rx) | | Complex α ± βi | y = e^(αx)(c₁cos(βx) + c₂sin(βx)) |

Example: y'' - 5y' + 6y = 0. r² - 5r + 6 = (r-2)(r-3) = 0. y = c₁e^(2x) + c₂e^(3x).

Example: y'' + 4y = 0. r² + 4 = 0, r = ±2i. y = c₁cos(2x) + c₂sin(2x). (Oscillation!)

Non-Homogeneous

ay'' + by' + cy = f(x)

Solution: y = y_h + y_p (homogeneous + particular).

Method of undetermined coefficients: Guess y_p based on f(x):

| f(x) | Guess for y_p | |---|---| | Polynomial degree n | Polynomial degree n | | e^(kx) | Ae^(kx) | | sin(kx) or cos(kx) | A sin(kx) + B cos(kx) | | Product of above | Product of corresponding guesses |

If the guess is a solution to the homogeneous equation, multiply by x (or x²).

Variation of parameters: Works for any f(x). If y₁, y₂ are homogeneous solutions:

y_p = -y₁ ∫ (y₂f)/(W) dx + y₂ ∫ (y₁f)/(W) dx

where W = y₁y₂' - y₂y₁' is the Wronskian.

Laplace Transforms

The Laplace transform converts ODEs into algebraic equations:

L{f(t)} = F(s) = ∫₀^∞ e^(-st) f(t) dt

Key Transforms

| f(t) | F(s) | |---|---| | 1 | 1/s | | t^n | n!/s^(n+1) | | e^(at) | 1/(s-a) | | sin(bt) | b/(s²+b²) | | cos(bt) | s/(s²+b²) | | δ(t) (impulse) | 1 | | u(t-a) (step) | e^(-as)/s |

Properties

Linearity: L{af + bg} = aF + bG
Derivative: L{f'} = sF(s) - f(0)
Second derivative: L{f''} = s²F(s) - sf(0) - f'(0)
Shift: L{e^(at)f(t)} = F(s-a)
Convolution: L{f * g} = F(s) · G(s)

Solving ODEs with Laplace Transforms

Take Laplace transform of both sides (using initial conditions).
Solve algebraically for Y(s).
Inverse Laplace transform to get y(t).

Example: y'' + y = sin(t), y(0) = 0, y'(0) = 0.

L: s²Y + Y = 1/(s²+1) → Y = 1/(s²+1)² → y(t) = ½(sin t - t cos t).

Systems of ODEs

x' = Ax    where x ∈ ℝⁿ, A ∈ ℝⁿˣⁿ

Solution: x(t) = e^(At) x(0).

If A = PDP⁻¹ (diagonalizable):

e^(At) = P · diag(e^(λ₁t), ..., e^(λₙt)) · P⁻¹

Phase Portraits (2D)

The eigenvalues of A determine the qualitative behavior:

| Eigenvalues | Phase Portrait | Stability | |---|---|---| | Both negative real | Stable node (sink) | Asymptotically stable | | Both positive real | Unstable node (source) | Unstable | | Opposite signs | Saddle point | Unstable | | Complex, Re < 0 | Stable spiral | Asymptotically stable | | Complex, Re > 0 | Unstable spiral | Unstable | | Purely imaginary | Center (ellipses) | Stable (not asymptotic) |

Stability Analysis

For nonlinear system x' = f(x), linearize near equilibrium x* (where f(x*) = 0):

x' ≈ J(x*) · (x - x*)

where J is the Jacobian of f at x*.

Stability is determined by the eigenvalues of J(x*):

All eigenvalues have Re(λ) < 0 → asymptotically stable
Any eigenvalue has Re(λ) > 0 → unstable
All Re(λ) ≤ 0 with some Re(λ) = 0 → inconclusive (need nonlinear analysis)

Lyapunov stability: Find a Lyapunov function V(x) > 0 with V̇(x) < 0. If such V exists, the equilibrium is stable.

Introduction to PDEs

Partial differential equations involve multiple independent variables.

Heat Equation

∂u/∂t = α ∂²u/∂x²

Models heat diffusion, smoothing processes. Related to Gaussian blur in image processing.

Wave Equation

∂²u/∂t² = c² ∂²u/∂x²

Models vibrations, wave propagation, sound.

Laplace's Equation

∂²u/∂x² + ∂²u/∂y² = 0    (∇²u = 0)

Models steady-state heat, electrostatic potential, fluid potential. Solutions are harmonic functions.

Solution Methods

Separation of variables: Assume u(x,t) = X(x)T(t), reduce PDE to ODEs.
Fourier series: Expand solutions in orthogonal basis functions.
Numerical methods: Finite differences, finite elements, spectral methods (covered in numerical methods topic).

Applications in CS

Neural network training dynamics: Gradient flow ODE: dθ/dt = -∇L(θ). Continuous limit of gradient descent. Neural tangent kernel theory uses linearized dynamics.
Control systems: System behavior modeled by ODEs. Stability analysis determines if a controller works. Transfer functions use Laplace transforms.
Signal processing: LTI systems described by ODEs. Laplace/Z-transforms convert to algebraic operations. Filter design uses pole/zero analysis.
Physics simulation: Game engines solve ODEs for rigid body dynamics (Euler, RK4 integrators). Fluid simulation solves Navier-Stokes (PDEs).
Population models: Logistic equation, SIR epidemic model, predator-prey (Lotka-Volterra). All ODE systems.
Network dynamics: Epidemic spreading, rumor propagation, opinion dynamics on graphs.
Diffusion processes: Random walks, heat kernels on graphs, diffusion models in generative AI.
Chemical reaction networks: Mass-action kinetics give ODE systems.