From derivatives to constrained optimization, rebuilt chapter by chapter as interactive lessons. Golden section search, gradient descent, simulated annealing, and more — all with live simulations.
History, optimization process, mathematical formulation, minima, optimality conditions.
Derivatives, gradients, Hessians, numerical and automatic differentiation.
Unimodality, Fibonacci search, golden section search, quadratic fit, bisection.
Descent direction iteration, step factors, line search, trust regions.
Gradient descent, conjugate gradient, momentum, Adam, hypergradient descent.
Newton's method, secant method, quasi-Newton methods, BFGS.
Coordinate descent, Powell's method, Nelder-Mead simplex, DIRECT.
Noisy descent, simulated annealing, cross-entropy, CMA-ES.
Genetic algorithms, differential evolution, particle swarm, firefly algorithm.
Lagrange multipliers, penalty methods, interior point methods, projected descent.
Dual problem, primal-dual methods, dual ascent, ADMM.
Problem formulation, simplex algorithm, dual certificates.
Least squares, nonnegative least squares, dual certificates.
Canonical form, verification, canonicalization, solving.
Pareto optimality, constraint methods, weight methods, preference elicitation.
Full factorial, stratified sampling, space-filling, quasi-random sequences.
Linear models, basis functions, model selection, multifidelity.
Gaussian processes, prediction, fitting, noisy measurements.
Prediction-based exploration, expected improvement, safe optimization.
Set-based uncertainty, probabilistic uncertainty, robust optimization.