The quest for the optimal is ubiquitous in nature and human behavior. The field of mathematical optimization has a long history and remains active today, particularly in the development of machine learning.Classical and Modern Optimization presents a self-contained overview of classical and modern ideas and methods in approaching optimization problems. The approach is rich and flexible enough to address smooth and non-smooth, convex and non-convex, finite or infinite-dimensional, static or dynamic situations. The first chapters of the book are devoted to the classical toolbox: topology and functional analysis, differential calculus, convex analysis and necessary conditions for differentiable constrained optimization. The remaining chapters are dedicated to more specialized topics and applications.Valuable to a wide audience, including students in mathematics, engineers, data scientists or economists, Classical and Modern Optimization contains more than 200 exercises to assist with self-study or for anyone teaching a third- or fourth-year optimization class.
Contents:
-
Topological and Functional Analytic Preliminaries:
- Metric Spaces
- Normed Vector Spaces
- Banach Spaces
- Hilbert Spaces
- Weak Convergence
- On the Existence and Generic Uniqueness of Minimizers
- Exercises
-
Differential Calculus:
- First-Order Differential Calculus
- Second-Order Differential Calculus
- The Inverse Function and Implicit Function Theorems
- Smooth Functions on ?d, Regularization, Integration by Parts
- Exercises
-
Convexity:
- Hahn-Banach Theorems
- Convex Sets
- Convex Functions
- The Legendre Transform
- Exercises
-
Optimality Conditions for Differentiable Optimization:
- Unconstrained Optimization
- Equality Constraints
- Equality and Inequality Constraints
- Exercises
-
Problems Depending on a Parameter:
- Setting and Examples
- Continuous Dependence
- Parameter-Independent Constraints, Envelope Theorems
- Parameter-Dependent Constraints
- Discrete-Time Dynamic Programming
- Exercises
-
Convex Duality and Applications:
- Generalities
- Convex Duality with Respect to a Perturbation
- Applications
- On the Optimal Transport Problem
- Exercises
-
Iterative Methods for Convex Minimization:
- On Newton's Method
- The Gradient Method
- The Proximal Point Method
- Splitting Methods
- Exercises
-
When Optimization and Data Meet:
- Principal Component Analysis
- Minimization for Linear Systems
- Classification
- Exercises
-
An Invitation to the Calculus of Variations:
- Preliminaries
- On Integral Functionals
- The Direct Method
- Euler-Lagrange Equations and Other Necessary Conditions
- A Focus on the Case d = 1
- Exercises
Readership: Thought-leaders, executives, industry strategists, research scientists, graduate students, advanced undergraduate students, policy-makers, research funding agencies, private research institutions, government regulators, investors, corporate managers, purchasing agents, and entrepreneurs in the areas of computer science, quantum computing, information theory, neuroscience, and physics.
Key Features:
- Most textbooks on optimization focus on a special type of problems. This book gives a self-contained overview (covering finite or infinite-dimensional, convex or nonconvex, smooth or non-smooth, static or dynamic problems, iterative methods ...)
- This self-contained book starts with a rigorous but flexible toolbox. It then develops more specialized topics and applications: problems depending on a parameter, dynamic programming in discrete and continuous time, calculus of variations, convex duality (including duality for linear and SD programming), optimal transport, iterative convex minimization methods, applications of optimization to data sciences
- The book contains more than 200 exercises which may also make it useful to anyone teaching a third/fourth year optimization class
- The book contains some results which are important and useful but not easy to find in textbooks (e.g. generic uniqueness of extremizers, Ekeland's local surjection theorem, a detailed treatment of first and second-order optimality conditions for constrained minimizers in Banach spaces, the so-called envelope theorem, the subdifferential of a supremum of convex functions, Birkhoff-von Neumann theorem, Nesterov's accelerated gradient method, convergence of the Douglas-Rachford algorithm, sparse solutions of basis pursuit ...)
