Taylor expansion

The Taylor series of a real or complex-valued function $f(x)$ that is infinitely differentiable at a real or complex number $a$ is the power series ^[1]

$$\sum _{n=0}^{\infty }\frac{f^{(n)}(a)}{n!}(x-a{)}^n=f(a)+\frac{f^{\prime }(a)}{1!}(x-a)+\frac{f^{\prime \prime }(a)}{2!}(x-a{)}^2+\frac{f^{\prime \prime \prime }(a)}{3!}(x-a{)}^3+\dots$$

The taylor polynomial for any polynomial is the polynomial itself. When $a=0$ the series is also called the a Maclaurin series. The series is very useful to approximate a function at $a$ as a polynomial. Here we approximate $\text{sin}$ at $a=0$

$$\text{sin}(x)\approx x-\frac{x^3}{3!}+\frac{x^5}{5!}-\frac{x^7}{7!}+O(x^9)$$

where $O(x^9)$ is the error term.

Differentiability class

The function $f$ is said to be of differentiability class $C^k$ if the derivatives $f^{(1)},f^{(2)},\dots ,f^{(k)}$ exist and are continous.

Gradient

Suppose that $f:{\mathbb{R}}^n\to \mathbb{R}$ is a function and that $f\in C^1$ we can define the gradient $\nabla f$ as

$$\nabla f=\left[\begin{array}{c}\frac{\partial f}{\partial x_1}\\ \frac{\partial f}{\partial x_2}\\ ⋮\\ \frac{\partial f}{\partial x_n}\end{array}\right]$$

Hessian

Suppose that $f:{\mathbb{R}}^n\to \mathbb{R}$ is a function taking as input a vector $\mathbf{x}\in {\mathbb{R}}^n$ and output a scalar $f(\mathbf{x})\in \mathbb{R}$. If $f\in C^2$ then the Hessian matrix $\mathbf{H}$ of $f$ is a square $n\times n$ matrix defined as follows

$$\mathbf{H}=\left[\begin{array}{cccc}\frac{{\partial }^{2}f}{\partial x_1^2}& \frac{{\partial }^{2}f}{\partial x_1\partial x_2}& \cdots & \frac{{\partial }^{2}f}{\partial x_1\partial x_n}\\ \frac{{\partial }^{2}f}{\partial x_2\partial x_1}& \frac{{\partial }^{2}f}{\partial x_2^2}& \cdots & \frac{{\partial }^{2}f}{\partial x_2\partial x_n}\\ ⋮& ⋮& \ddots & ⋮\\ \frac{{\partial }^{2}f}{\partial x_n\partial x_1}& \frac{{\partial }^{2}f}{\partial x_n\partial x_2}& \cdots & \frac{{\partial }^{2}f}{\partial x_n^2}\end{array}\right]$$

Matrix calculus

https://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf

Jensen's inequality

https://en.wikipedia.org/wiki/Jensen%27s_inequality

$$f(t{x}_1+(1-t)x_2)\le tf(x_1)+(1-t)f(x_2).$$ $$\phi (E[X])\le E\left[\phi (X)\right]$$

References