Random variable

A random variable is a variable whose outcome depends on a random event. In probability theory a random variable is understood as a measurable function defined on the probability space . A random variable maps from the sample space to any measurable space with some probability.

Probability mass function

The probability mass function, also known as the discrete density function, is a function that gives the exact probability of a discrete random variable to some value. It differs from the probability density function in that it is associated with discrete random variables instead of continuous random variables.

Probability density function

The probability density function (PDF) must be integrated over an interval to yield the probability. It is defined as follows

$$\mathrm{Pr}(a\le X\le b)={\int }_a^b{f}_X(x)dx$$

In the continuous case the probability of a point always gives the probability 0, $\mathrm{Pr}(X=x)=0$, which is why we need to evaluate it over an interval instead.

Stochastic process

A stochastic process is a random process that is usually defined as a family of random variables. Thus, each random variable takes the value from the same mathematical space known as the state space . There are two types of stochastic processes, that is, discrete-time and continuous-time stochastic processes. Examples of stochastic processes are the Bernoulli process ^[1] and random walk among others. The Bernoulli process can be looked as flipping a coin multiple times where the sequence of flipped coins represents several independent and identically distributed (i.i.d) Bernoulli random variables.

Statistical inference

Statistical inference is the process of inferring properties of an underlying distribution of probability using data analysis. Creating logical claims that is justified by the data.

Classical inference

In classical inference (Frequentist) parameters are fixed or non-random quantities and the probability only concerns the data. For a Frequentist the probability of an event is the proportion of that event in the long run.

Bayesian inference

Bayesian inference is a method used to update the probability of a model using Bayes' theorem

$$\mathrm{Pr}(A\mid B)=\frac{\mathrm{Pr}(B\mid A)\mathrm{Pr}(A)}{\mathrm{Pr}(B)}$$

Contrary to how classical inference work, Bayesian inference take into account the uncertainty of the parameters when creating the model. The parameters themselves are random variables. The Bayesian approach bases its decision on prior knowledge.

Kolmogorov axioms

The Kolmogorov axioms consist of of three axioms that is the foundation of probability theory.

First axiom

The probability of an event is always positive.

$$\mathrm{Pr}(E)\in {\mathbb{R}}_{+},\forall E\in F$$

where $F$ is the event space.

Second axiom

The probability that at least one of outcomes in the sample space will occur has the probability of 1.

$$\mathrm{Pr}(\Omega )=1$$

where $\Omega$ is the sample space.

Third axiom

Any countable sequence of mutually exclusive events $E_1,E_2,...$ satisfies

$$\mathrm{Pr}\left(\bigcup _{i=1}^{\infty }{E}_i\right)=\sum _{i=1}^{\infty }\mathrm{Pr}(E_i).$$

Conditional probability

The conditional probability of event $A$ occurring after event $B$ is defined as

$$\mathrm{Pr}(A\mid B)=\frac{\mathrm{Pr}(A\cap B)}{\mathrm{Pr}(B)}$$

Independent events

Two events are independent if

$$\mathrm{Pr}(A\cap B)=\mathrm{Pr}(A)\mathrm{Pr}(B)$$

Thus the following holds for conditional independent events

$$\mathrm{Pr}(A\mid B)=\frac{\mathrm{Pr}(A\cap B)}{\mathrm{Pr}(B)}=\frac{\mathrm{Pr}(A)\mathrm{Pr}(B)}{\mathrm{Pr}(B)}=\mathrm{Pr}(A)$$

Total law of probability

Given an event $A$, what is the probability of $A$ given $B$ every single $B$? The total law of probability states that if we have a sequence of events $B_n$ that partitions the sample space the following holds

$$P(A)=\sum _{n}P(A\cap B_n)=\sum _{n}P(A\mid B_n)P(B_n)$$

Joint distributions

Given multiple different random variables $X_1,\dots ,X_n$ defined on the same probability space is a probability distribution that gives the probability that each random variable falls into a particular set of values.

$$p_{X_1,\dots ,X_n}(x_1,\dots ,x_n)=\mathrm{Pr}(X_1=x_1,\dots ,X_n=x_n)$$

It could be written in terms of conditional probabilities with the chain rule property

$$\begin{array}{rl}{p}_{X_1,\dots ,X_n}(x_1,\dots ,x_n)& =\mathrm{Pr}(X_1=x_1)\cdot \mathrm{Pr}(X_2=x_2\mid X_1=x_1)\\ & \cdot \mathrm{Pr}(X_3=x_3\mid X_1=x_1,X_2=x_2)\\ & \dots \\ & \cdot \mathrm{Pr}(X_n=x_n\mid X_1=x_1,X_2=x_2,\dots ,X_{n-1}=x_{n-1})\end{array}$$

Chain rule

The chain rule of probabilities can be described by the following example

$$\begin{array}{rl}\mathrm{P}\mathrm{r}(X_4,X_3,X_2,X_1)& =\mathrm{P}\mathrm{r}(X_4\mid X_3,X_2,X_1)\cdot \mathrm{P}\mathrm{r}(X_3,X_2,X_1)\\ & =\mathrm{P}\mathrm{r}(X_4\mid X_3,X_2,X_1)\cdot \mathrm{P}\mathrm{r}(X_3\mid X_2,X_1)\cdot \mathrm{P}\mathrm{r}(X_2,X_1)\\ & =\mathrm{P}\mathrm{r}(X_4\mid X_3,X_2,X_1)\cdot \mathrm{P}\mathrm{r}(X_3\mid X_2,X_1)\cdot \mathrm{P}\mathrm{r}(X_2\mid X_1)\cdot \mathrm{P}\mathrm{r}(X_1)\end{array}$$

Expectation

Expectation is the expected value a distribution takes on, the most common outcome.

Discrete

$$E[X]=\sum _{x}x\mathrm{Pr}(X=x)$$

Continuous

$$E[X]={\int }_{x}x\mathrm{Pr}(X=x)dx$$

Conditional discrete

$$E[X\mid Y=y]=\sum _{x}x\mathrm{Pr}(X=x\mid y)$$

Conditional continuous

$$E[X\mid Y=y]={\int }_{x}x\mathrm{Pr}(X=x\mid y)dx$$

Total law of expectation discrete

$$E[X]=\sum _{y}E[X\mid Y=y]\mathrm{Pr}(X=x)$$

Total law of expectation continuous

$$E[X]={\int }_{y}E[X\mid Y=y]\mathrm{Pr}(X=x)dy$$

In both the discrete and the continuous case they could be written as

$$E[X]=E[E[X\mid Y]]$$

Linearity of expectation

Linearity of expectation is a property that states that the expected value of the sum of random variables is equal to individually sum the expectation of each random variable regardless if they are independent.

$$E[X_1+\cdots +X_n]=E[X_1]+\cdots +E[X_n]$$

More generally the following holds

$$E[\sum _{i=1}^n{c}_i{X}_i]=\sum _{i=1}^n(c_i\cdot E[X_i])$$

Variance

Variance is defined as

$$\text{Var}(X)=E[X^2]-E[X{]}^2$$

Total law of variance

$$\text{Var}(X)=E[\text{Var}(X\mid Y)]+\text{Var}(E[X\mid Y])$$

Covariance

Covariance is defined as

$$\begin{array}{rl}\mathrm{cov}(X,Y)& =E\left[\left(X-E\left[X\right]\right)\left(Y-E\left[Y\right]\right)\right]\\ & =E\left[XY-XE\left[Y\right]-E\left[X\right]Y+E\left[X\right]E\left[Y\right]\right]\\ & =E\left[XY\right]-E\left[X\right]E\left[Y\right]-E\left[X\right]E\left[Y\right]+E\left[X\right]E\left[Y\right]\\ & =E\left[XY\right]-E\left[X\right]E\left[Y\right],\end{array}$$

However this is susceptible to catastrophic cancellation ^[2] , which means that subtracting good approximations of two nearby numbers may yield a bad approximation to the difference of the original numbers.

Correlation

Correlation is defined as

$${\rho }_{X,Y}=\mathrm{corr}(X,Y)=\frac{\mathrm{cov}(X,Y)}{{\sigma }_X{\sigma }_Y}=\frac{E[(X-{\mu }_X)(Y-{\mu }_Y)]}{{\sigma }_X{\sigma }_Y}$$

where ${\mu }_X=E[X]$, ${\mu }_Y=E[Y]$, ${\sigma }_X$ and ${\sigma }_Y$ represents the standard deviation.

Order statistics

The kth order statistic of a statistical sample is equal to its kth-smallest value.

Hoeffding inequality

https://en.wikipedia.org/wiki/Hoeffding%27s_inequality

Hoeffding inequality states an upper bound on the probability that the sum

$$\begin{array}{rl}P\left(S_n-\mathrm{E}\left[S_n\right]\ge t\right)& \le \exp \left(-\frac{2t^2}{{\sum }_{i=1}^n(b_i-a_i{)}^2}\right)\\ P\left(\left|S_n-\mathrm{E}\left[S_n\right]\right|\ge t\right)& \le 2\exp \left(-\frac{2t^2}{{\sum }_{i=1}^n(b_i-a_i{)}^2}\right)\end{array}$$

Boole's inequality

https://en.wikipedia.org/wiki/Boole%27s_inequality

Boole's inequality is also known as the union bound. It states that for any finite set of events the probability that at least one of the events happens is no greater than the sum of the probabilities of the individual events.

$$\mathbb{P}\left(\bigcup _{i=1}^{\infty }{A}_i\right)\le \sum _{i=1}^{\infty }\mathbb{P}(A_i).$$

References