A primer on Bayes

This article is the first of the Bayesian series.

You probably already know the Bayes theorem or stumble upon Naive Bayes when comparing machine learning models. Maybe, like me, you feel that you barely saw the shores of the continent that is the Bayesian land and want to set foot in it.

I wrote this article from a Stanford course with the intent for you to understand the relationships between all Bayesian concepts without having to go through an entire classroom (like I did). As a matter of readability, I hide most of the maths in dropdowns. This way, you can follow the logic easily and come back to deep dive into formulas later.

We start with comparing frequentist and bayesian methods, before focusing on different distributions and conjugate families.

Suppose we want to estimate the probability of heads of a (possibly biased) given coin. We flip the coin N times, generating a sequence of observations $(x_1 , ..., x_N )$ of an iid random variable

$(X_1 , ..., X_N)$, each of which has the Bernoulli distribution with unknown head probability $\theta$.

The fundamental notion here is that randomness comes from sampling/replicates of the experiment, and all probability statements made in frequentist inference are statements with respect to the probability distribution induced by hypothetical repetitions of the experiment.

This leads us to the basic program of frequentist inference: point estimation, interval estimation, and testing.

The laws of large numbers imply that the sample mean is in some sense a “good” frequentist estimator for the expectation of a random variable.

We toss a coin N times and use the sample mean of the random variable $X = \bold{1}_{heads}$ as an estimator of the probability of heads $\theta$. Is it a good estimator?

The sample mean is the optimal unbiased estimator in the sense that among all unbiased estimators, it has minimum variance —thus minimum MSE, see Rao-Blackwell theorem. The point is that the sample mean is a “good” estimator of $\theta$ because in a hypothetical infinite sequence of experiments just like the one we performed, it will typically give us a number that is close to $\theta$.

Point estimation is an exercise in producing a single number that is in some sense a “best guess” at the value of a parameter $\theta$. Interval estimation aims to produce a range of values that are “plausible”.

What would an exact $1 − α$ confidence interval look like for θ for N coin tosses?

Let $\phi$ be the standard Gaussian CDF and $\hat{SE}$ the standard error of $\hat{\theta}$, $\hat{SE}(\hat{\theta})=\frac{\sigma_{\hat{\theta}}}{\sqrt{N}}$​

However this approach is too heavy in computation. Instead, we consider intervals that have asymptotic $1 − α$ coverage, using the central limit theorem of the general form:

$\hat{\theta} \pm \Phi^{-1}(1-\alpha/2)\hat{SE}(\hat{\theta})$​

We want to test whether the coin is fair through hypothesis:

$H_0: \theta = 1/2$ and $H_1: \theta ≠ 1/2$.

The classical way is to use Neyman’s fixed type I error rate testing. We reject $H_0$ when our estimate $\hat{{\theta}}$ is far from 1/2. We use the Binomial distribution of $\hat{\theta}$ to find c (according to our chosen type I error rate) such as:

$|θ̂ − 1/2| > c$​

There are two components here.

$p(x_1,..., x_N|\theta)=\prod^N_{i=1}f_{\theta}(x_i)=\theta^{S_N}(1-\theta)^{N-S_N}$​

i) Informative prior choice. An informative prior choice would likely put a lot of prior mass near 1/2. However, exactly how much mass to place is tricky. We could try to elicit a prior from “experts” (I guess gamblers?).

ii) Objective prior choice. Prior that will have the smallest possible impact on the outcome of our analysis. Properties close to frequentist, but a poor choice in high-dimensional and non-parametric models.

iii) Empirical prior choice. Estimate the prior from the data, then plug the prior and perform a Bayesian Analysis. Advantage: No somewhat arbitrary prior choice. Shortcoming: Lies in the grey area between the “safety” of frequentist guarantees and that of Bayesian analysis using informative priors based on real prior info. Nice use-case:

These two components allow us to define the posterior:

$p(\theta|{x})=\frac{p({x}|\theta)p(\theta)}{\int p({x}|\theta)p(\theta)d\theta}=\frac{p({x}|\theta)p(\theta)}{p({x})}$​

Bayes’ theorem tells us how we should update our prior beliefs about parameters after observing data distributed according to the likelihood.

A very common choice is to pick a conjugate prior: posterior and prior belong to the same family of probability distributions, so that the marginal likelihood p(x) is often available analytically. For our coin-tossing example, with the N Bernoulli likelihood, our prior is in the Beta family.

If the posterior contains everything we want to know about parameters, then we must be able to use it to construct point estimates of the parameters. The posterior is a distribution, and parameters are numbers, so point estimates must be maps from the space of distributions to real numbers (i.e. expectations).

We find the minimum of the integrated mean squared error (IMSE) and even MSE by using the posterior expectation (called the Bayes estimator) as our point estimate.

$θ̂_B=𝔼[p(θ|x)]=∫θp(θ|x)dθ$​

One obvious way to construct an interval is to use quantiles of the posterior. An equal tailed interval leaves equal posterior probability to the left and to the right of the endpoints of the interval.

Our interval satisfies

$P(θ ∈ [a, b]) = 1 − α$​

where the probability here is the posterior probability not the probability with respect to hypothetical repeated sampling.

We again need to be able to compute everything using $p(θ|x)$. Hypotheses are subsets of the parameter space – in this case, the subset is just $\theta=\{1/2\}$.

If the posterior distribution is continuous, then the posterior probability of the null hypothesis is

$P(\theta=1/2)=0$​

So we need to give nonzero positive probability to the null hypothesis by choosing a mixture prior.

$p(θ)=qδ(θ − 1/2)+(1 − q)f(θ)$​

where $f(θ)$ is the density of a $Beta(a,b)$ distribution and $q\in[0,1]$. The left part is associated with $H_0$ and the right with $H_1$.

This is the first thing that might seem odd: in order to carry out the standard Bayes hypothesis test, I have to change my prior.

Thus, unlike point estimation, where Bayesians and frequentists mostly agree, and interval estimation, where typically the Bayesian credible intervals are fairly similar to frequentist confidence intervals unless a strong prior is chosen, Bayesian hypothesis tests can reach very different conclusions from frequentist ones. In this case, the Bayesian finds less evidence against the null than the frequentist does.

One of the most commonly used in applications is Jeffreys prior. It is defined as

$p(\theta) \propto |I(\theta)|^{-1/2}$​

where I(θ) is the Fisher information matrix.

Jeffrey prior is invariant under one-to-one reparametrizations which means that if we choose to parametrize the model in terms of

$μ=g(θ)$​

then the prior will still take the form

$|I(μ)|^{1/2}$​

Suppose we observe $N$ iid data points $x_1, ..., x_N$ from a $Poisson(θ)$ distribution. It has the form of a Gamma distribution, so we choose a prior with the same distribution family.

Which is:

$Gamma(a,b)=\frac{b^a}{\Gamma(a)}\theta^{a-1}e^{-b\theta}$​

The likelihood of the normal distribution looks like the kernel of a Gamma distribution, so we define the prior this way. It is, in fact, the normal-inverse gamma distribution.

The conjugate prior of the multinomial likelihood is a multivariate of the Beta distribution and is called Dirichlet distribution.

Thank you for reading, I hope this guide has been helpful for you so far. Now that you took a bite of crunchy Bayesians, we will spice things up in the next article with high-dimension.