5.1 Bayesian decision theory

Bayesian inference allows us to update our belief about hidden quantities $H$ based on observed data $x\in X$. To convert this inference into an action, we use Bayesian decision theory.

5.1.1 Basics

We assume an agent has a set of possible actions $\mathcal{A}$. Each of these actions have a cost and benefits, depending on some state $H \in\mathcal{H}$.

We can encode this information using a loss function $\ell(h, a)$, specifying the cost of taking action $a\in\mathcal{A}$ with state $h\in\mathcal{H}$.

In medical circles, a common measure to quantify the benefits of a drug is the quality-adjusted life years (QALY)

Once we have the loss function, we can compute the posterior expected loss or risk

The optimal policy specifies what action to take given observed data $x$

In the case of Covid, let say that $x$ is the age of each patient. We can convert test results into a distribution over disease states, and by combining it with the loss matrix above we can compute the optimal policy for each observation.

Let say we improve the precision of the test, the probability that a positive test is correct increases and the cost of no-op for elderly increases as well, thus changing the action to take into 1.

5.1.2 Classification problems

Given an observed input $x$, what is the optimal class label to predict?

Zero-one loss

Suppose the states are some classes $\mathcal{H}=\mathcal{Y}=\{1,...,C\}$, and the action correspond to class labels $\mathcal{A = Y}$.

Which corresponds to

In this case the posterior expected loss is

Hence the action that minimize the risk is the most probable label, that is the mode of the posterior distribution, aka the MAP estimate

Cost sensitive classification

Let’s now have the following loss function $\ell(y^*,\hat{y})$

Let $p_0=p(y^*=0|x)$ and $p_1=1-p_0$. We select $\hat{y}=0$ iff

when $\ell_{00}=\ell_{11}=0$ that simplifies to

And we have $\ell_{10}=c\, \ell_{01}$

So if a false negative cost 2 times as much as a false positive, our positive threshold is $1/3$.

The reject option

In some cases we would like to be able “I don’t know” instead of answering: that’s the reject option. The actions are $\mathcal{A}=\mathcal{Y} \cup \{0\}$ where 0 is the reject option.

The loss function is

And we choose the reject action whenever the highest probability is below $\lambda^*=1-\frac{\lambda_r}{\lambda_e}$. Otherwise choose the most probable class.

5.1.3 ROC curve

For any fixed threshold $\tau$ we consider the following

We can compute the number of false positives

By normalizing by row $p(\hat{y}|y)$ or by columns $p(y|\hat{y})$ we can derive various summary statistics.

We can plot the $TPR_\tau$ and $FPR_\tau$ as a function of $\tau$ (ROC curve) and the PR curve.

We summarize the ROC curve using its area under the curve (AUC), the best being 1 and the worst being 0.5 (random prediction).

PR curve can be summarized by taking the precision of the $K$ first recalled elements: this is the precision at $K$ score (P@k).

5.1.4 PR Curve

In rank retrieval problems, a PR curve can be non-monotonic (opens in a new tab). Suppose a classifier has a precision of 90% with recall of 10%, and a precision of 96% with a recall of 20%. We can compute the maximum precision (96%) we can achieve with at least a recall of $K$ (10%): this is the interpolated precision.

The average of the interpolated precisions (AP) is the area under the interpolated PR curve.

The mean average precision (mAP) is the mean of the AP over a set of different PR curves.

The $F_\beta$ score can be defined as follow for $\beta>0$:

Or

If we set $\beta=1$ we got the harmonic mean of the precision and recall:

The harmonic mean is more conservative than the arithmetic mean and it requires both precision and recall to be high.

Suppose the recall is very high and the precision is very low, we would have $\frac{\mathcal{P+R}}{2} \approx R/2 \approx50\%$

whereas $F_1 \approx\mathcal{2P.R/R\approx2P<< 50\%}$

Using the $F_1$ score weights precision and recall equally. If recall is more important we may use $\beta=2$, and if precision is more important we may use $\beta=0.5$.

Class imbalance doesn’t affect ROC curve, since the TPR and FPR are a class ratio.

However the RC curve is affected, since the precision can be written:

with $r=\frac{P}{N}$, so that $Prec_\tau \rightarrow 1$ when $N\rightarrow0$ and $Prec_\tau \rightarrow 0$ when $P\rightarrow0$

The F1 score is also affected, since:

5.1.5 Regression problems

In regression settings, the set of state and action space is $\mathcal{H=A}=\mathbb{R}$

L2 loss

The most common loss is the $\ell_2$ loss:

In this case, the risk is given by:

The optimal action reduce the risk at that point to 0, in this case the posterior mean:

L1 loss

$\ell_1$ loss is less sensitive to outliers than $\ell_2$ loss:

The optimal action is the posterior median, $a$ such that:

Huber loss

Another robust loss function:

where $r=h-a$

5.1.6 Probabilistic prediction problems

We now assume the set of possible actions is to pick a probability distribution over some value. The state is a distribution $h=p(Y|x)$ and the action is another $a=q(Y|x)$.

We want to pick $q$ to minimize $\mathbb{E}[\ell(p,q)]$ for a given $x$.

Kullback Leibler divergence

$\mathbb{H}(p)$ and $\mathbb{H}(p,q)$ are the entropy and cross-entropy.

$y$ is discrete, but this can be generalized to continuous values.

To find the optimal distribution to use, we minimize $D_{KL}(p||q)$ and as $\mathbb{H}(p)$ is constant wrt $q$, we only minimize the cross-entropy term:

If the state is degenerate and puts all its mass on a single outcome i.e. $h=p(Y|x)=\mathbb{I}(y=c)$, the cross entropy becomes:

This is the log loss of the predictive distribution $q$ when given target label $c$.

Proper scoring rule

The key property we desire is that the loss function is minimized iff the chosen distribution $q$ matches the true distribution $p$, ie $\ell(p,p)\leq \ell(p,q)$ $$, $\ell$ is the proper scoring rule.

The cross-entropy is a proper scoring rule since $D_{KL}(p||p)\leq D_{KL}(p||q)$, but the $\log p(y)/q(y)$ term is sensitive to errors for low probability events

A common alternative is the Brier score:

See this discussion on scikit-learn (opens in a new tab) about the benefit and drawbacks of Log loss vs Brier score.