Bayes’ Rule

There is a rare disease: 1% of the population carries it, and the other 99% are healthy. Now there is a patient who takes the screening test for it, and the result comes back positive. Among people who carry the disease, the test comes back positive 95% of the time. Among healthy people, it comes back positive 10% of the time. How likely is it that this patient actually has the disease?

A positive result can happen in two ways. Either the patient carries the disease and the test caught it, or the patient is healthy and the test came back positive anyway:

• Carries the disease. 1% of the population carries it, and among these people the test comes back positive 95% of the time: $0.01 \times 0.95 = 0.0095$.
• Healthy. The other 99% are healthy, and among these people the test comes back positive 10% of the time: $0.99 \times 0.10 = 0.099$.

Each product is a share of the whole population: the first is the share of people who both carry the disease and test positive; the second is the share who are healthy and still test positive.

Shares of a population are easier to picture as people. Out of every 10,000 people, the first product, $0.0095 \times 10{,}000 = 95$, means 95 people carry the disease and test positive. The second, $0.099 \times 10{,}000 = 990$, means 990 people are healthy and test positive anyway. The 990 outnumber the 95 about ten to one: healthy people vastly outnumber sick ones, so even a 10% positive rate among the many healthy yields far more positive results than a 95% rate among the few sick. Every positive result is one or the other, so the probability that this patient has the disease is a plain count — 95 sick-and-positive people out of 1,085 positive results:

$$P(\text{disease} \mid +) = \frac{95}{1{,}085} \approx 8.8\%$$

These two kinds of positive results have names: a positive result on someone who actually has the disease is a true positive (the 95), and a positive result on someone who does not have it is a false positive (the 990) — informally, a false alarm. Read in these terms, the patient’s probability is true positives over all positive results.

Notice that the 95 sick-and-positive people answer three different questions, with three different answers, depending on what they are divided by:

Same 95 people — three different denominators. The patient’s case matches the middle row: the test result is known — positive — but the disease status is not. So the patient’s question is the middle one: given a positive result, how likely is the disease? Its answer is 8.8%. The 95% is a true fact about sick people, but it answers a question about a group the patient may not belong to.

Two different probabilities appear in this problem, and they condition on different groups. $P(+ \mid \text{disease})$ asks about the disease group: among people who carry the disease, what share tests positive? That is what the test specs tell us, 95%. $P(\text{disease} \mid +)$ asks about the positive group: among people who test positive, what share carries the disease? That is what the patient wants to know, 8.8%. Same two events, same two groups, but the conditioning runs in opposite directions. Bayes’ rule is the formula that converts one into the other. When you know the share of the observation inside each group, plus how large each group is, you can compute the share of each group inside the observation.

Now give every piece a name. The disease and healthy groups are mutually exclusive, so no one is both. They also cover all possibilities, so everyone is one or the other. In notation, this collection is called event set $A$. The disease group is $A_1$ with probability $P(A_1) = 0.01$. The healthy group is $A_2$ with $P(A_2) = 0.99$. With more scenarios we would add $A_3, \ldots, A_n$, each with its own $P(A_i)$. The test result is event set $B$. A positive result is $B_1$ with $P(B_1) = 0.1085$, which is the 1,085 out of 10,000 we already counted. A negative result would be $B_2$. Within the two $A$ groups, the positive result has its own shares. $P(B_1 \mid A_1) = 0.95$ means among the diseased, 95% test positive. $P(B_1 \mid A_2) = 0.10$ means among the healthy, 10% test positive. Bayes’ rule takes exactly these numbers and combines them into the answer we computed. It gives the share of group $A_1$ within all positive results, $P(A_1 \mid B_1) = 8.8\%$. The next section states this as a formula.

The Rule

Bayes’ rule answers one kind of question: we know the share of the observation within each event, and we want the share of each event within the observation. The formula converts the first direction into the second. Its format is:

$$P(A_i \mid B) = \frac{P(B \mid A_i) \cdot P(A_i)}{\sum_{j=1}^{n} P(B \mid A_j) \cdot P(A_j)}$$

Each item in the formula:

• $A_1, A_2, \ldots, A_n$ — the events of event set $A$, mutually exclusive and covering all possibilities. One of them is true, and exactly which one is what we are trying to find out.
• $B$ — the observed event.
• $P(A_i)$ — the share of event $A_i$ within the whole population: how much of the world this event covers. This is called the prior.
• $P(B \mid A_i)$ — the share of the observation within event $A_i$: how much of this event produces the observation. This is called the likelihood.
• $P(B \mid A_i) \cdot P(A_i)$ — the event’s contribution: its slice of the population that ends up in the observation.
• $\sum_{j=1}^{n} P(B \mid A_j) \cdot P(A_j)$ — the same contribution summed over all events, using the index $j$. Every event drops its slice into the observation, and this sum is all of it. It equals $P(B)$, the total share of the observation.
• $P(A_i \mid B)$ — the share of event $A_i$ within the observation, which is the event’s contribution divided by the total. This is called the posterior.

The formula is nothing more than that division: one event’s slice over the whole observation. When there are only two events, the sum has two terms, and it is often written in the single-event form:

$$P(A \mid B) = \frac{P(B \mid A) \cdot P(A)}{P(B)}$$

Derivation

Bayes’ rule follows from three tools we already know. Let’s derive it step by step.

Step 1: Start from the definition of conditional probability

By definition:

$$P(A_i | B) = \frac{P(A_i \cap B)}{P(B)}$$

This says: “the probability of $A_i$ given $B$ is the probability of both happening, divided by the probability of $B$.” The denominator normalizes — out of all ways $B$ can occur, what fraction also has $A_i$?

Step 2: Apply the multiplication rule to the numerator

The numerator $P(A_i \cap B)$ can be rewritten using the multiplication rule:

$$P(A_i \cap B) = P(B | A_i) \cdot P(A_i)$$

Why this form? Because $P(B | A_i)$ is usually given — it’s the “forward” probability (how likely the observation is under each scenario). The “reverse” probability $P(A_i | B)$ is what we’re trying to find.

Substituting into Step 1:

$$P(A_i | B) = \frac{P(B | A_i) \cdot P(A_i)}{P(B)}$$

Step 3: Apply the total probability theorem to the denominator

The denominator $P(B)$ can be expanded using the total probability theorem (since the $A_j$’s form a partition):

$$P(B) = \sum_j P(B | A_j) \cdot P(A_j)$$

Why expand? Because we don’t usually know $P(B)$ directly — we know the priors $P(A_j)$ and the likelihoods $P(B | A_j)$. The theorem lets us compute $P(B)$ from what we have.

Substituting into Step 2:

$$P(A_i | B) = \frac{P(B | A_i) \cdot P(A_i)}{\sum_j P(B | A_j) \cdot P(A_j)}$$

That’s Bayes’ rule. Three steps: definition → multiplication → total probability.

The full chain

$$\text{Conditional Definition} \;\rightarrow\; \text{Multiplication Rule} \;\rightarrow\; \text{Total Probability} \;\rightarrow\; \text{Bayes' Rule}$$

Each tool builds on the previous one. Bayes’ rule is the final step in this chain — it combines all three to reverse conditional probabilities.

Prior and Posterior

These three names describe the timeline of a Bayes calculation.

The prior is what you know before the observation. It is the group’s share of the population. The disease group has prior $P(A_1) = 0.01$. One in a hundred people carry the disease, and this patient could be anyone.

The likelihood is what the model tells you about the observation inside each group. It is the share of the group that shows the result. The likelihood of a positive result is 0.95 in the disease group and 0.10 in the healthy group.

The posterior is what you know after the observation. It is the group’s share within the observed results. The disease group has posterior $P(A_1 \mid B_1) = 0.088$. This patient tested positive, and among all positive results 8.8% come from the disease group.

The movement from prior to posterior is the update. The disease group started at 1% of the population. After one positive test it stands at 8.8% of the positive results. The observation moved the number, and it could only move it this far because the group was small to begin with. A rare group needs strong evidence, or repeated evidence, to reach a high posterior.

Example: Disease Testing

In the notation of the last section: event $A_1$ is carries the disease, event $A_2$ is healthy, and event $B$ is the test result. The priors are $P(A_1) = 0.01$ and $P(A_2) = 0.99$. The likelihoods are $P(B \mid A_1) = 0.95$ and $P(B \mid A_2) = 0.10$. The posterior we want is $P(A_1 \mid B)$: among all positive results, the share that comes from the disease event.

Plugging in:

$$P(A_1 \mid B) = \frac{0.95 \times 0.01}{0.95 \times 0.01 + 0.10 \times 0.99} = \frac{0.0095}{0.1085} \approx 8.8\%$$

The formula reproduces the by-hand count: the numerator is the disease event’s contribution, the denominator is all positive results.

$$P(A_1 \mid +) = \frac{0.95 \times 0.01}{0.95 \times 0.01 + 0.10 \times 0.99} = \frac{0.0095}{0.1085} \approx 8.8\%$$

The formula reproduces the by-hand count: the numerator is the sick-and-detected share, the denominator is all positive results.

The denominator (total positive probability) is 0.1085. Only 0.0095 of that comes from actual disease cases. The rest — 91% of positives — are false alarms from healthy people.

Key insight: When the prior is small (rare disease), even an accurate test produces mostly false positives. The posterior stays low because the prior was low — the observation can only update beliefs, not create them from scratch.

Example: Radar Detection

A radar detects airplanes:

• $P(\text{plane}) = 0.05$
• $P(\text{detect} \mid \text{plane}) = 0.99$
• $P(\text{detect} \mid \text{no plane}) = 0.10$

Given detection, probability of plane:

$$P(\text{plane} \mid \text{detect}) = \frac{0.99 \times 0.05}{0.99 \times 0.05 + 0.10 \times 0.95} \approx 34\%$$

Same pattern: false alarms dominate. Most detections come from empty sky, even though each empty-sky detection is only 10% likely.

Inference: Cause → Effect Reversed

Bayes’ rule is the foundation of inference — drawing conclusions from observations.

Causal Model: $A \rightarrow B$ (Cause produces effect)

• We know $P(B | A)$ — forward probability, how likely the effect given the cause
• This is usually given by the model (test specs, radar specs, etc.)

Inference Question: Given we observed $B$, what caused it?

• We want $P(A | B)$ — backward probability, how likely the cause given the effect
• This is what Bayes’ rule computes

This reversal — from causal direction to inference direction — is exactly what Bayes’ rule does. It lets you reason backwards from observations to causes.