Probability basics therefore provide the foundation for Bayesian reasoning.
3.15.35Probability Using Python
Python's standard library and scientific libraries can be used to work with probabilities.
For example:
import random
heads = 0
trials = 10000
for _ in range(trials):
if random.choice(["H", "T"]) == "H":
heads += 1probability = heads / trials
print(probability)
The result should be close to:
\[0.5\]
because a fair coin has:
\[P(H)=0.5\]
This demonstrates the law of large numbers experimentally.
3.15.36Law of Large Numbers
The Law of Large Numbers states that as the number of independent trials increases, the observed average tends to approach the expected value.
For a fair coin:
\[P(H)=0.5\]
- If we toss the coin:
- 10 times → result may be far from 50%
- 100 times → usually closer
- 10,000 times → typically much closer
- For example:
\[\frac{\text{Number of Heads}}{\text{Total Tosses}}
\rightarrow0.5\]
as the number of trials becomes very large under the usual assumptions.
This concept is important in statistical estimation and machine learning.
3.15.37Probability and Data Science
Probability provides the foundation for many data-science concepts:
\[\boxed{
\text{Probability}
\rightarrow
\text{Statistics}
\rightarrow
\text{Machine Learning}
}\]
- It is used for:
- Uncertainty estimation
- Hypothesis testing
- Statistical inference
- Classification
- Regression
- Bayesian models
- Probabilistic graphical models
- Generative models
- Risk prediction
3.15.38Important Probability Formulas
Basic Probability
\[\boxed{
P(A)=
\frac{\text{Favorable outcomes}}
{\text{Total outcomes}}
}\]
Complement
\[\boxed{
P(A^c)=1-P(A)
}\]
Addition Rule
[
\boxed{
P(A\cup B)
P(A)+P(B)-P(A\cap B)
}
]
Conditional Probability
[
\boxed{
P(A\mid B)
\frac{P(A\cap B)}
{P(B)}
}
]
Joint Probability
[
\boxed{
P(A\cap B)
P(A\mid B)P(B)
}
]
Independent Events
\[\boxed{
P(A\cap B)=P(A)P(B)
}\]
Expected Value
\[\boxed{
E[X]=\sum_xxP(X=x)
}\]
Variance
[
\boxed{
\operatorname{Var}(X)
E[X^2]-(E[X])^2
}
]
Standard Deviation
\[\boxed{
\sigma=\sqrt{\operatorname{Var}(X)}
}\]
3.15.39Common Mistakes
Mistake 1: Probability greater than 1
Invalid:
\[P(A)=1.5\]
Valid:
\[\boxed{0\leq P(A)\leq1}\]
Mistake 2: Confusing (P(A\cap B)) and (P(A\cup B))
\[A\cap B\]
\[A\cup B\]
- means:
- A or B.
- Mistake 3: Assuming events are independent
You cannot automatically assume:
\[P(A\cap B)=P(A)P(B)\]
The events must actually be independent.
Mistake 4: Confusing conditional probability
These are generally different:
\[P(A\mid B)\]
and:
\[P(B\mid A)\]
- This distinction is extremely important and leads directly to Bayes' theorem.
- Mistake 5: Confusing PDF with probability
- For continuous variables:
\[f(x)\]
is a density, not generally the probability that (X=x).
Probability over an interval is obtained by integrating:
[
P(a\leq X\leq b)
\int_a^b f(x),dx
]
3.15.40Probability Concepts for AI/ML
| Concept | AI/ML Application |
|---|
| Probability | Uncertainty |
| Conditional probability | Prediction given evidence |
| Joint probability | Relationships between variables |
| Independence | Model assumptions |
| Random variable | Representing uncertain quantities |
| PMF | Discrete outcomes |
| PDF | Continuous variables |
| CDF | Cumulative probability |
| Expected value | Average/expected outcome |
| Variance | Data uncertainty/spread |
| Standard deviation | Scale of variation |
| Bayes theorem | Bayesian inference |
| Probability distributions | Statistical modeling |
3.15.41AI/ML Example: Customer Churn
Suppose an ML system predicts whether a customer will leave.
Let:
\[C=\text{Customer churns}\]
Suppose the model estimates:
\[P(C)=0.20\]
So there is a 20% baseline probability of churn.
Now suppose we observe:
\[D=\text{Customer has reduced usage}\]
We may want to calculate:
\[P(C\mid D)\]
This asks:
What is the probability that the customer will churn given that their usage has decreased?
This is conditional probability and leads naturally to Bayesian modeling.
3.15.42AI/ML Example: Spam Detection
Suppose:
\[S=\text{Email is spam}\]
and:
\[W=\text{Email contains a suspicious word}\]
We might want:
\[P(S\mid W)\]
A probabilistic spam classifier estimates this probability using evidence contained in the email.
This is one of the classic applications of probability in machine learning.
3.15.43The Probability → ML Connection
The progression of this module is:
\[\boxed{
\text{Probability Basics}
\rightarrow
\text{Bayes Theorem}
\rightarrow
\text{Probability Distributions}
\rightarrow
\text{Statistical Modeling}
\rightarrow
\text{Machine Learning}
}\]
Probability allows an AI system to represent uncertainty.
For example:
\[P(\text{Fraud}\mid\text{Transaction Data})=0.92\]
means the model estimates a 92% probability of fraud given the observed information.
3.15.44Key Takeaways
Probability provides a mathematical framework for uncertainty.
The most important concepts are:
\[\boxed{
0\leq P(A)\leq1
}\]
\[\boxed{
P(A^c)=1-P(A)
}\]
[
\boxed{
P(A\cup B)
P(A)+P(B)-P(A\cap B)
}
]
[
\boxed{
P(A\mid B)
\frac{P(A\cap B)}{P(B)}
}
]
For independent events:
\[\boxed{
P(A\cap B)=P(A)P(B)
}\]
Expected value:
\[\boxed{
E[X]=\sum_xxP(X=x)
}\]
Variance:
[
\boxed{
\operatorname{Var}(X)
E[X^2]-(E[X])^2
}
]
AI/ML Connection
Probability is essential because machine-learning models deal with uncertainty.
A model may predict:
\[P(\text{Cat}\mid X)=0.90\]
\[P(\text{Dog}\mid X)=0.08\]
\[P(\text{Other}\mid X)=0.02\]
Instead of simply saying "Cat," the model can represent its uncertainty.
The overall mathematical progression is:
\[\boxed{
\text{Probability}
\rightarrow
\text{Conditional Probability}
\rightarrow
\text{Bayes Theorem}
\rightarrow
\text{Distributions}
\rightarrow
\text{Statistical Inference}
\rightarrow
\text{Machine Learning}
}\]
The next topic, 3.16 Bayes Theorem, builds directly on conditional and joint probability and explains how AI systems can update their beliefs when new evidence becomes available.
Bayes Theorem is a fundamental concept in probability that describes how we can update the probability of an event when new evidence becomes available.
- In simple terms:
- Bayes Theorem tells us how to update our belief about something after observing new information.
- It is extremely important in:
- Artificial Intelligence
- Machine Learning
- Medical diagnosis
- Spam detection
- Fraud detection
- Risk analysis
- Recommendation systems
- Bayesian inference
- Natural Language Processing
3.16.1Bayes Theorem Formula
genui{"learning_viz":{"type_id":"BAYES_THEOREM"}}
The fundamental formula is:
[
\boxed{
P(A\mid B)
\frac{P(B\mid A)P(A)}
{P(B)}
}
]
- Where:
- (P(A\mid B)) = Posterior probability
- (P(B\mid A)) = Likelihood
- (P(A)) = Prior probability
- (P(B)) = Evidence
A useful way to remember it is:
[
\boxed{
\text{Posterior}
\frac{\text{Likelihood}\times\text{Prior}}
{\text{Evidence}}
}
]
3.16.2Understanding the Terms
Suppose:
\[A=\text{Patient has a disease}\]
and:
\[B=\text{Test result is positive}\]
Then:
Prior
\[P(A)\]
is the probability that the patient has the disease before seeing the test result.
Likelihood
\[P(B\mid A)\]
is the probability of getting a positive test if the patient actually has the disease.
Evidence
\[P(B)\]
is the overall probability of receiving a positive test.
Posterior
\[P(A\mid B)\]
is the probability that the patient has the disease after seeing the positive test.
3.16.3The Main Idea
Bayes Theorem follows this reasoning:
\[\boxed{
\text{Initial Belief}
+
\text{New Evidence}
\rightarrow
\text{Updated Belief}
}\]
Mathematically:
\[\boxed{
\text{Prior}
+
\text{Evidence}
\rightarrow
\text{Posterior}
}\]
This concept is fundamental to Bayesian reasoning.
3.16.4Simple Example
- Suppose a factory produces components.
- Assume:
- 1% of components are defective.
- A test detects a defective component 95% of the time.
- The test has a 5% false-positive rate.
- Let:
\[D=\text{Defective}\]
and:
\[P=\text{Positive Test}\]
We know:
\[P(D)=0.01\]
\[P(P\mid D)=0.95\]
and:
\[P(P\mid D^c)=0.05\]
We want:
\[P(D\mid P)\]
3.16.5Calculate the Evidence
A positive test can happen in two ways:
The component is defective and tests positive.
The component is not defective but tests falsely positive.
Therefore:
[
P(P)
P(P\mid D)P(D)
+
P(P\mid D^c)P(D^c)
]
We know:
\[P(D^c)=1-0.01=0.99\]
Therefore:
[
P(P)
(0.95)(0.01)
+
(0.05)(0.99)
]
\[=0.0095+0.0495\]
\[\boxed{P(P)=0.059}\]
3.16.6Apply Bayes Theorem
Now:
[
P(D\mid P)
\frac{P(P\mid D)P(D)}
{P(P)}
]
Substitute:
[
P(D\mid P)
\frac{(0.95)(0.01)}
{0.059}
]
[
\frac{0.0095}{0.059}
]
\[\approx0.161\]
Therefore:
\[\boxed{
P(D\mid P)\approx16.1%
}\]
This is an important result.
Even though the test is fairly accurate, a positive result does not mean there is a 95% chance that the component is defective.
The reason is the relatively low base rate of defective components.
3.16.7Why Base Rates Matter
Suppose we test:
\[10,000\]
components.
Approximately:
\[1%=100\]
are defective.
Of those 100 defective components:
\[95%=95\]
will test positive.
Now there are:
\[9,900\]
non-defective components.
At a 5% false-positive rate:
\[9,900(0.05)=495\]
non-defective components will also test positive.
Therefore, total positive tests:
\[95+495=590\]
Actual defective components among positives:
\[95\]
Therefore:
[
P(D\mid P)
\frac{95}{590}
]
\[\approx16.1%\]
This is the same result obtained using Bayes Theorem.
3.16.8Bayes Theorem Using a Probability Tree
The reasoning can be represented as:
Components
/ \
Defective Not Defective
1% 99%
| |
Positive 95% Positive 5%
| |
0.95% 4.95%
Among all positive results:
\[0.95%+4.95%=5.90%\]
Only:
\[0.95%\]
are genuinely defective.
Therefore:
\[\frac{0.95}{5.90}
\approx16.1%\]
3.16.9Prior Probability
The prior probability represents our belief before observing new evidence.
It is:
\[\boxed{P(A)}\]
For the factory example:
\[P(D)=0.01\]
This means that before looking at the test result, we believe that approximately 1% of components are defective.
- The prior often comes from:
- Historical data
- Previous observations
- Domain knowledge
- Population statistics
3.16.10Likelihood
The likelihood is:
\[\boxed{P(B\mid A)}\]
- It answers:
- If (A) is true, how likely is the observed evidence (B)?
- In the example:
\[P(P\mid D)=0.95\]
This means:
If the component is defective, there is a 95% probability that the test is positive.
3.16.11Evidence
The evidence is:
\[\boxed{P(B)}\]
It represents the overall probability of observing the evidence.
Using the law of total probability:
[
\boxed{
P(B)
P(B\mid A)P(A)
+
P(B\mid A^c)P(A^c)
}
]
This denominator ensures that the posterior is properly normalized.
3.16.12Posterior Probability
The posterior is:
\[\boxed{P(A\mid B)}\]
- It answers:
- After seeing the evidence, what should we now believe about (A)?
- In the factory example:
\[P(D\mid P)=16.1%\]
The prior was:
\[P(D)=1%\]
After observing a positive test, the probability increased to:
\[16.1%\]
This is Bayesian updating.
3.16.13Bayes Theorem in Machine Learning
Bayes Theorem is particularly important in probabilistic machine learning.
Suppose:
\[C=\text{Class}\]
and:
\[X=\text{Observed features}\]
We want:
\[P(C\mid X)\]
Bayes Theorem gives:
[
\boxed{
P(C\mid X)
\frac{P(X\mid C)P(C)}
{P(X)}
}
]
This allows us to estimate the probability of a class after observing input features.
3.16.14Example: Spam Detection
Suppose:
\[S=\text{Spam}\]
and:
\[W=\text{Email contains suspicious words}\]
We want:
\[P(S\mid W)\]
Bayes Theorem gives:
[
\boxed{
P(S\mid W)
\frac{P(W\mid S)P(S)}
{P(W)}
}
]
- The model combines:
- Prior probability of spam
- Probability of seeing the suspicious word in spam
- Overall probability of seeing the word
to estimate the probability that the email is spam.
This idea forms the foundation of Naive Bayes classifiers.
3.16.15Naive Bayes
Naive Bayes is a probabilistic machine-learning algorithm based on Bayes Theorem.
Suppose an email contains features:
\[X_1,X_2,\ldots,X_n\]
We want:
\[P(C\mid X_1,X_2,\ldots,X_n)\]
Naive Bayes makes a simplifying conditional-independence assumption:
\[P(X_1,\ldots,X_n\mid C)
\approx
\prod_{i=1}^{n}P(X_i\mid C)\]
Therefore:
\[\boxed{
P(C\mid X_1,\ldots,X_n)
\propto
P(C)
\prod_{i=1}^{n}P(X_i\mid C)
}\]
The class with the highest resulting probability can be selected as the prediction.
3.16.16Example: Medical Diagnosis
Suppose:
\[D=\text{Disease}\]
and:
\[S=\text{Symptom}\]
We want:
\[P(D\mid S)\]
Bayes Theorem gives:
[
P(D\mid S)
\frac{P(S\mid D)P(D)}
{P(S)}
]
Suppose:
\[P(D)=0.02\]
\[P(S\mid D)=0.90\]
and:
\[P(S)=0.10\]
Then:
[
P(D\mid S)
\frac{0.90(0.02)}
{0.10}
]
[
0.18
]
Therefore:
\[\boxed{
P(D\mid S)=18%
}\]
This illustrates why knowing the probability of a symptom among patients with a disease is not enough. We need the prior probability and overall evidence probability as well.
3.16.17Bayes Theorem and Classification
For a classification problem, suppose we have classes:
\[C_1,C_2,\ldots,C_k\]
and observed features:
\[X\]
We calculate:
\[P(C_1\mid X)\]
\[P(C_2\mid X)\]
\[\ldots\]
\[P(C_k\mid X)\]
Then a Bayesian classifier may choose:
- [
\boxed{
\hat C
- \arg\max_C P(C\mid X)
}
]
- In simple terms:
Choose the class with the highest posterior probability.
3.16.18Bayes Theorem and Fraud Detection
Suppose:
\[F=\text{Transaction is fraudulent}\]
and:
\[X=\text{Observed transaction characteristics}\]
We want:
\[P(F\mid X)\]
Bayes Theorem allows us to update the probability of fraud based on evidence such as:
- Unusual transaction amount
- Unusual location
- New device
- Unusual transaction time
- Abnormal transaction pattern
- Conceptually:
\[\boxed{
\text{Prior Fraud Risk}
+
\text{Transaction Evidence}
\rightarrow
\text{Updated Fraud Probability}
}\]
3.16.19Bayes Theorem and Customer Churn
Suppose:
\[C=\text{Customer will churn}\]
and observed information includes:
\[X=
{\text{low usage, complaints, reduced purchases}}\]
We want:
\[P(C\mid X)\]
Bayesian reasoning allows the system to update the customer's churn probability based on the evidence.
This is useful for:
- Customer retention
- Risk scoring
- Recommendation systems
- Predictive analytics
3.16.20Bayes Theorem and Generative AI
Probability is fundamental to modern generative models.
A language model estimates conditional probabilities such as:
\[\boxed{
P(x_t\mid x_1,x_2,\ldots,x_{t-1})
}\]
where (x_t) represents the next token.
Bayesian reasoning is not identical to how every modern language model is trained, but the broader probabilistic framework is fundamental to understanding uncertainty and conditional probability in AI.
3.16.21Bayes Theorem in Odds Form
- Bayes Theorem can also be expressed using odds.
- The posterior odds are:
- [
\boxed{
\text{Posterior Odds}
- \text{Prior Odds}
\times
\text{Likelihood Ratio}
}
]
- Where:
- [
\text{Prior Odds}
\frac{P(A)}{1-P(A)}
]
and:
[
\text{Likelihood Ratio}
\frac{P(B\mid A)}
{P(B\mid A^c)}
]
This form is particularly useful in medical diagnosis, risk analysis, and statistical inference.
3.16.22Odds Example
Suppose:
\[P(D)=0.01\]
Then prior odds are:
\[\frac{0.01}{0.99}\]
\[\approx0.0101\]
Suppose:
\[P(P\mid D)=0.95\]
and:
\[P(P\mid D^c)=0.05\]
Then the likelihood ratio is:
\[\frac{0.95}{0.05}=19\]
Therefore:
\[\text{Posterior Odds}
\approx0.0101\times19\]
\[\approx0.1919\]
Converting these odds back to probability:
\[P=
\frac{0.1919}{1+0.1919}\]
\[\approx0.161\]
or:
\[\boxed{16.1%}\]
Same result.
3.16.23Bayes Theorem vs Conditional Probability
Conditional probability:
[
\boxed{
P(A\mid B)
\frac{P(A\cap B)}
{P(B)}
}
]
Bayes Theorem rewrites this using the reverse conditional probability:
[
\boxed{
P(A\mid B)
\frac{P(B\mid A)P(A)}
{P(B)}
}
]
The key benefit is that sometimes:
\[P(B\mid A)\]
is much easier to determine than:
\[P(A\mid B)\]
Bayes Theorem lets us reverse the direction.
3.16.24The Most Important Distinction
A common mistake is confusing:
\[\boxed{P(A\mid B)}\]
with:
\[\boxed{P(B\mid A)}\]
They are generally not equal.
For example:
\[P(\text{Disease}\mid\text{Positive Test})\]
is not necessarily equal to:
\[P(\text{Positive Test}\mid\text{Disease})\]
- The first is the probability of disease given the test result.
- The second is the probability of the test result given the disease.
- Bayes Theorem connects them.
3.16.25Bayes Theorem Using a Table
Consider 10,000 components:
| Defective | Not Defective | Total |
|---|
| Positive | 95 | 495 | 590 |
| Negative | 5 | 9,405 | 9,410 |
| Total | 100 | 9,900 | 10,000 |
From the table:
[
P(D\mid P)
\frac{\text{Defective and Positive}}
{\text{All Positive}}
]
Therefore:
[
P(D\mid P)
\frac{95}{590}
]
\[\boxed{
P(D\mid P)\approx16.1%
}\]
This is often the easiest way to understand Bayes Theorem intuitively.
3.16.26Bayesian Updating
Bayesian reasoning is an iterative process.
Suppose we initially believe:
\[P(A)=0.20\]
Then we observe evidence (B).
After applying Bayes Theorem:
\[P(A\mid B)=0.60\]
Now the new probability becomes the prior for future evidence.
If another piece of evidence (C) arrives, we can update again:
\[P(A\mid B,C)\]
Therefore:
\[\boxed{
\text{Prior}
\rightarrow
\text{Evidence}
\rightarrow
\text{Posterior}
\rightarrow
\text{New Prior}
\rightarrow
\text{More Evidence}
}\]
This is the core idea behind Bayesian updating.
3.16.27Bayesian Workflow
A typical Bayesian reasoning process is:
Step 1 — Establish Prior
\[P(A)\]
Step 2 — Observe Evidence
Observe:
\[B\]
Step 3 — Calculate Likelihood
\[P(B\mid A)\]
Step 4 — Calculate Evidence
\[P(B)\]
Step 5 — Calculate Posterior
\[P(A\mid B)\]
Step 6 — Update Belief
Use the posterior as the basis for future reasoning.
3.16.28Bayes Theorem Using Python
Bayes Theorem can be implemented directly in Python:
- prior = 0.01
- likelihood = 0.95
- false_positive = 0.05
- evidence = (
- likelihood * prior
- + false_positive * (1 - prior)
)
- posterior = (
- likelihood * prior
- / evidence
)
print("Posterior:", posterior)Output will be approximately:
Posterior: 0.161016949...
Therefore:
\[\boxed{P(D\mid P)\approx16.1%}\]
3.16.29Common Mistakes
Mistake 1: Reversing the probabilities
Incorrect:
\[P(A\mid B)=P(B\mid A)\]
Correct:
[
\boxed{
P(A\mid B)
\frac{P(B\mid A)P(A)}
{P(B)}
}
]
Mistake 2: Ignoring the prior
A common mistake is focusing only on:
\[P(B\mid A)\]
The prior:
\[P(A)\]
can have a major impact on the posterior.
Mistake 3: Ignoring false positives
In classification and diagnosis, false positives can significantly affect:
\[P(A\mid B)\]
especially when the event (A) is rare.
Mistake 4: Assuming a highly accurate test gives a highly certain diagnosis
A test can have high sensitivity or likelihood while the posterior probability remains relatively low when the underlying event is rare.
This is the base-rate effect.
3.16.30Key Formulas
Bayes Theorem
[
\boxed{
P(A\mid B)
\frac{P(B\mid A)P(A)}
{P(B)}
}
]
Evidence
[
\boxed{
P(B)
P(B\mid A)P(A)
+
P(B\mid A^c)P(A^c)
}
]
Joint Probability
[
\boxed{
P(A\cap B)
P(A\mid B)P(B)
}
]
- Odds Form
- [
\boxed{
\text{Posterior Odds}
- \text{Prior Odds}
\times
\text{Likelihood Ratio}
}
]
3.16.31Bayes Theorem in AI/ML
| Concept | AI/ML Meaning |
|---|
| Prior | Initial belief |
| Evidence | Observed data |
| Likelihood | How compatible data is with a hypothesis |
| Posterior | Updated belief |
| Conditional probability | Probability given information |
| Bayesian inference | Updating beliefs using evidence |
| Naive Bayes | Probabilistic classification algorithm |
3.16.32Key Takeaways
- Bayes Theorem allows us to update the probability of a hypothesis when new evidence becomes available.
- The most important formula is:
- [
\boxed{
P(A\mid B)
\frac{P(B\mid A)P(A)}
{P(B)}
}
]
- Remember the four components:
- [
\boxed{
\text{Posterior}
- \frac{
\text{Likelihood}\times\text{Prior}
}{
\text{Evidence}
}
}
]
- The conceptual flow is:
\[\boxed{
\text{Prior Belief}
+
\text{New Evidence}
\rightarrow
\text{Posterior Belief}
}\]
AI/ML Connection
Bayesian reasoning is useful whenever an AI system needs to reason about uncertainty:
\[\boxed{
\text{Observed Data}
\rightarrow
\text{Probability}
\rightarrow
\text{Updated Belief}
\rightarrow
\text{Prediction}
}\]
- Examples include:
- Spam detection
- Fraud detection
- Medical diagnosis
- Customer churn prediction
- Risk assessment
- Bayesian classification
The next topic, 3.17 Probability Distributions, builds on these concepts by explaining how probabilities are distributed across possible values and introduces important distributions such as Bernoulli, Binomial, Normal, Poisson, and Uniform distributions.