Statistics

The mean is one of the most commonly used measures of central tendency in statistics. It represents the average value of a dataset.

For example, if the marks of five students are

60, 70, 80, 90, 100

the mean tells us the typical or average mark of the group.

1. Formula for Mean

The arithmetic mean is calculated as

Or

Where

• (x_1, x_2, ..., x_n) = individual observations
• (n) = total number of observations
• (\bar{x}) = sample mean

Example

Consider

10, 20, 30, 40, 50

Sum

Number of observations

Therefore

Mean = 30

2. Mean in Python

Using Python

numbers = [10, 20, 30, 40, 50]
mean = sum(numbers) / len(numbers)
print(mean)

Output

30.0

Using NumPy

import numpy as np
numbers = [10, 20, 30, 40, 50]
mean = np.mean(numbers)
print(mean)

Output

30.0

3. Mean with Decimal Values

Consider

12, 15, 18, 21

So

Mean = 16.5

4. Mean and Outliers

One important limitation of the mean is that it is sensitive to extreme values, also called outliers.

Consider

10, 20, 30, 40, 50

Mean

Now add an extreme value

10, 20, 30, 40, 50, 1000

New mean

The mean changed from 30 to 191.67 because of the single outlier 1000.

This is why median is often preferred when a dataset contains extreme values.

5. Population Mean vs Sample Mean

There are two common types of mean.

Population Mean

When we have data for the entire population, we use

Where

• (\mu) = population mean
• (N) = population size
• Sample Mean

When we only have a sample of the population

Where

(\bar{x}) = sample mean

(n) = sample size

Example

Suppose a company has 1,000 employees, but we analyze the salaries of only 50 employees.

The 50 employees are a sample, so we calculate the sample mean.

6. Weighted Mean

Sometimes every value does not have equal importance. In that situation, we use a weighted mean.

Formula

Suppose a student's grades are

Weighted mean

Weighted Mean = 83

Weighted averages are commonly used for GPA, grades, portfolio returns, and business metrics.

7. Mean in Data Science

Mean is extremely important in Data Science and Machine Learning.

It is used for

• Understanding datasets
• Summarizing numerical variables
• Detecting unusual values
• Data preprocessing
• Missing-value imputation
• Feature scaling
• Statistical analysis
• Exploratory Data Analysis (EDA)

For example, if an age column contains missing values, one basic approach is

df["age"] = df["age"].fillna(df["age"].mean())

This replaces missing ages with the average age.

However, if the data contains significant outliers, median imputation may be more appropriate.

8. Mean vs Median

Example: Salaries

Suppose salaries are

• 30K, 35K, 40K, 45K, 500K
• The mean becomes very high because of the 500K salary.
• The median gives a better representation of the typical employee salary.

9. Important Properties of Mean

Remember these points

Mean uses every observation in the dataset.

The sum of deviations from the mean is always zero

• Mean is sensitive to outliers.
• Mean is commonly used with variance and standard deviation.
• Mean is particularly useful for numerical data.
• Mean can be a decimal even when all observations are integers.

10. Quick Example

Dataset

5, 8, 10, 12, 15

Therefore, the mean is 10.

Key takeaway

Mean = Total of all values ÷ Number of values

In Machine Learning and Data Science, mean is a fundamental concept because it forms the basis for variance, standard deviation, normalization, statistical inference, and many ML algorithms.

The median is a measure of central tendency that represents the middle value of a dataset after the values are arranged in ascending or descending order.

Unlike the mean, the median is less affected by extreme values (outliers).

1. Basic Concept

Consider the dataset

10, 20, 30, 40, 50

The values are already sorted.

The middle value is

Therefore

Median = 30

2. How to Calculate the Median

The calculation depends on whether the number of observations is odd or even.

Case 1: Odd Number of Observations

If there are an odd number of values, the median is the value exactly in the middle.

Example

10, 20, 30, 40, 50

There are 5 values.

Middle position

The 3rd value is 30.

• Case 2: Even Number of Observations
• If there are an even number of values, there are two middle values.
• The median is the average of those two middle values.

Example

10, 20, 30, 40, 50, 60

There are 6 values.

The two middle values are

30 and 40

Therefore

3. Median Calculation Formula

For an ordered dataset with (n) observations

When (n) is odd

When (n) is even

The most important step is

Always sort the data before calculating the median.

4. Example with Unsorted Data

Consider

50, 10, 30, 20, 40

First arrange the values

10, 20, 30, 40, 50

The middle value is 30.

Therefore

5. Median and Outliers

The biggest advantage of the median is that it is resistant to outliers.

Consider

10, 20, 30, 40, 50

Mean

Median

Now replace 50 with 1000

10, 20, 30, 40, 1000

Mean

But the median is still

So

The outlier dramatically changes the mean, but not the median.

6. Median for Real-World Data

Median is particularly useful when data is skewed.

For example, consider annual salaries

₹3L, ₹4L, ₹5L, ₹6L, ₹7L, ₹8L, ₹50L

The ₹50L salary is an extreme value.

The mean will be pulled upward by ₹50L.

The median will remain

Therefore, when reporting typical salary, house prices, income, or wealth, median can sometimes be more representative than mean.

7. Median in Python

Using Python

Python's statistics module provides a median() function.

from statistics import median
numbers = [10, 20, 30, 40, 50]
result = median(numbers)
print(result)

Output

30

Even number of values

from statistics import median
numbers = [10, 20, 30, 40, 50, 60]
print(median(numbers))

Output

35.0

8. Median Using NumPy

import numpy as np
numbers = [10, 20, 30, 40, 50]
result = np.median(numbers)
print(result)

Output

30.0

9. Median Using Pandas

Suppose we have a DataFrame

import pandas as pd
df = pd.DataFrame({
    "salary": [30000, 40000, 50000, 60000, 500000]
})

print(df["salary"].median())

Output

50000.0

This is useful in Exploratory Data Analysis (EDA).

10. Median for Missing Values

Median is frequently used for missing-value imputation.

Example

df["age"] = df["age"].fillna(df["age"].median())

If the age column contains extreme values, median is often safer than mean for filling missing values.

For example

25, 27, 28, 30, 32, 150

The 150 is likely an outlier.

Using the mean could produce a misleading replacement value, while the median is much more robust.

11. Mean vs Median

12. Relationship with Distribution Shape

• The relationship between mean and median can help us understand the shape of a distribution.
• Symmetric distribution
• Mean ≈ Median
• Right-skewed distribution
• Mean > Median

Example

• Income distribution
• A few very high incomes pull the mean to the right.
• Left-skewed distribution
• Mean < Median

13. Important Properties

Remember these points

• Data must be ordered before finding the median.
• With an odd number of observations, there is one middle value.
• With an even number, average the two middle values.
• Median is robust to outliers.
• Median is particularly useful for skewed distributions.
• Median is commonly used in EDA and data preprocessing.
• Median can be used for missing-value imputation.

14. Quick Example

Dataset

• 12, 5, 8, 20, 15
• Step 1: Sort
• 5, 8, 12, 15, 20
• Step 2: Find the middle value

There are 5 observations, so the middle position is

Third value = 12

Therefore

Key takeaway

Median is the middle value of an ordered dataset. It is especially useful when the data contains outliers or is highly skewed.

For Data Science, a good rule to remember is

• Mean → average
• Median → middle
• Mode → most frequent value

The mode is a measure of central tendency that represents the value or category that occurs most frequently in a dataset.

Unlike the mean and median, the mode can be used for both numerical and categorical data.

1. Basic Concept

Consider the dataset

10, 20, 20, 30, 40

Count the occurrences

The value 20 occurs most frequently.

Therefore

2. How to Find the Mode

To find the mode

• List the observations.
• Count how many times each value occurs.
• Identify the value with the highest frequency.

Example

Dataset

5, 7, 8, 7, 9, 7, 10

Frequency

• 5 → 1
• 7 → 3
• 8 → 1
• 9 → 1
• 10 → 1

Therefore

3. Dataset Can Have Multiple Modes

A dataset can have more than one mode.

Example

10, 20, 20, 30, 30, 40

Frequency

Both 20 and 30 occur twice.

Therefore

This is called a bimodal distribution.

Terminology

4. Dataset with No Mode

Consider

10, 20, 30, 40, 50

Every value occurs exactly once.

Therefore, there is no unique mode.

5. Mode with Categorical Data

One of the major advantages of mode is that it works with categorical data.

For example

Red, Blue, Green, Blue, Red, Blue, Yellow

Frequencies

Therefore

You cannot calculate a meaningful mean of colors, but you can calculate their mode.

6. Real-World Examples

Most common product

Suppose an online store has sales

• Laptop
• Phone
• Phone
• Tablet
• Phone
• Laptop

Mode

• Phone
• This tells the business which product was purchased most frequently.
• Most common customer age
• 25, 30, 25, 35, 40, 25, 30

Mode

• 25
• Most common payment method
• UPI, Card, UPI, Cash, UPI, Card

Mode

UPI

7. Mode in Python

Python's statistics module provides a mode() function.

from statistics import mode
numbers = [10, 20, 20, 30, 40]
result = mode(numbers)
print(result)

Output

20

8. Finding Multiple Modes in Python

Use multimode() when a dataset can have multiple modes.

from statistics import multimode
numbers = [10, 20, 20, 30, 30, 40]
result = multimode(numbers)
print(result)

Output

9. Mode Using Pandas

In Data Science, pandas is commonly used.

import pandas as pd
numbers = pd.Series([10, 20, 20, 30, 30, 40])
print(numbers.mode())

Output

0 20

1 30

dtype: int64

Pandas returns all modes, which is useful for multimodal datasets.

10. Mode and Missing Values

Mode is particularly useful for filling missing values in categorical columns.

Suppose

City

------

• Hyderabad
• Bangalore
• Hyderabad
• NULL
• Hyderabad
• Chennai

The mode is

Hyderabad

We can replace the missing value with the mode

df["city"] = df["city"].fillna(df["city"].mode()[0])

This is a common preprocessing technique in Machine Learning.

11. Mean vs Median vs Mode

12. Relationship Between Mean, Median and Mode

For a perfectly symmetric unimodal distribution

For a moderately right-skewed distribution

For a moderately left-skewed distribution

This relationship is useful when interpreting distributions during Exploratory Data Analysis (EDA).

13. Example: E-Commerce Data

Suppose an e-commerce company records the number of products purchased per order

1, 2, 1, 3, 2, 1, 4, 1, 2, 1

Frequency

Therefore

This means one product per order is the most common purchasing behavior.

14. Key Points to Remember

Mode = most frequently occurring value

• A dataset can have one, two, or multiple modes.
• A dataset may have no mode.
• Mode works with categorical data.
• Mode is not significantly affected by extreme values.
• Use statistics.mode() for a single mode.
• Use statistics.multimode() when multiple modes are possible.
• Pandas uses .mode() to find modes.
• Mode is frequently used in EDA and data preprocessing.
• Easy way to remember
• Mean → Average Median → Middle Mode → Most frequent

Next in your syllabus is 5.4 Variance, which is where we start measuring how much the data varies around the mean.

Variance is a statistical measure that tells us how spread out the values in a dataset are around their mean.

In simple terms

• Variance measures how far the data values tend to be from the average.
• Small variance → values are close to the mean.
• Large variance → values are more spread out.

genui{"learning_viz":{"type_id":"VARIANCE"}}

1. Simple Example

Consider two datasets

• Dataset A
• 48, 49, 50, 51, 52
• Dataset B
• 10, 30, 50, 70, 90

Both have the same mean

But Dataset A is tightly clustered around 50, while Dataset B is widely spread.

Therefore

This is the main purpose of variance: measuring spread.

2. How Variance Is Calculated

The basic process is

• Calculate the mean.
• Find the difference between each value and the mean.
• Square each difference.
• Calculate the average of those squared differences.

Consider

2, 4, 6, 8, 10

Step 1: Calculate the mean

Step 2: Calculate deviations

Step 3: Square the deviations

Sum

Step 4: Divide by the number of observations

For a population

So the population variance is 8.

3. Population Variance

When your dataset contains the entire population, use

Where

• (\sigma^2) = population variance
• (x_i) = individual observation
• (\mu) = population mean
• (N) = population size

4. Sample Variance

Usually, in statistics, we don't have the entire population. We have a sample.

For a sample, we divide by (n-1) rather than (n)

The (n-1) is called the degrees-of-freedom correction or Bessel's correction.

Why (n-1)?

Because the sample mean is itself estimated from the sample. Dividing by (n-1) gives an unbiased estimate of the population variance under the usual assumptions.

5. Population vs Sample Variance

6. Variance in Python

Using NumPy

import numpy as np
data = [2, 4, 6, 8, 10]
population_variance = np.var(data)
print(population_variance)

Output

• 8.0
• NumPy's default is population variance (ddof=0).
• Sample variance

import numpy as np
data = [2, 4, 6, 8, 10]
sample_variance = np.var(data, ddof=1)
print(sample_variance)

Output

10.0

Because

7. Using Python statistics

Python also provides variance()

from statistics import variance
data = [2, 4, 6, 8, 10]
print(variance(data))

Output

10

statistics.variance() calculates sample variance.

For population variance

from statistics import pvariance
data = [2, 4, 6, 8, 10]
print(pvariance(data))

Output

8

So remember

variance() → sample variance

pvariance() → population variance

8. Variance in Pandas

import pandas as pd
data = pd.Series([2, 4, 6, 8, 10])
print(data.var())

Output

10.0

Pandas .var() uses sample variance by default (ddof=1).

For population variance

print(data.var(ddof=0))

Output

8.0

9. Why Do We Square the Differences?

You might wonder why we calculate

instead of simply calculating

Consider

2, 4, 6, 8, 10

The deviations are

-4, -2, 0, +2, +4

Their sum is

The positive and negative deviations cancel each other.

By squaring them

• 16, 4, 0, 4, 16
• they all become positive.
• That's why variance uses squared deviations.

10. Important Relationship: Variance and Standard Deviation

Variance is measured in squared units.

If your original data is measured in

kilograms

then variance is measured in

• kg²
• This can make variance difficult to interpret directly.
• That's why we often use standard deviation.

For our example

Therefore

Standard deviation is in the same units as the original data.

11. Variance and Outliers

Variance is highly sensitive to outliers because deviations are squared.

Consider

10, 20, 30, 40, 50

versus

10, 20, 30, 40, 500

The value 500 is far from the mean.

Its deviation becomes large, and squaring that deviation makes its contribution to variance very large.

Therefore

Outliers can dramatically increase variance.

12. Variance in Machine Learning

Variance is extremely important in Machine Learning.

It is used in

• Feature analysis
• Feature selection
• Data preprocessing
• Standardization
• Probability distributions
• Statistical inference
• Bias-variance tradeoff
• Model evaluation
• Detecting low-variance features
• Low-variance feature

Suppose

• Age: 30, 31, 30, 32, 31, 30
• There is little variation.
• High-variance feature
• Income: 20K, 40K, 80K, 150K, 500K
• There is substantial variation.

A feature with almost zero variance may provide very little information because its value hardly changes.

13. Variance and Bias-Variance Tradeoff

In Machine Learning, you'll later encounter

[ \boxed{\text{Prediction Error}

\text{Bias}^2+\text{Variance}+\text{Irreducible Error}} ]

Here, variance has a different but related meaning: it describes how much a model's predictions can change when trained on different samples of the training data.

A high-variance model tends to

• Fit training data extremely well
• Be sensitive to training data changes
• Potentially overfit

This becomes important when studying Decision Trees, Random Forests, Regression, and other ML algorithms.

14. Quick Example

Dataset

4, 6, 8

Mean

Deviations

-2, 0, +2

• Squared deviations
• 4, 0, 4
• Population variance

Sample variance

15. Mean, Median, Mode and Variance

So far you've covered

The natural next step is Standard Deviation, which is simply the square root of variance and is much easier to interpret because it uses the same units as the original data.

Standard deviation (SD) is one of the most important measures of dispersion in statistics.

It tells us how far the values in a dataset typically spread out from the mean.

Small standard deviation → values are close to the mean. Large standard deviation → values are widely spread from the mean.

genui{"learning_viz":{"type_id":"STANDARD_DEVIATION"}}

1. Simple Example

Consider two datasets

Dataset A

48, 49, 50, 51, 52

Mean = 50

• The values are very close to 50, so the standard deviation is small.
• Dataset B
• 10, 30, 50, 70, 90

Mean = 50

The values are much farther from 50, so the standard deviation is large.

Both datasets have the same mean, but their spread is different.

2. Relationship Between Variance and Standard Deviation

This is the most important formula to remember

And

For example, if

then

So

3. Population Standard Deviation

For an entire population

Where

• (\sigma) = population standard deviation
• (x_i) = individual observation
• (\mu) = population mean
• (N) = population size

4. Sample Standard Deviation

For a sample

Notice that we use (n-1) for sample standard deviation.

5. Step-by-Step Example

Consider

2, 4, 6, 8, 10

Step 1: Calculate the mean

Step 2: Calculate deviations

Step 3: Square deviations

Sum

Step 4: Calculate population variance

Step 5: Take square root

So the population standard deviation is approximately 2.83.

6. Why Standard Deviation Is Better to Interpret Than Variance

Suppose we have heights measured in centimeters.

If

then

Variance is expressed in squared units, while standard deviation is expressed in the original units.

Therefore

Standard deviation is usually easier to interpret.

7. Standard Deviation in Python

NumPy

import numpy as np
data = [2, 4, 6, 8, 10]
sd = np.std(data)
print(sd)

Output

• 2.8284271247461903
• NumPy uses population standard deviation by default.
• Sample standard deviation

import numpy as np
data = [2, 4, 6, 8, 10]
sd = np.std(data, ddof=1)
print(sd)

Output

3.1622776601683795

Remember

ddof=0 → Population SD
ddof=1 → Sample SD

8. Python statistics Module

from statistics import stdev
data = [2, 4, 6, 8, 10]
print(stdev(data))

Output

3.1622776601683795

stdev() calculates sample standard deviation.

For population standard deviation

from statistics import pstdev
data = [2, 4, 6, 8, 10]
print(pstdev(data))

Output

2.8284271247461903

So

9. Standard Deviation Using Pandas

import pandas as pd
data = pd.Series([2, 4, 6, 8, 10])
print(data.std())

Output

3.1622776601683795

Pandas .std() uses sample standard deviation by default.

For population SD

print(data.std(ddof=0))

10. Standard Deviation and the Normal Distribution

Standard deviation becomes especially important when data follows a normal distribution.

For a normal distribution

Within 1 standard deviation

Approximately 68% of observations fall within

Within 2 standard deviations

Approximately 95% fall within

Within 3 standard deviations

Approximately 99.7% fall within

This is called the 68–95–99.7 rule or Empirical Rule.

11. Example: Exam Scores

Suppose exam scores have

For a roughly normal distribution

Within 1 SD

to

Approximately 68% of students score between 60 and 80.

Within 2 SD

to

Approximately 95% score between 50 and 90.

Within 3 SD

to

Approximately 99.7% fall between 40 and 100, assuming a normal distribution.

12. Standard Deviation and Outliers

Standard deviation is sensitive to outliers.

Consider

10, 20, 30, 40, 50

Now add

1000

The extreme value creates a very large deviation from the mean.

Because standard deviation is based on squared deviations, the outlier can significantly increase the SD.

Therefore, when dealing with highly skewed data, we should also consider

• Median
• Interquartile range (IQR)
• Robust statistical methods

13. Standard Deviation in Machine Learning

Standard deviation is extremely important in Machine Learning.

1. Feature scaling

Standardization commonly uses

This transforms data so that it has approximately

and

In Python

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

2. Detecting unusual observations

A z-score tells us how many standard deviations an observation is away from the mean.

For example

means the observation is 2 standard deviations above the mean.

means it is 3 standard deviations below the mean.

3. Feature analysis

Consider

The SD tells us how much each variable varies, although the raw magnitudes cannot be directly compared when units differ.

14. Variance vs Standard Deviation

15. Mean, Median, Mode, Variance and SD

You have now covered five important statistical concepts

Easy memory trick

Variance = Spread²
Standard Deviation = Spread

Or simply

Key takeaway

Standard deviation tells you how much the data typically varies around its mean, in the same units as the original data.

Probability is the branch of mathematics used to measure the likelihood or chance that an event will occur.

In simple terms

Probability tells us how likely something is to happen.

Examples

• What is the probability of getting heads when tossing a coin?
• What is the probability of rolling a 6 on a die?
• What is the probability that a customer will purchase a product?
• What is the probability that a machine-learning model makes a correct prediction?

Probability is one of the most important foundations of Statistics, Data Science, and Machine Learning.

1. Probability Range

Probability always lies between 0 and 1

Where (P(A)) means the probability of event (A).

Probability can also be expressed as a percentage.

2. Basic Probability Formula

If all outcomes are equally likely

Example: Rolling a Die

A standard die has

1, 2, 3, 4, 5, 6

What is the probability of getting 4?

There is

1 favorable outcome

6 total outcomes

Therefore

or

3. Experiment, Outcome and Event

Three important terms

Experiment

An action that produces an outcome.

Example

• Rolling a die.
• Outcome
• A possible result of the experiment.

Example

1, 2, 3, 4, 5, or 6.

Event

A collection of one or more outcomes.

Example

Getting an even number.

The event consists of

{2, 4, 6}

Therefore

So

4. Sample Space

The sample space is the set of all possible outcomes.

It is usually represented by (S).

For a coin toss

For a die

For two coin tosses

5. Probability of a Coin Toss

A fair coin has two possible outcomes

Heads

Tails

Therefore

and

So

or

50% probability each.

6. Complementary Probability

The complement of an event means the event does not occur.

The formula is

Suppose

Then

So if there is a 70% probability of rain, the probability of no rain is 30%, assuming those are the only two possibilities.

7. Addition Rule

The addition rule is used when we want the probability of A OR B.

For mutually exclusive events

Example

Roll a die.

What is the probability of getting 2 OR 5?

Therefore

[ P(2\text{ or }5)

\frac{1}{6}+\frac{1}{6} ]

8. General Addition Rule

When events can overlap

The intersection is subtracted because the overlapping outcomes would otherwise be counted twice.

9. Multiplication Rule

The multiplication rule is used for A AND B.

For independent events

Example: Two Coin Tosses

Probability of getting

Heads AND Heads

Therefore

25%

10. Independent Events

Two events are independent when one event does not affect the probability of the other.

Examples

• Tossing a coin twice
• Rolling a die twice
• Selecting independent random observations

If

and

then

11. Dependent Events

Events are dependent when one event affects another.

For dependent events

Here

means

Probability of B occurring given that A has already occurred.

12. Conditional Probability

Conditional probability is one of the most important concepts in statistics and machine learning.

The formula is

It means

Probability of A given that B has occurred.

Example

Suppose a dataset contains 100 customers

• 40 customers are male.
• 20 male customers purchased a product.
• What is the probability that a customer purchased the product given that the customer is male?
• [ P(\text{Purchase}|\text{Male})

\frac{20}{40} ]

Therefore

50%

13. Bayes' Theorem

Bayes' theorem allows us to update a probability based on new evidence.

The formula is

Where

• (P(A|B)) = posterior probability
• (P(B|A)) = likelihood
• (P(A)) = prior probability
• (P(B)) = evidence

Bayes' theorem is extremely important in

• Machine Learning
• Medical diagnosis
• Spam detection
• Fraud detection
• Recommendation systems
• Naive Bayes classification
• Risk analysis

14. Probability Example: Spam Detection

Suppose an email contains the word

"Congratulations"

We want to calculate

This means

What is the probability that an email is spam, given that it contains the word "Congratulations"?

A machine-learning model can use probabilities such as

and

to estimate the probability that the email is spam.

This is the basic idea behind Naive Bayes classification.

15. Probability in Python

Python's random module can be used for simple probability simulations.

Coin Toss Simulation

import random
heads = 0
trials = 10000
for _ in range(trials):
    if random.choice(["H", "T"]) == "H":
        heads += 1

probability = heads / trials
print(probability)

A typical result might be

0.502

The theoretical probability is

As the number of trials increases, the experimental probability generally gets closer to the theoretical probability.

16. Probability Using NumPy

import numpy as np
rolls = np.random.randint(1, 7, size=10000)
probability_six = np.mean(rolls == 6)
print(probability_six)

The result should be close to

17. The Law of Large Numbers

The Law of Large Numbers states that as the number of experiments increases, the observed probability tends to approach the theoretical probability.

For example

10 coin tosses

You might get

7 Heads

3 Tails

Experimental probability of heads

10,000 tosses

You might get approximately

5,000 Heads

5,000 Tails

Experimental probability

The result becomes more stable as the number of trials increases.

18. Probability in Machine Learning

Probability is fundamental to ML.

Classification

A model might output

Cat → 0.85

Dog → 0.10

Rabbit → 0.05

The model predicts Cat because it has the highest probability.

Logistic Regression

Logistic regression predicts a probability

For example

The model estimates an 82% probability of churn.

Neural Networks

Classification models commonly produce probabilities using functions such as

Sigmoid

Softmax

19. Probability vs Statistics

These concepts are related but have different directions.

Probability

Starts with known assumptions and predicts possible outcomes.

Statistics

Starts with observed data and tries to learn about the underlying population.

A useful way to remember

Probability → From assumptions to outcomes Statistics → From data to conclusions

20. Important Probability Formulas

21. Key Terms to Remember

Quick Summary

The most important ideas to remember are

And the most important ML connection

Probability allows machine-learning models to represent uncertainty and make predictions in terms of likelihoods.

Next in your syllabus is 5.7 Normal Distribution, where probability, mean, standard deviation, and z-scores come together.

The Normal Distribution is one of the most important probability distributions in Statistics, Data Science, and Machine Learning.

It describes data that tends to cluster around a central value, with fewer observations occurring as we move farther away from the center.

It is commonly called the bell curve because of its shape.

1. What Does a Normal Distribution Look Like?

A normal distribution is approximately

^

/ \

/ \

/ \

/ \

__________/___________________\__________

Mean

The highest point is at the mean, and the curve is symmetric around the mean.

For a perfectly normal distribution

2. Example

Suppose the heights of a large population have

Most people will have heights close to 170 cm.

Fewer people will be extremely short or extremely tall.

The distribution might look conceptually like

Frequency

|

| ***

| *******

| ***********

| ***************

| *******************

|________*************************________

146 154 162 170 178 186 194

↑

Mean

3. Two Important Parameters

A normal distribution is determined by two parameters

• Mean (\mu)
• Determines the center of the distribution.
• Standard deviation (\sigma)
• Determines the spread of the distribution.

So we often write

This means that (X) follows a normal distribution with

Mean = (\mu)
Variance = (\sigma^2)

4. Properties of Normal Distribution

A normal distribution has several important properties.

1. Bell-shaped

The curve has a bell shape.

2. Symmetric

The left and right sides are mirror images.

3. Mean = Median = Mode

All three measures of central tendency are equal.

4. Total area = 1

The entire area under the probability density curve represents probability.

Therefore

5. Tails extend indefinitely

The curve approaches zero but theoretically never actually reaches zero.

5. Mean and Standard Deviation

Suppose

and

Then

Normal Distribution

Mean

↓

___/ \___

__/ \__

_/ \_

___________/___________________\___________

55 70 85 100 115 130 145

-3σ -2σ -1σ μ +1σ +2σ +3σ

The standard deviation tells us how far we are from the mean.

6. Empirical Rule: 68–95–99.7 Rule

One of the most important concepts in a normal distribution is the Empirical Rule.

For normally distributed data

Within 1 standard deviation

Approximately

of observations fall within

Within 2 standard deviations

Approximately

fall within

Within 3 standard deviations

Approximately

fall within

7. Example of the Empirical Rule

Suppose exam scores follow a normal distribution

Within 1 SD

Approximately 68% of students score between

Within 2 SD

to

Approximately 95% score between

Within 3 SD

to

Approximately 99.7% score between

8. Z-Score

A z-score tells us how many standard deviations a particular observation is away from the mean.

The formula is

Where

• (x) = observation
• (\mu) = mean
• (\sigma) = standard deviation
• (z) = z-score

genui{"learning_viz":{"type_id":"STANDARD_SCORE_Z"}}

9. Z-Score Example

Suppose

A student scores

Then

The student's score is 2 standard deviations above the mean.

10. Negative Z-Score

Suppose a student scores

Then

The score is 2 standard deviations below the mean.

11. Interpreting Z-Scores

A z-score close to zero means the observation is close to the mean.

A large positive or negative z-score indicates an observation far from the mean.

12. Standard Normal Distribution

A special normal distribution is called the standard normal distribution.

It has

and

So

Every observation can be converted into a z-score so that we can work with the standard normal distribution.

13. Probability Under the Normal Curve

The area under a normal curve represents probability.

For example

|

_____|_____

/ \

/ \

________/___________________\________

μ

The probability of an observation falling in a particular interval is the area under the curve over that interval.

This is why normal distributions are extremely useful for calculating probabilities.

14. Example: Probability Above the Mean

Because a normal distribution is symmetric

Therefore

Similarly

15. Normal Distribution in Python

Python's SciPy library provides tools for working with normal distributions.

from scipy.stats import norm
mean = 70
std = 10
probability = norm.cdf(80, loc=mean, scale=std)
print(probability)

This calculates

For a normal distribution with mean 70 and standard deviation 10.

The result is approximately

0.8413

So

16. Probability Between Two Values

Suppose

What is the probability that

Using Python

from scipy.stats import norm
mean = 70
std = 10
probability = (
    norm.cdf(80, loc=mean, scale=std)
    - norm.cdf(60, loc=mean, scale=std)
)

print(probability)

The result is approximately

0.6827

So approximately

This matches the 68% empirical rule.

17. Probability Density Function

The probability density function (PDF) of a normal distribution is

You do not need to memorize this formula immediately if you're learning statistics for practical Data Science.

The more important concepts are

• Mean
• Standard deviation
• Z-score
• Probability/area
• 68–95–99.7 rule

18. Normal Distribution in Data Science

Normal distribution appears frequently in

• Statistics
• Confidence intervals
• Hypothesis testing
• Sampling distributions
• Machine Learning
• Feature scaling
• Statistical assumptions
• Model residual analysis
• Gaussian models
• Probability estimation
• Data Analysis
• Identifying unusual observations
• Understanding distributions
• Calculating percentiles
• Comparing observations using z-scores

19. Normal Distribution and Outliers

Z-scores can help identify potentially unusual observations.

For example

is relatively close to the mean.

But

is very far from the mean.

A common practical rule is that observations beyond approximately ±3 standard deviations may deserve investigation, although this is not a universal outlier rule and should depend on the dataset and context.

20. Normal Distribution vs Uniform Distribution

Normal Distribution

Values are concentrated around the center.

/\

/ \

/ \

______/__________\______

Uniform Distribution

Values have approximately equal probability across a range.

____________

| |

_____|____________|_____

21. Important Concepts to Remember

Final Summary

The most important relationship is

And remember the 68–95–99.7 rule

Easy way to remember

Mean tells you where the center is. Standard deviation tells you how wide the distribution is. Z-score tells you how far an individual value is from the center.

These three concepts are the foundation for the next topics in your syllabus, especially Sampling, Central Limit Theorem, Hypothesis Testing, p-values, and Confidence Intervals.

Sampling is the process of selecting a smaller group of observations from a larger population so that we can study the smaller group and draw conclusions about the entire population.

Population → Sample → Analysis → Conclusion about Population

Sampling is extremely important in Statistics, Data Science, Machine Learning, surveys, and business analytics.

1. Population vs Sample

These are the two most important terms.

Population

The population is the complete group that we want to study.

Example

All customers of Amazon in India.

Sample

A sample is a smaller subset selected from that population.

Example

5,000 randomly selected Amazon customers in India.

Population

┌─────────────────────────────────────┐

│ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ │

│ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ │

│ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ │

└─────────────────────────────────────┘

↓

Sampling

↓

┌─────────────┐

│ ○ ○ ○ ○ ○ ○ │

│ ○ ○ ○ ○ ○ ○ │

└─────────────┘

Sample

We analyze the sample and use the results to make inferences about the population.

2. Why Do We Need Sampling?

Studying an entire population is often

• Expensive
• Time-consuming
• Difficult
• Sometimes impossible

Example

Suppose a company has

10 million customers.

It would be expensive and unnecessary to interview every customer.

Instead, we might select

10,000 customers.

If the sample is properly selected, we can estimate

• Average customer satisfaction
• Average spending
• Product preferences
• Customer demographics
• Churn behavior

3. Census vs Sampling

Census

Study every member of the population.

Example

Government census collecting information from every household.

Sampling

Study only a subset of the population.

4. Sampling Terminology

Some important terms

• Population
• Entire group being studied.
• Sample
• Subset selected from the population.
• Parameter
• A numerical characteristic of the population.

Example

Statistic

A numerical characteristic calculated from a sample.

Example

This distinction is very important.

5. Sampling Methods

Sampling methods are broadly divided into

• Probability Sampling
• Every member has a known probability of being selected.
• Simple Random Sampling
• Systematic Sampling
• Stratified Sampling
• Cluster Sampling
• Non-Probability Sampling
• Selection is not based on a known random probability.
• Convenience Sampling
• Voluntary Response Sampling
• Purposive/Judgment Sampling

6. Simple Random Sampling

In simple random sampling, every member of the population has an equal chance of being selected.

Example

• A company has 10,000 employees.
• We randomly select 500 employees.
• Every employee has the same probability of selection.

This is one of the simplest and most important sampling methods.

7. Simple Random Sampling in Python

Using NumPy

import numpy as np
population = np.arange(1, 101)
sample = np.random.choice(
    population,
    size=10,
    replace=False
)

print(sample)

Here

Population = numbers 1–100

Sample size = 10

replace=False means an observation cannot be selected twice.

8. Systematic Sampling

In systematic sampling, we select observations at a fixed interval.

Suppose we have 1,000 customers and need a sample of 100.

Sampling interval

We might randomly choose a starting point and then select

7, 17, 27, 37, 47, ...

Every 10th observation is selected.

9. Stratified Sampling

In stratified sampling, the population is divided into meaningful groups called strata, and samples are taken from each group.

Suppose a company has

Instead of randomly selecting employees without considering departments, we select employees from each department.

This ensures every important subgroup is represented.

Example

If we need 100 employees

IT → 50

Finance → 20

HR → 10

Sales → 20

This maintains the approximate population proportions.

10. Cluster Sampling

In cluster sampling, the population is divided into groups called clusters, and entire clusters or members within selected clusters are sampled.

Example

Suppose we want to survey school students across India.

Instead of selecting individual students from every school

• Divide schools into clusters.
• Randomly select schools.
• Survey students from those schools.

This can significantly reduce the cost of data collection.

11. Stratified vs Cluster Sampling

This is a common exam/interview question.

• Easy memory trick
• Stratified → Take some from every group.
• Cluster → Select some groups.

12. Convenience Sampling

In convenience sampling, we select people who are easiest to reach.

Example

Asking people standing outside a shopping mall.

It is

• Easy
• Fast
• Cheap
• But it can introduce sampling bias.

The people available at the mall may not represent the entire population.

13. Voluntary Response Sampling

Participants choose whether to participate.

Example

An online survey asking: "Do you think our service is good?"

People with very strong opinions may be more likely to respond.

Therefore, the results may not represent the entire population.

14. Sampling Bias

Sampling bias occurs when the sampling process systematically produces a sample that does not properly represent the population.

Example

Suppose you want to know

• How satisfied are Indians with internet services?
• You survey only people in a technology conference.
• The sample is unlikely to represent the entire Indian population.

This is an example of potential selection bias.

15. Sampling Error

Even if we use random sampling, the sample will usually not exactly match the population.

Suppose

Population mean

Sample 1 mean

Sample 2 mean

The differences occur because we're observing only a sample.

This difference is called sampling error.

Sampling error is not necessarily a mistake. It is natural variation caused by using a sample instead of the entire population.

16. Sample Size

Generally, larger samples provide more reliable estimates of population characteristics.

For example

• Sample size = 10
• Sample size = 100
• Sample size = 10,000

A sample of 10 may produce a very unstable estimate.

A properly selected sample of 10,000 will generally provide a more precise estimate.

However

A larger sample does not automatically eliminate bias.

A huge biased sample can still produce a misleading conclusion.

17. Sampling Distribution

Suppose we repeatedly take samples from a population and calculate the mean of each sample.

For example

• Sample 2 → Mean = 50.2
• Sample 3 → Mean = 50.1
• Sample 4 → Mean = 49.6
• Sample 5 → Mean = 50.4

...

The collection of these sample means forms a sampling distribution of the sample mean.

This concept leads directly to the next major topic

Central Limit Theorem (CLT).

18. Standard Error

The standard error of the mean measures how much sample means tend to vary from sample to sample.

For a population with standard deviation (\sigma)

Where

• (SE_{\bar{x}}) = standard error of the sample mean
• (\sigma) = population standard deviation
• (n) = sample size
• Important relationship

As sample size increases

then

Therefore

Larger samples generally produce more precise estimates of the population mean.

19. Example of Standard Error

Suppose

and

Then

Now increase the sample size to 400

The larger sample produces a smaller standard error.

20. Sampling in Python

Let's simulate sampling from a population.

import numpy as np
population = np.arange(1, 10001)
sample = np.random.choice(
    population,
    size=100,
    replace=False
)

print("Sample Mean:", np.mean(sample))

The sample mean will be close to the population mean, although it will generally not be exactly equal.

21. Sampling and Machine Learning

Sampling is extremely important in ML.

Train/Test Split

A dataset is often divided into

Dataset

↓

┌──────────────┐

│ Training │

│ Data │

└──────────────┘

┌──────────────┐

│ Test │

│ Data │

└──────────────┘

For example

• 80% → training
• 20% → testing
• Random Sampling

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(

X,

y,

test_size=0.2,
random_state=42

)

22. Stratified Sampling in Machine Learning

Suppose a classification dataset contains

90% → Class 0

10% → Class 1

A random split could accidentally create different proportions in training and testing data.

We can use stratification

X_train, X_test, y_train, y_test = train_test_split(

X,

y,

test_size=0.2,
random_state=42,
stratify=y

)

This attempts to preserve the class proportions in both datasets.

23. Sampling Bias vs Sampling Error

These are frequently confused.

24. Complete Sampling Workflow

A typical statistical study looks like

25. Important Terms

26. Key Takeaways

Remember these five points

Population = entire group.
Sample = subset of the population.

• A good sample should be representative of the population.
• Larger samples generally reduce sampling variability.
• Sampling is the foundation for statistical inference.

The most important relationship to remember is

And the conceptual flow is

The next topic, 5.9 Central Limit Theorem (CLT), explains one of the most powerful ideas in statistics: why the distribution of sample means tends to become approximately normal as the sample size gets large—even when the original population is not normally distributed.

5.9Central Limit Theorem (CLT)

The Central Limit Theorem (CLT) is one of the most important concepts in Statistics and Data Science.

In simple terms

When we take many sufficiently large random samples from a population and calculate the mean of each sample, the distribution of those sample means tends to become approximately normal, even if the original population is not normally distributed.

This is the key idea behind many statistical methods.

1. The Basic Idea

Imagine a population containing thousands of values.

Population

↓

┌─────────┐

│ Data │

│ Data │

│ Data │

│ Data │

└─────────┘

↓

• Sample 2 → Mean = 51.2
• Sample 3 → Mean = 49.8
• Sample 4 → Mean = 50.4

The individual observations don't necessarily need to be normally distributed.

It is the distribution of sample means that becomes approximately normal.

2. Why Is CLT Important?

CLT allows us to make conclusions about a population using samples.

It is fundamental to

• Confidence intervals
• Hypothesis testing
• Estimation
• Statistical inference
• A/B testing
• Quality control
• Survey analysis
• Machine Learning statistics
• Business analytics

Without CLT, many practical statistical techniques would be much harder to justify.

3. Example

Suppose the income of a population is highly skewed.

Frequency

|

| **

| ****

| *******

| ***********

| ****************

|************************

+------------------------→ Income

The population is not normally distributed.

Now randomly select 30 people and calculate their average income.

Repeat this thousands of times

• Sample 1 → Mean = ₹48,000
• Sample 2 → Mean = ₹51,000
• Sample 3 → Mean = ₹49,500
• Sample 4 → Mean = ₹50,200
• Sample 5 → Mean = ₹50,700

...

Now plot all those sample means.

The resulting distribution tends to become approximately

***

*********

*************

*****************

*********************

___________***************___________

μ

That's the Central Limit Theorem.

4. Three Important Conditions

The CLT generally works well when

1. Samples are random

Samples should be selected randomly or using an appropriate sampling method.

2. Observations are independent

One observation should not improperly influence another.

3. Sample size is sufficiently large

A common rule of thumb is

However, 30 is not a universal requirement.

If the population is already approximately normal, smaller samples may be sufficient.

If the population is extremely skewed or has heavy tails, a much larger sample may be required.

5. Sampling Distribution of the Mean

Suppose the population has

and

Take many samples of size (n).

Calculate the mean of each sample.

The resulting distribution is called the

CLT tells us that this distribution tends to become approximately normal as (n) increases.

6. Mean of the Sampling Distribution

An important result of CLT is

This means

The average of the sample means is equal to the population mean.

Example

Population mean

After taking thousands of samples, the average of all sample means will approach

7. Standard Error

The standard deviation of the sampling distribution of the sample mean is called the Standard Error (SE).

The formula is

Where

• (\sigma) = population standard deviation
• (n) = sample size
• (SE_{\bar{x}}) = standard error of the sample mean

This is one of the most important formulas in statistics.

8. Example of Standard Error

Suppose

and

Then

So the distribution of sample means has

and approximately

9. Effect of Sample Size

Notice

As (n) increases, the standard error decreases.

For example

So

Larger samples → sample mean becomes more stable.

10. Why Square Root?

Notice that increasing sample size from

is a 4× increase.

But the standard error changes from

which is a 2× decrease.

That's because

To cut the standard error in half, you generally need approximately 4 times the sample size.

11. CLT Does NOT Say This

A common misunderstanding is

"CLT says every dataset becomes normally distributed."

• Incorrect.

CLT does not say the original population becomes normal.

Instead

The sampling distribution of a statistic, especially the sample mean, tends to become approximately normal as sample size increases under suitable conditions.

This distinction is very important.

12. Population vs Sampling Distribution

Population

The original data may look like

• Highly skewed
• Sample
• A random subset of the population.
• Sampling distribution

Distribution of the statistic calculated from many samples.

• Sample 2 → Mean
• Sample 3 → Mean
• Sample 4 → Mean

...

↓

13. CLT Example with Dice

A single die roll has a uniform distribution

1 2 3 4 5 6

It is definitely not normally distributed.

Now

Take 1 die roll

Distribution

Uniform

• Take 5 dice and calculate their average
• The distribution of averages starts becoming more bell-shaped.
• Take 30 dice and calculate their average
• The distribution of averages becomes much closer to normal.

So even though

the

This is an excellent demonstration of CLT.

14. Python Demonstration

We can simulate CLT using NumPy.

import numpy as np
import matplotlib.pyplot as plt
population = np.random.uniform(0, 100, 100000)
sample_means = []
for _ in range(10000):
sample = np.random.choice(population, size=30)

sample_means.append(np.mean(sample))

plt.hist(sample_means, bins=50)
plt.xlabel("Sample Mean")
plt.ylabel("Frequency")
plt.title("Sampling Distribution of Sample Means")
plt.show()

The original population is approximately uniform, but the distribution of the sample means will look approximately bell-shaped.

15. Another Python Example

We can use an exponential distribution, which is strongly right-skewed.

import numpy as np
import matplotlib.pyplot as plt
sample_means = []
for _ in range(10000):
sample = np.random.exponential(scale=10, size=30)

sample_means.append(np.mean(sample))

plt.hist(sample_means, bins=50)
plt.xlabel("Sample Mean")
plt.ylabel("Frequency")
plt.title("CLT with a Skewed Population")
plt.show()

The original exponential population is strongly skewed.

But the distribution of the sample means becomes approximately normal.

16. CLT and Confidence Intervals

CLT is one of the reasons we can construct confidence intervals for population means.

For sufficiently large samples

This allows us to estimate the population mean using sample data.

For example

gives an approximate 95% confidence interval when the relevant assumptions are satisfied and (\sigma) is known.

17. CLT and Hypothesis Testing

CLT also forms a foundation for many hypothesis tests.

Suppose

We collect a sample and calculate

Using the sampling distribution, we can determine how unusual 104 would be if the true population mean were actually 100.

This leads to

• Test statistics
• p-values
• Critical values
• Hypothesis testing

which are the next topics in your Statistics module.

18. CLT vs Law of Large Numbers

These two concepts are often confused.

Law of Large Numbers

As sample size increases

It tells us that the sample mean approaches the population mean.

Central Limit Theorem

As sample size increases

It tells us about the shape of the sampling distribution.

19. Important Formula Summary

Mean of sampling distribution

Standard error

Approximate sampling distribution

for sufficiently large (n), under appropriate conditions.

20. Practical Data Science Example

• Imagine an e-commerce company has 10 million customers.
• You want to estimate the average order value.
• Studying every customer is impractical.

Instead

Now we can use statistical methods to estimate the true average order value of all 10 million customers.

This is the practical power of CLT.

21. Key Takeaways

Remember these points

• CLT is about sampling distributions, not the original population.
• The distribution of sample means tends to become approximately normal as sample size increases.
• The original population does not need to be normal.
• Randomness and independence assumptions matter.
• Larger sample sizes generally produce a more stable sampling distribution.

The mean of the sampling distribution is

Its standard deviation is the standard error

• CLT is a foundation for confidence intervals and hypothesis testing.
• Easy way to remember
• Law of Large Numbers: "The sample mean gets closer to the population mean."
• Central Limit Theorem: "The distribution of sample means becomes approximately normal."
• One-line definition

Next: 5.10 Hypothesis Testing — where we'll use sample data to determine whether there is enough statistical evidence to reject a claim about a population.

Hypothesis testing is a statistical method used to determine whether there is enough evidence in sample data to support or reject a claim about a population.

In simple terms

Hypothesis testing helps us decide whether an observed result is likely to be real or could reasonably have occurred by chance.

It is widely used in Data Science, Machine Learning, A/B testing, medical research, business analytics, and scientific experiments.

1. Simple Example

Suppose a company claims

"Our new website increases the average order value."

We collect data from customers and observe that the new website has a higher average order value.

The question is

Is the increase actually caused by the new website, or could it simply be random variation?

Hypothesis testing gives us a framework for answering this question.

2. The Two Hypotheses

Hypothesis testing starts with two competing statements.

Null Hypothesis

The null hypothesis, written as (H_0), represents the default assumption.

Usually it says

There is no effect, no difference, or no relationship.

Alternative Hypothesis

The alternative hypothesis, written as (H_1) or (H_a), represents what we are trying to find evidence for.

Usually it says

There is an effect, difference, or relationship.

Example

Suppose the current average delivery time is

• A company introduces a new delivery system.
• We want to know whether the average delivery time has changed.
• Null hypothesis

There is no change.

Alternative hypothesis

There is a change.

3. Hypothesis Testing Workflow

A typical hypothesis test follows

4. Significance Level

The significance level, represented by (\alpha), is the threshold we use for deciding whether evidence against (H_0) is strong enough.

Common choices

or

or

The most commonly used value is

This corresponds to a 5% significance level.

5. Understanding α = 0.05

If

we are using a rule that treats results with sufficiently small p-values as strong evidence against (H_0).

It is important to understand

α = 0.05 does NOT mean there is a 5% probability that the null hypothesis is true.

That's a common misunderstanding.

6. Test Statistic

A test statistic measures how far the observed sample result is from what we would expect under the null hypothesis.

For example, for a z-test

Where

• (\bar{x}) = sample mean
• (\mu_0) = mean assumed under (H_0)
• (\sigma) = population standard deviation
• (n) = sample size

7. Example: One-Sample Test

Suppose a company claims

Average package weight is 50 kg.

We collect a sample

Sample mean

Population standard deviation

We want to test whether the actual mean is different from 50 kg.

Step 1: Hypotheses

Step 2: Calculate z-score

The sample mean is 2 standard errors above the hypothesized population mean.

8. Two-Tailed Test

Our alternative hypothesis was

We don't care whether the mean is higher or lower.

We are testing for a difference in either direction.

This is called a

Conceptually

Reject Fail to reject Reject

↓ ↓ ↓

-------|------------------|----------------|-------

-1.96 0 +1.96

For a two-tailed z-test at

the critical values are approximately

9. One-Tailed Test

Suppose the company specifically claims

"The new system reduces delivery time."

Now we may test

versus

This is a left-tailed test.

If instead we want to test whether the mean is greater

we use a right-tailed test.

10. Types of Tests

Two-tailed

Tests for a difference in either direction.

Left-tailed

Tests whether the value is smaller.

Right-tailed

Tests whether the value is larger.

11. p-value

The p-value is one of the most important concepts in hypothesis testing.

It measures how compatible the observed result is with the null hypothesis.

A small p-value means

• The observed result would be relatively unusual if (H_0) were true.
• We compare the p-value with (\alpha).
• Decision rule

If

then

If

then

12. Example Using p-value

Suppose

and our test produces

Since

we reject (H_0).

Conclusion

There is statistically significant evidence against the null hypothesis at the 5% significance level.

Another example

Suppose

Since

• we fail to reject (H_0).
• This does not prove that (H_0) is true.
• It means that we don't have sufficient evidence to reject it.

13. "Reject" vs "Fail to Reject"

This terminology is important.

Don't say

"The null hypothesis is true."

when

Instead say

We fail to reject the null hypothesis.

Why?

Because failing to find sufficient evidence against (H_0) does not prove (H_0) is true.

14. Type I Error

A Type I error occurs when we

This is also called a false positive.

The probability of a Type I error is controlled by

For example, if

the test is designed with a 5% significance threshold for Type I error.

15. Type II Error

A Type II error occurs when we

This is often described as a false negative.

Its probability is represented by

And

16. Type I vs Type II Error

Easy way to remember

Type I → False Positive

Type II → False Negative

17. Statistical Significance

If

we often describe the result as

If

we generally describe it as

But remember

Statistical significance does not necessarily mean practical significance.

18. Statistical vs Practical Significance

Suppose a new website increases average customer spending by

and because the company has millions of customers, the result produces

It is statistically significant.

But ₹1 might not be practically important to the business.

Therefore, always consider

• Effect size
• Business impact
• Confidence interval
• Cost
• Practical importance

19. Common Hypothesis Tests

Different situations require different tests.

You'll study Chi-Square and ANOVA later in this module.

20. T-Test vs Z-Test

A simplified rule

• Z-test
• Often used when population standard deviation (\sigma) is known and normality/large-sample assumptions are appropriate.
• T-test

Commonly used when population standard deviation is unknown and is estimated from the sample.

For a one-sample t-test

where (s) is the sample standard deviation.

In practical Data Science, t-tests are often encountered more frequently than z-tests because population standard deviation is usually unknown.

21. Hypothesis Testing Using Python

SciPy provides many statistical tests.

One-sample t-test

Suppose

from scipy.stats import ttest_1samp
data = [52, 49, 51, 53, 50, 48, 52, 51, 49, 54]
result = ttest_1samp(data, popmean=50)
print("t-statistic:", result.statistic)
print("p-value:", result.pvalue)

We can then compare

alpha = 0.05
if result.pvalue < alpha:
print("Reject H0")
else:
print("Fail to reject H0")

22. Two-Sample t-Test

Suppose we want to compare two groups

from scipy.stats import ttest_ind
group_a = [10, 12, 11, 13, 15]
group_b = [8, 9, 10, 11, 9]
result = ttest_ind(group_a, group_b)
print("t-statistic:", result.statistic)
print("p-value:", result.pvalue)

This tests whether there is evidence that the two group means differ, subject to the assumptions of the test.

23. Paired t-Test

Suppose we measure employee productivity

Before training

70, 72, 68, 75, 71

After training

75, 76, 72, 78, 77

Because the same employees are measured twice, this is paired data.

from scipy.stats import ttest_rel
before = [70, 72, 68, 75, 71]
after = [75, 76, 72, 78, 77]
result = ttest_rel(before, after)
print("t-statistic:", result.statistic)
print("p-value:", result.pvalue)

24. A/B Testing

Hypothesis testing is heavily used in A/B testing.

Suppose an e-commerce website has

Version A

Conversion rate

Version B

Conversion rate

The observed difference is

The question is

Is Version B genuinely better, or could this difference be due to random sampling variation?

We formulate hypotheses and perform an appropriate statistical test.

This is one of the most important real-world applications of hypothesis testing.

25. Complete Example

Suppose

A company claims the average delivery time is 30 minutes.

We collect a sample and obtain

We want to know whether the average delivery time is different from 30 minutes.

Step 1 — Null hypothesis

Step 2 — Alternative hypothesis

Step 3 — Significance level

• Step 4 — Test
• Since the population standard deviation is unknown, a one-sample t-test is appropriate.
• Step 5 — Calculate test statistic

The exact p-value would be calculated using the t-distribution with

Step 6 — Decision

If the resulting p-value is below 0.05

we reject (H_0).

Step 7 — Business conclusion

We would say

There is statistically significant evidence that the average delivery time differs from 30 minutes.

Notice that we don't simply say

"The alternative hypothesis is proven."

Statistical testing provides evidence, not absolute proof.

26. Important Assumptions

Different statistical tests have different assumptions, but common considerations include:

• Random or appropriately designed sampling
• Independence of observations
• Appropriate measurement scale
• Distributional assumptions where required
• Adequate sample size
• Appropriate test selection

You should not blindly run a t-test just because you have two numbers/groups.

The test must match the data and research question.

27. Hypothesis Testing Cheat Sheet

1. Define the question

↓

2. State H₀

↓

3. State Hₐ

↓

4. Choose α

↓

5. Select statistical test

↓

6. Calculate test statistic

↓

7. Calculate p-value

↓

8. Compare p with α

↓

9. Reject / Fail to Reject H₀

↓

10. Interpret in business/scientific context

28. Most Important Formulas

Z-test statistic

One-sample t-test

Decision rule

29. Key Takeaways

Remember these concepts

• (H_0) = null hypothesis, usually no effect/difference.
• (H_a) = alternative hypothesis.
• (\alpha) = significance level.
• Test statistic = measures how far the sample result is from the null assumption.
• p-value = measures how surprising the observed result is under (H_0).
• Reject (H_0) when the p-value is sufficiently small.
• Fail to reject (H_0) when there isn't enough evidence.
• Type I error = false positive.
• Type II error = false negative.
• Statistical significance ≠ practical significance.
• Easy way to remember
• Hypothesis → Test → p-value → Decision → Conclusion

And the key decision rule

provided (0.05) was chosen as the significance level beforehand.

The next topic, 5.11 p-value, goes deeper into exactly what a p-value means, what it does NOT mean, and how to interpret p-values correctly in real-world Data Science and A/B testing.

The p-value is one of the most important concepts in hypothesis testing.

In simple terms

The p-value tells us how unusual the observed data would be if the null hypothesis (H_0) were true.

A small p-value means the observed result is difficult to explain under (H_0), so we have stronger evidence against (H_0).

1. Basic Idea

Suppose a company claims

"Our new process does not change average delivery time."

We set

Then we collect sample data and find

The question is

If the true average really is 30 minutes, how likely is it to observe a result this extreme just because of random sampling?

The answer is related to the p-value.

2. Definition

For a hypothesis test, the p-value is

The phrase "as extreme or more extreme" is important.

3. What Does a Small p-value Mean?

Suppose

This means that, assuming the null hypothesis is true, the observed result or a more extreme result would have probability about 1%, according to the test model.

That is relatively unusual.

Therefore, we have strong evidence against (H_0).

4. What Does a Large p-value Mean?

Suppose

This means the observed result is not particularly unusual under (H_0).

Therefore, we don't have enough evidence to reject (H_0).

Important

A large p-value does not prove that (H_0) is true.

5. p-value and Significance Level

We compare the p-value with the significance level

A common choice is

Decision rule

If

then

If

then

6. Examples

Example 1

Since

we reject (H_0).

Conclusion: Statistically significant evidence against (H_0).

Example 2

Since

we reject (H_0).

Example 3

Since

we fail to reject (H_0).

7. p-value Does NOT Mean Probability That H₀ Is True

This is one of the biggest statistical misconceptions.

Suppose

It is incorrect to say

"There is a 3% probability that the null hypothesis is true."

• Wrong.

The p-value is calculated assuming (H_0) is true.

It tells us about the compatibility of the observed data with (H_0), not the probability that (H_0) itself is true.

8. p-value Does NOT Tell You the Size of the Effect

Suppose

This indicates strong statistical evidence against (H_0).

But it doesn't tell us whether the effect is

• Tiny
• Moderate
• Huge

For that, we need to examine the effect size and often a confidence interval.

9. Statistical Significance vs Practical Significance

Suppose an e-commerce company tests a new website.

Version A

Conversion rate

Version B

Conversion rate

With a very large sample, we might obtain

This is statistically significant.

But the improvement is only

It may have almost no practical business value.

Therefore

A small p-value does not automatically mean the effect is important.

10. Visual Intuition

Imagine the null hypothesis predicts that most test statistics should be near zero

H₀ distribution

/\

/ \

/ \

/ \

_________________/________________\____________

-2 0 +2 +4

↑

observed

If the observed statistic is far into the tail, the area beyond it becomes small.

That tail area corresponds to the p-value for the appropriate one- or two-sided test.

So

Farther into the tail → smaller p-value → stronger evidence against (H_0).

11. One-Tailed vs Two-Tailed p-values

The p-value depends on the alternative hypothesis.

Two-tailed test

Suppose

We care about extreme results in both directions.

left tail right tail

↓ ↓

_______████________________________████_______

Right-tailed test

Only the right tail matters.

Left-tailed test

Only the left tail matters.

Therefore, you cannot interpret a p-value correctly without knowing the test direction.

12. Example with a Z-Test

Suppose

We calculate

For a two-tailed test, the p-value is approximately

At

we have

Therefore

The result is statistically significant at the 5% level.

13. p-value and Z-score

For a two-tailed z-test

For example

gives approximately

As (|z|) increases, the p-value decreases.

| (|z|) | Approx. two-tailed p-value | |---:|---:| | 0 | 1.000 | | 1 | 0.317 | | 1.96 | 0.050 | | 2 | 0.0455 | | 2.58 | 0.010 | | 3 | 0.0027 |

This is why a test statistic far from zero usually produces a small p-value.

14. p-value in Python

SciPy can calculate p-values for statistical tests.

One-sample t-test

from scipy.stats import ttest_1samp
data = [52, 49, 51, 53, 50, 48, 52, 51, 49, 54]
result = ttest_1samp(data, popmean=50)
print("t-statistic:", result.statistic)
print("p-value:", result.pvalue)

You can then make the decision

alpha = 0.05
if result.pvalue < alpha:
print("Reject H0")
else:
print("Fail to reject H0")

15. Two-Sample Test

Suppose we want to compare two groups

from scipy.stats import ttest_ind
group_a = [10, 12, 11, 13, 15]
group_b = [8, 9, 10, 11, 9]
result = ttest_ind(group_a, group_b)
print("t-statistic:", result.statistic)
print("p-value:", result.pvalue)

The p-value helps determine whether there is evidence that the group means differ.

16. p-value in A/B Testing

Suppose an e-commerce company tests two website versions.

Observed improvement

We perform an appropriate test for the conversion rates.

Suppose the result is

With

we reject (H_0).

We can say

There is statistically significant evidence that the conversion rates differ.

But we should also consider whether a 0.3 percentage-point improvement is commercially meaningful.

17. Common p-value Thresholds

There is nothing mathematically magical about 0.05, but it is widely used.

These are conventional descriptions, not absolute scientific rules.

18. p-value and Sample Size

An important point

p-values are strongly affected by sample size.

With a very large sample, even a tiny effect can produce a very small p-value.

For example

Conversely

That's why p-value should not be considered alone.

Also examine

• Effect size
• Confidence interval
• Sample size
• Statistical power
• Practical significance

19. p-value vs Confidence Interval

These two concepts work together.

Suppose we estimate a difference between two groups

with a 95% confidence interval

Because the interval does not include zero, this corresponds to evidence of a non-zero difference in the usual two-sided 5% framework.

If the confidence interval were

zero is included, so the corresponding two-sided test would generally not be statistically significant at the 5% level.

20. Common Mistakes

• Mistake 1

"p = 0.03 means there is a 3% chance H₀ is true."

Wrong.

• Mistake 2

"p = 0.20 proves H₀ is true."

Wrong.

• Mistake 3

"p < 0.05 means the effect is large."

Wrong.

• Mistake 4

"p = 0.051 means absolutely nothing happened."

Wrong.

The difference between

and

is not a magical scientific boundary. Statistical evidence should be interpreted in context.

21. Correct Interpretation

Suppose

A good interpretation is

Assuming the null hypothesis is true, the observed result (or a more extreme result) would be relatively unlikely under the statistical model. Therefore, at the 5% significance level, we reject the null hypothesis.

A shorter practical interpretation

The result is statistically significant at the 5% level.

22. Hypothesis Testing Decision Table

Remember that these are interpretations relative to the chosen significance level.

23. Complete Example

A company claims that its average customer waiting time is

We test

Suppose our statistical test gives

and

Choose

Compare

Therefore

Conclusion

There is statistically significant evidence that the average waiting time differs from 30 minutes.

But we should still examine the estimated difference and confidence interval to understand how large the difference actually is.

24. The Most Important Rule

Remember

For the commonly used

we have

25. Quick Revision

• Easy way to remember
• Small p → data is surprising under H₀ → evidence against H₀.
• Large p → data isn't surprising under H₀ → insufficient evidence against H₀.

The next topic, 5.12 Confidence Intervals, connects directly to p-values and will show how to estimate a population parameter together with a range of plausible values.

A Confidence Interval (CI) is a range of values calculated from sample data that is used to estimate an unknown population parameter.

In simple terms

A confidence interval gives us an estimate plus a measure of uncertainty.

Instead of saying

"The average customer waiting time is 30 minutes."

we might say

"The estimated average is 30 minutes, with a 95% confidence interval from 28 to 32 minutes."

That gives much more information.

1. Why Do We Need Confidence Intervals?

Suppose a company has 1 million customers.

We cannot measure every customer.

Instead, we take a sample of 1,000 customers.

Suppose

We estimate that the population mean is approximately 50.

But the actual population mean might be

• 49
• 50
• 51
• 52
• We don't know exactly.
• A confidence interval represents this uncertainty.
• Estimated population mean

↓

─────────────────●─────────────────

48 50 52

2. General Form

A confidence interval generally has the form

For a population mean

For example

gives

3. What Does 95% Confidence Mean?

This is one of the most commonly misunderstood concepts.

Suppose we calculate a 95% confidence interval

The classical interpretation is

If we repeatedly took samples and constructed confidence intervals using the same method, approximately 95% of those intervals would contain the true population parameter.

It is not technically correct to say

"There is a 95% probability that the fixed population mean is inside this particular interval."

The population parameter is treated as fixed; the interval is what varies from sample to sample.

4. Confidence Level

Common confidence levels are

• 90%
• 95%
• 99%

The most commonly used is

Higher confidence requires a wider interval.

Generally

and

5. Example

Suppose a sample gives

and the calculated margin of error is

Then the 95% confidence interval is

Therefore

We would report

The estimated population mean is 100, with a 95% confidence interval of 95 to 105.

6. Margin of Error

The margin of error (MOE) represents how far the confidence interval extends from the sample estimate.

For example

Therefore

and

So

7. Confidence Interval Using Z-Score

If the population standard deviation (\sigma) is known, a confidence interval for the population mean can be calculated as:

Where

• (\bar{x}) = sample mean
• (\sigma) = population standard deviation
• (n) = sample size
• (z_{\alpha/2}) = critical z-value

8. Important Z Critical Values

For common confidence levels

For a 95% confidence interval

9. 95% Confidence Interval Example

Suppose

For 95% confidence

First calculate the standard error

Now calculate margin of error

Therefore

10. Confidence Interval Using t-Distribution

In real-world situations, the population standard deviation is usually unknown.

Instead, we use the sample standard deviation (s).

Then we commonly use the t-distribution.

The formula is

Where

• (s) = sample standard deviation
• (n-1) = degrees of freedom
• (t_{\alpha/2,n-1}) = critical t-value

This is very important for practical statistics.

11. Example Using t-Distribution

Suppose

For a 95% confidence interval

The critical t-value is approximately

Standard error

Margin of error

Therefore

Approximately

12. Confidence Interval in Python

SciPy can calculate confidence intervals.

For a one-sample mean

import numpy as np
from scipy import stats
data = np.array([45, 50, 52, 48, 55, 51, 49, 53, 47, 50])
result = stats.ttest_1samp(data, popmean=0)
confidence_interval = result.confidence_interval(
    confidence_level=0.95
)

print(confidence_interval)

For a simple mean CI using the t-distribution, we can also calculate it directly

import numpy as np
from scipy.stats import t
data = np.array([45, 50, 52, 48, 55, 51, 49, 53, 47, 50])
n = len(data)
mean = np.mean(data)
std = np.std(data, ddof=1)
se = std / np.sqrt(n)
t_critical = t.ppf(0.975, df=n-1)
margin_of_error = t_critical * se
lower = mean - margin_of_error
upper = mean + margin_of_error
print("Mean:", mean)
print("95% CI:", lower, upper)

13. Confidence Interval Using NumPy/Pandas

Pandas is useful for calculating sample statistics

import pandas as pd
data = pd.Series([45, 50, 52, 48, 55, 51, 49, 53, 47, 50])
print("Mean:", data.mean())
print("Standard deviation:", data.std())

Then we can use SciPy to construct the confidence interval.

14. Effect of Sample Size

Remember the standard error

As

then

Therefore, larger samples generally produce narrower confidence intervals.

Example

Suppose

For

For

So the second estimate is more precise.

15. Effect of Confidence Level

Higher confidence produces a wider interval.

Suppose

and the standard error is 2.

90% CI

Approximately

95% CI

99% CI

Approximately

Notice

Higher confidence → wider interval.

16. Confidence Level vs Precision

There is a trade-off

So a 99% CI gives more confidence but less precision than a 90% CI, assuming the same data.

17. Confidence Interval and p-value

Confidence intervals and hypothesis tests are closely related.

Suppose we are testing

At

we can look at the 95% confidence interval.

Case 1

The hypothesized value 100 is not inside the interval.

Therefore, the corresponding two-sided test would reject

Case 2

100 is inside the interval.

Therefore, we would generally fail to reject (H_0) at the 5% level.

18. Confidence Interval for a Difference

Confidence intervals aren't limited to one population mean.

Suppose we compare

Group A mean

Group B mean

Estimated difference

Suppose the 95% CI for the difference is

Because zero is not inside the interval, there is evidence of a difference at the corresponding 5% level.

If instead

contains zero, the data would not provide statistically significant evidence of a non-zero difference at the 5% level.

19. Confidence Interval for a Proportion

Confidence intervals can also estimate proportions.

Suppose

1,000 customers are surveyed.

600 say they are satisfied.

Sample proportion

So

We can construct a confidence interval around that 60% estimate.

For sufficiently large samples, an approximate 95% CI is

20. Real-World Example

Suppose an online store has a customer satisfaction survey.

Sample

Satisfied customers

Sample proportion

So

Suppose the 95% CI is approximately

We could report

The estimated customer satisfaction rate is 75%, with a 95% confidence interval of approximately 73.1% to 76.9%.

This is much more informative than simply saying

"Customer satisfaction is 75%."

21. Common Mistakes

• Mistake 1

"A 95% confidence interval means there is a 95% chance that the true mean is inside this particular interval."

Not the classical frequentist interpretation.

• Mistake 2

"A wider confidence interval is always better."

No.

A wider interval gives greater coverage confidence at the cost of precision.

• Mistake 3

"A larger sample always gives a wider interval."

Usually the opposite.

Larger samples generally produce narrower intervals because the standard error decreases.

• Mistake 4

"If the CI contains the sample mean, something is wrong."

No. The sample estimate is typically at the center of a symmetric confidence interval.

22. Confidence Interval Cheat Sheet

23. Important Formulas

Known population standard deviation

Unknown population standard deviation

Margin of error

24. Connection with Previous Topics

You have now covered

These concepts are strongly connected.

For example

comes from sampling theory and is used in confidence intervals.

Then

gives the confidence interval.

And hypothesis testing and confidence intervals provide two closely related ways of making statistical inferences.

Key Takeaway

A confidence interval provides a range of plausible values for an unknown population parameter, together with a specified confidence level.

Remember the three most important ideas

The next topic in your syllabus is 5.13 Correlation, where we move from measuring the distribution of a single variable to understanding how two variables move together.

Correlation is a statistical measure that tells us about the strength and direction of the relationship between two variables.

In simple terms

Correlation tells us whether two variables tend to move together, and how strongly they do so.

Examples

• Study hours ↔ Exam scores
• Temperature ↔ Ice-cream sales
• Advertising spend ↔ Sales
• Age ↔ Income

genui{"learning_viz":{"type_id":"CORRELATION"}}

1. Simple Example

Suppose we have

As study hours increase, exam scores also tend to increase.

Therefore, these two variables have a positive correlation.

2. Correlation Coefficient

Correlation is commonly represented by

For Pearson correlation, the value of (r) lies between

3. Positive Correlation

In a positive correlation, both variables tend to increase together.

Example

Suppose

This indicates a strong positive linear relationship.

Conceptually

Score

|

90| *

80| *

70| *

60| *

50| *

40| *

|________________________

Study Hours

4. Negative Correlation

In a negative correlation, one variable tends to increase while the other decreases.

Example

Suppose

This indicates a strong negative linear relationship.

Demand

|

90| *

80| *

70| *

60| *

50| *

40| *

|________________________

Price

5. Zero Correlation

If

there is little or no linear relationship between the variables.

Example

Exam Score

|

90| * *

80| *

70| * *

60| * *

50| * *

|____________________

Shoe Size

There is no obvious linear pattern.

However, an important point

Correlation of 0 does not necessarily mean there is no relationship at all.

There could be a strong non-linear relationship.

6. Pearson Correlation

The most commonly used correlation coefficient is Pearson's correlation coefficient.

Its formula is

You don't necessarily need to calculate this manually in Data Science because Python libraries can calculate it directly.

7. Understanding the Formula

The formula compares how the two variables vary from their respective means.

If

• (X) is above its mean
• (Y) is also above its mean
• their product contributes positively.

If

• (X) is above its mean
• (Y) is below its mean
• their product contributes negatively.

This is why correlation captures the direction of the relationship.

8. Correlation vs Covariance

You will study covariance in the next topic.

The concepts are closely related.

• Covariance
• Tells us whether two variables move together and in which direction.
• Correlation

Standardizes covariance so that its value always lies between

A useful relationship is

where

• (\operatorname{Cov}(X,Y)) = covariance
• (\sigma_X) = standard deviation of X
• (\sigma_Y) = standard deviation of Y

9. Correlation Does NOT Mean Causation

This is one of the most important rules in statistics.

Correlation does not imply causation.

Suppose we observe

and

• These variables may have positive correlation.
• But buying ice cream doesn't necessarily cause swimming pool accidents.
• A third variable—temperature—may influence both.
• Temperature

↙ ↘

↓ ↓

Ice Cream Sales Pool Accidents

This is called a confounding variable.

10. Strong Correlation Doesn't Mean Causation

Suppose

• That is a very strong correlation.
• It still doesn't prove that one variable causes the other.
• Correlation tells us about association, not causality.

To establish causality, we generally need stronger evidence, often through controlled experiments or appropriate causal-inference methods.

11. Correlation and Outliers

Correlation can be strongly affected by outliers.

Suppose most points follow this pattern

Y

|

| *

| *

| *

| *

|________________ X

Now add one extreme point.

That single point can substantially change the calculated correlation.

Therefore, when analyzing correlation

Always inspect the data and preferably visualize the relationship using a scatter plot.

12. Correlation and Non-Linear Relationships

Pearson correlation measures linear association.

Consider

There is a clear relationship between (X) and (Y), but it is curved rather than linear.

A Pearson correlation close to zero does not mean there is no relationship.

For example

Y

|

| * *

| * *

| * *

| * *

| *

|________________ X

This is a strong non-linear relationship.

Therefore

Always look at the scatter plot, not just the correlation coefficient.

13. Correlation in Python

Using NumPy

import numpy as np
x = [1, 2, 3, 4, 5]
y = [2, 4, 6, 8, 10]
correlation = np.corrcoef(x, y)[0, 1]
print(correlation)

Output

0.9999999999999999

Approximately

This is a perfect positive linear relationship.

14. Using Pandas

import pandas as pd
df = pd.DataFrame({
    "study_hours": [1, 2, 3, 4, 5],
    "score": [45, 50, 60, 70, 80]
})

print(df["study_hours"].corr(df["score"]))

This calculates the Pearson correlation by default.

15. Correlation Matrix

In Data Science, we often have many variables.

For example

import pandas as pd
df = pd.DataFrame({
    "age": [20, 25, 30, 35, 40],
    "income": [20, 30, 45, 55, 70],
    "experience": [1, 3, 5, 8, 12]
})

print(df.corr())

Output might look conceptually like

age income experience

age 1.0 0.95 0.98

income 0.95 1.0 0.92

experience 0.98 0.92 1.0

This is called a correlation matrix.

16. Correlation Heatmap

Correlation matrices are often visualized using a heatmap.

import seaborn as sns
import matplotlib.pyplot as plt

sns.heatmap(

df.corr(),
annot=True,
cmap="coolwarm"

)

plt.show()

A heatmap makes it easier to identify

• Strong positive relationships
• Strong negative relationships
• Weak relationships

This is commonly used during Exploratory Data Analysis (EDA).

17. Spearman Correlation

Pearson is not the only correlation method.

Spearman's rank correlation measures the strength of a monotonic relationship using ranks.

It can be useful when

• Data is ordinal
• Relationship isn't linear
• Outliers make Pearson less appropriate
• Variables aren't normally distributed

Python

from scipy.stats import spearmanr
x = [1, 2, 3, 4, 5]
y = [10, 20, 25, 40, 100]
result = spearmanr(x, y)
print("Correlation:", result.statistic)
print("p-value:", result.pvalue)

18. Pearson vs Spearman

19. Correlation and Machine Learning

Correlation is very useful in ML preprocessing.

Feature selection

Suppose

Feature A → Target correlation = 0.90

Feature B → Target correlation = 0.05

Feature A may be more useful for predicting the target, although correlation alone does not determine whether a feature should be selected.

Multicollinearity

Suppose

Age ↔ Years of Experience

has

These variables are highly correlated.

In some models, highly correlated predictors can create multicollinearity problems.

This is especially important in linear regression.

20. Correlation Does Not Automatically Mean a Feature Is Useful

Suppose

between a feature and target.

It doesn't necessarily mean the feature is useless.

The relationship could be

• Non-linear
• Conditional on another variable
• Interaction-based

For example

can have little Pearson correlation while still having a very strong deterministic relationship.

Therefore, correlation should be one tool among many.

21. Statistical Significance of Correlation

We can also test whether an observed correlation is statistically significant.

For example

where (\rho) is the population correlation.

Alternative

Python

from scipy.stats import pearsonr
x = [1, 2, 3, 4, 5]
y = [2, 4, 6, 8, 10]
result = pearsonr(x, y)
print("Correlation:", result.statistic)
print("p-value:", result.pvalue)

The result provides both

Correlation coefficient

p-value

So you can evaluate both strength and statistical evidence.

22. Example: Sales and Advertising

Suppose a company records

The correlation may be close to

This indicates a strong positive linear association.

But we should not automatically conclude

Advertising caused all of the increase in sales.

Other factors could include

• Seasonality
• Discounts
• Competitor activity
• Economic conditions
• Product launches

23. Quick Interpretation Guide

A rough practical interpretation

| (|r|) | Approximate strength | |---:|---| | 0.00–0.19 | Very weak | | 0.20–0.39 | Weak | | 0.40–0.59 | Moderate | | 0.60–0.79 | Strong | | 0.80–1.00 | Very strong |

These are rules of thumb, not universal scientific thresholds.

Context matters.

24. Key Formulas

Pearson correlation

Range

Interpretation

25. Key Takeaways

Remember these points

• Correlation measures association between two variables.
• Pearson correlation ranges from -1 to +1.
• Positive correlation → variables tend to increase together.
• Negative correlation → one tends to increase as the other decreases.
• (r\approx0) means little linear association.
• Correlation does not imply causation.
• Pearson correlation measures linear association.
• Outliers can strongly affect correlation.
• Always consider a scatter plot.
• Spearman correlation is useful for rank/monotonic relationships.
• Correlation is widely used in EDA and feature analysis.
• Easy way to remember
• +1 → Move together perfectly
• 0 → No linear relationship
• −1 → Move perfectly in opposite directions

And the most important warning

Next in your syllabus is 5.14 Covariance, which is closely related to correlation and will explain how two variables vary together before that relationship is standardized into the -1 to +1 correlation scale.

Covariance is a statistical measure that tells us how two variables change together.

In simple terms

Covariance tells us whether two variables tend to increase together, decrease together, or move in opposite directions.

For example

• Advertising spend and sales
• Temperature and ice-cream sales
• Study hours and exam scores
• Age and income

1. Basic Idea

Suppose we have

As study hours increase, exam scores also tend to increase.

Therefore, the covariance is positive.

2. Types of Covariance

Covariance can be

Positive

Both variables tend to move in the same direction.

Example

X ↑ → Y ↑

X ↓ → Y ↓

Negative

The variables tend to move in opposite directions.

Example

X ↑ → Y ↓

X ↓ → Y ↑

Approximately Zero

There is little evidence of a linear co-movement.

Important: zero covariance does not necessarily mean there is no relationship at all.

3. Covariance Formula

For a population

[ \boxed{ \operatorname{Cov}(X,Y)

\frac{ \sum_{i=1}^{N}(x_i-\mu_X)(y_i-\mu_Y) }{N} } ]

For a sample

[ \boxed{ s_{XY}

\frac{ \sum_{i=1}^{n}(x_i-\bar{x})(y_i-\bar{y}) }{n-1} } ]

Where

• (X,Y) = two variables
• (\bar{x},\bar{y}) = sample means
• (n) = sample size

4. Step-by-Step Example

Consider

X = [2, 4, 6]
Y = [10, 20, 30]

Step 1: Calculate means

For X

For Y

Step 2: Calculate deviations

Step 3: Multiply deviations

Sum

Step 4: Calculate Population Covariance

The covariance is positive, which tells us that X and Y tend to increase together.

5. Sample Covariance

For the same data, if these observations represent a sample

Notice the difference

6. Why Do We Multiply Deviations?

This is the key idea behind covariance.

Suppose both values are above their means

and

Then

So covariance becomes positive.

If both are below their means

Again, positive.

But if one is above its mean and the other is below

So covariance becomes negative.

Therefore

Covariance captures whether two variables tend to deviate from their means in the same or opposite directions.

7. Positive Covariance Example

Consider

X = [1, 2, 3, 4, 5]
Y = [10, 20, 30, 40, 50]

As X increases, Y also increases.

Therefore

8. Negative Covariance Example

Consider

X = [1, 2, 3, 4, 5]
Y = [50, 40, 30, 20, 10]

As X increases, Y decreases.

Therefore

9. Covariance vs Correlation

This is extremely important.

• Both covariance and correlation describe how two variables move together.
• But they are not the same.
• Covariance
• Indicates direction.
• Its magnitude depends on the units of the variables.
• Has no fixed range.
• Correlation
• Indicates direction and standardized strength.
• Always lies between -1 and +1.
• Is unitless.

The relationship is

Where

• (r) = correlation
• (\operatorname{Cov}(X,Y)) = covariance
• (\sigma_X) = SD of X
• (\sigma_Y) = SD of Y

10. Example of the Difference

Suppose

This tells us the variables have positive co-movement.

But is 50 a strong relationship?

We cannot determine that easily from covariance alone.

If

and

then

That's a relatively weak positive linear correlation.

So

Covariance tells direction; correlation gives standardized strength.

11. Why Covariance Has Units

Suppose

X = income in ₹
Y = age in years

Then covariance has units

This makes covariance difficult to compare across different datasets or variable pairs.

Correlation solves this problem because it is unitless.

12. Covariance in Python

NumPy

import numpy as np
x = [2, 4, 6]
y = [10, 20, 30]
covariance_matrix = np.cov(x, y)
print(covariance_matrix)

Output

The off-diagonal values represent the sample covariance

13. Population Covariance with NumPy

By default, np.cov() uses

for the denominator.

For population covariance, use

import numpy as np
x = [2, 4, 6]
y = [10, 20, 30]
covariance_matrix = np.cov(
    x,
    y,
    ddof=0
)

print(covariance_matrix)

The off-diagonal covariance will be

14. Covariance Using Pandas

import pandas as pd
df = pd.DataFrame({
    "X": [2, 4, 6],
    "Y": [10, 20, 30]
})

print(df["X"].cov(df["Y"]))

Output

10.0

Pandas .cov() calculates sample covariance by default.

15. Covariance Matrix

When we have many variables, we can calculate a covariance matrix.

Suppose

• Age
• Income
• Experience

We can calculate

print(df.cov())

A covariance matrix might look like

Age Income Experience

Age 25 1200 18

Income 1200 900000 15000

Experience 18 15000 12

The diagonal contains each variable's variance.

The off-diagonal values contain covariances between pairs of variables.

16. Covariance Matrix Structure

For variables (X) and (Y)

Since

the matrix is symmetric.

17. Covariance in Machine Learning

Covariance is important in several ML and statistical techniques.

1. Feature relationships

It helps identify how numerical features vary together.

2. Principal Component Analysis (PCA)

PCA uses the covariance structure of variables to find directions of maximum variance.

A simplified workflow is

3. Multivariate statistics

Covariance matrices are fundamental when dealing with multiple variables simultaneously.

4. Gaussian distributions

Multivariate normal distributions use covariance matrices to describe relationships between variables.

18. Covariance and PCA

Suppose we have

Height

Weight

People who are taller often tend to weigh more.

Therefore

PCA can use this covariance structure to identify the main direction in which the data varies.

This is why covariance is an important foundation for understanding dimensionality reduction.

19. Covariance Does Not Mean Causation

Just like correlation

If two variables have positive covariance, it only means they tend to move together.

It does not prove that one causes the other.

20. Covariance vs Variance

These concepts are closely related.

Variance

Measures how one variable varies around its mean.

Covariance

Measures how two variables vary together.

So you can think of covariance as a generalization of variance to two variables.

21. Variance, Covariance and Correlation

22. Important Example

Suppose

and

Both variables increase together.

Therefore

and

because the relationship is perfectly linear.

Now suppose

As X increases, Y decreases.

Therefore

and

because the relationship is perfectly negative linear.

23. Key Formulas

Population covariance

Sample covariance

Correlation

Variance as covariance

24. Quick Revision

Remember

• Variance → How one variable varies.
• Covariance → How two variables vary together.
• Correlation → Standardized covariance.

And

Easy memory trick

Co-variance = variables vary together.

If they generally move

• Together → Positive covariance
• Opposite → Negative covariance
• No clear linear co-movement → Near-zero covariance

The next topic in your syllabus is 5.15 Chi-Square Test, which moves from relationships between numerical variables to testing relationships and distributions involving categorical data.

The Chi-Square ((\chi^2)) test is a statistical test mainly used with categorical data.

It helps us determine whether the difference between observed frequencies and expected frequencies is large enough to provide evidence against the null hypothesis.

In simple terms

Chi-Square tests whether categorical observations differ from what we would expect by chance or under a specified assumption.

1. Where Is Chi-Square Used?

There are two major applications

1. Chi-Square Goodness-of-Fit Test

Checks whether observed frequencies match an expected distribution.

Example

Is a die actually fair?

2. Chi-Square Test of Independence

Checks whether two categorical variables are associated.

Example

Is customer satisfaction related to membership type?

These are the two most important forms to learn.

2. Chi-Square Formula

The basic statistic is

Where

(O) = Observed frequency

(E) = Expected frequency

The larger the difference between observed and expected counts, the larger the (\chi^2) statistic tends to be.

3. Goodness-of-Fit Test

Suppose we roll a die 60 times.

If the die is fair, we expect each number to occur

Suppose we observe

We want to determine whether these differences are consistent with a fair die.

genui{"learning_viz":{"type_id":"CHI_SQUARE_GOODNESS_OF_FIT"}}

4. Hypotheses for Goodness-of-Fit

Null hypothesis

For our die

The die is fair.

Alternative hypothesis

For our die

The die is not fair.

5. Calculate Chi-Square

For the first category

Contribution

[ \frac{(8-10)^2}{10}

\frac{4}{10}

0.4 ]

For all categories

The total is

6. Degrees of Freedom

For a goodness-of-fit test

where (k) is the number of categories.

For our six-sided die

Degrees of freedom are needed to determine the appropriate chi-square distribution.

7. p-value

Once we calculate

and

we calculate the p-value.

Using the chi-square distribution, the p-value is approximately

Suppose

Then

Therefore

Conclusion

There isn't sufficient statistical evidence to conclude that the observed die results differ from the expected distribution.

This does not prove that the die is fair. It means the observed differences are not statistically unusual enough to reject the fairness assumption.

8. Chi-Square Test of Independence

The second major type tests whether two categorical variables are associated.

Suppose a company wants to know whether

Membership type is associated with product purchase.

We collect

Both variables are categorical

Membership → Basic/Premium

Purchase → Yes/No

9. Hypotheses for Independence

Null hypothesis

There is no association between membership type and purchase.

Alternative hypothesis

There is an association between membership type and purchase.

10. Expected Frequencies

For each cell

For Basic + Purchased

For Basic + Did Not Purchase

So the expected table becomes

11. Calculate Chi-Square

We compare observed and expected values

Then

The result is

12. Degrees of Freedom for Independence

For a contingency table

where

(r) = number of rows

(c) = number of columns

For our 2 × 2 table

The p-value for

is approximately

Since

we reject (H_0).

Conclusion

There is statistically significant evidence of an association between membership type and purchase behavior.

13. Important: Association ≠ Causation

Even if a chi-square test finds

we should not conclude

• "Premium membership causes customers to purchase."
• The test establishes evidence of an association, not necessarily causation.
• There could be other factors influencing both membership and purchasing behavior.

14. Chi-Square in Python

SciPy provides chi2_contingency().

import numpy as np
from scipy.stats import chi2_contingency
observed = np.array([
    [30, 70],

])

chi2, p, dof, expected = chi2_contingency(observed)

print("Chi-Square:", chi2)
print("p-value:", p)
print("Degrees of Freedom:", dof)
print("Expected Frequencies:")
print(expected)

Output will be approximately

• Chi-Square: 8.3333
• p-value: 0.0039
• Degrees of Freedom: 1

Expected Frequencies

15. Goodness-of-Fit in Python

For the die example

from scipy.stats import chisquare
observed = [8, 12, 9, 11, 7, 13]
expected = [10, 10, 10, 10, 10, 10]
result = chisquare(
    f_obs=observed,
    f_exp=expected
)

print("Chi-Square:", result.statistic)
print("p-value:", result.pvalue)

The result will be approximately

Chi-Square: 2.0

p-value: 0.849

16. Chi-Square Assumptions

The chi-square test has important assumptions.

1. Categorical/count data

The test is primarily designed for frequency counts, not continuous measurements.

2. Independent observations

Observations should generally be independent.

3. Expected frequencies should be sufficiently large

A common rule of thumb is that expected cell counts should generally be at least around 5, although exact requirements depend on the test and situation.

If expected counts are very small, alternative methods may be appropriate.

4. Random or appropriately designed sampling

The data collection method should support the inference being made.

17. Chi-Square vs t-Test vs ANOVA

A useful decision guide

What type of data do I have?

↓

Categorical counts?

↓ ↓

Yes No

↓

Chi-Square

Numerical outcome?

↓

• Compare 2 means → t-test
• Compare 3+ means → ANOVA
• Two numerical variables → Correlation

18. Chi-Square Statistic Interpretation

The chi-square statistic is always

A larger value generally means the observed frequencies are farther from the expected frequencies.

But you cannot interpret the magnitude alone.

You need

• Degrees of freedom
• p-value
• Significance level
• Context

19. Chi-Square and p-value

The decision process is the same as other hypothesis tests.

If

then

If

then

For example

and

Then

so we reject (H_0).

20. Real-World Applications

Chi-square tests are widely used in

• Marketing
• Is customer preference associated with demographic group?
• E-commerce
• Is payment method associated with customer segment?
• Healthcare
• Is treatment outcome associated with treatment group?
• Banking
• Is loan approval associated with customer category?
• Manufacturing
• Does defect type differ from expected proportions?
• Data Science
• Is a categorical feature associated with the target variable?

21. Example: Feature Selection

Suppose you are building a classification model

Target: Churn

and have a categorical feature

Contract Type

You can use a chi-square test to examine whether

and

are statistically associated.

A significant result suggests that the feature contains information related to the target.

However, statistical significance alone does not guarantee that the feature will improve a machine-learning model.

22. Chi-Square vs Covariance/Correlation

This distinction is important.

Covariance

Used primarily with

Pearson Correlation

Used primarily with

Chi-Square

Used primarily with

For example

23. Key Formulas

Chi-square statistic

Expected frequency

Goodness-of-fit degrees of freedom

Independence degrees of freedom

24. Quick Revision

Easy way to remember

Chi-Square = Categorical Data + Counts + Observed vs Expected

The core formula is

And the key decision rule remains

Next in your syllabus is 5.16 ANOVA (Analysis of Variance), which is used primarily to determine whether the means of three or more groups differ significantly.

5.16ANOVA — Analysis of Variance

ANOVA stands for Analysis of Variance.

It is a statistical method used primarily to determine whether the means of three or more groups are significantly different.

In simple terms

ANOVA tells us whether the differences between group means are large enough that they are unlikely to be explained by random variation alone.

1. Why Do We Need ANOVA?

Suppose a company wants to compare the performance of three marketing strategies.

We could calculate the mean of each group

Clearly, the means look different.

But the statistical question is

Are these differences large enough to conclude that the underlying population means are different?

That's what ANOVA helps us determine.

genui{"learning_viz":{"type_id":"ANOVA_DECOMPOSITION"}}

2. ANOVA Hypotheses

Suppose we have three groups

Null hypothesis

All population means are equal.

Alternative hypothesis

This distinction is extremely important.

ANOVA does not initially tell us which specific groups differ.

3. Why Is It Called "Analysis of Variance"?

This sounds confusing because we're comparing means.

ANOVA works by comparing two types of variation

• Between-group variation
• How far the group means are from the overall mean.
• Within-group variation

How much individual observations vary within each group.

The basic idea is

If between-group variation is much larger than within-group variation, there is evidence that the group means differ.

4. Example

Consider three groups

• Group A: 10, 11, 9, 10
• Group B: 20, 21, 19, 20
• Group C: 30, 31, 29, 30
• The groups are internally very consistent.

But their means are very different

So

Within-group variation → Small

Between-group variation → Large

Therefore, the F-statistic will be large and the p-value will likely be small.

5. The F-Statistic

ANOVA uses the F-statistic

Where

• (MS_{between}) = mean square between groups
• (MS_{within}) = mean square within groups
• Interpretation

If

the between-group variation is similar to the within-group variation.

If

the between-group variation is much larger.

That provides stronger evidence that the group means aren't all equal.

6. ANOVA Decomposition

Total variation in the data can be separated into

[ \boxed{ SS_{Total}

SS_{Between} + SS_{Within} } ]

Where

• (SS_{Total}) = total sum of squares
• (SS_{Between}) = between-group sum of squares
• (SS_{Within}) = within-group sum of squares

This is the fundamental idea behind ANOVA.

7. Between-Group Variation

Between-group variation measures how far each group mean is from the grand mean.

The grand mean is the mean of all observations combined.

Suppose

• Group A mean = 10
• Group B mean = 20
• Group C mean = 30

The grand mean is

Group A is 10 below the grand mean.

Group C is 10 above it.

Therefore, there is substantial between-group variation.

8. Within-Group Variation

Within-group variation measures how much individual observations vary around their own group mean.

Example

Group A

9, 10, 10, 11

These observations are very close to their group mean.

Therefore

is relatively small.

9. ANOVA Table

A typical one-way ANOVA table looks like

Where

(k) = number of groups

(N) = total number of observations

10. Degrees of Freedom

For one-way ANOVA

Between groups

Within groups

Total

11. Mean Square

Mean square is calculated as

Therefore

[ MS_{between}

\frac{SS_{between}} {df_{between}} ]

and

[ MS_{within}

\frac{SS_{within}} {df_{within}} ]

Then

12. Complete Example

Suppose three groups have test scores

• Group A = [10, 12, 11]
• Group B = [20, 21, 19]
• Group C = [30, 29, 31]

Group means

Grand mean

The group means are far apart, while the observations within each group are close together.

Therefore

which produces a large

and consequently a very small p-value.

We would likely reject

and conclude that there is evidence that the population means are not all equal.

13. ANOVA Decision Using p-value

Just like hypothesis testing

and

Usually

14. Example of ANOVA Result

Suppose Python gives

F-statistic = 24.65

p-value = 0.00002

Using

we have

Therefore

Conclusion

There is statistically significant evidence that the group means are not all equal.

Notice the wording

"Not all means are equal."

We cannot yet say exactly which groups differ.

15. What ANOVA Does NOT Tell Us

Suppose we compare

• Group A
• Group B
• Group C

and ANOVA gives

We know

But we don't know whether

• A differs from B
• B differs from C
• A differs from C
• All three differ

To determine this, we need post-hoc tests.

16. Post-Hoc Tests

A common post-hoc method is

Tukey's test performs pairwise comparisons while controlling the overall error rate appropriately.

Example

ANOVA

↓

Significant?

↓

A vs C

B vs C

You might discover

Then A and C and B and C show evidence of differences, while A and B do not at the chosen threshold.

17. ANOVA in Python

Using SciPy

from scipy.stats import f_oneway
group_a = [10, 12, 11, 9, 10]
group_b = [20, 21, 19, 20, 22]
group_c = [30, 29, 31, 30, 32]
result = f_oneway(
    group_a,
    group_b,
    group_c
)

print("F-statistic:", result.statistic)
print("p-value:", result.pvalue)

Then

alpha = 0.05
if result.pvalue < alpha:
print("Reject H0")
else:
print("Fail to reject H0")

18. Tukey HSD in Python

After finding a significant ANOVA result, we can perform post-hoc comparisons.

Using Statsmodels

from statsmodels.stats.multicomp import pairwise_tukeyhsd
import numpy as np
scores = np.array([
    10, 12, 11, 9, 10,
    20, 21, 19, 20, 22,
    30, 29, 31, 30, 32
])

groups = (
    ["A"] * 5 +
    ["B"] * 5 +
    ["C"] * 5
)

result = pairwise_tukeyhsd(
    scores,
    groups,
    alpha=0.05
)

print(result)

This tells us which pairs of groups differ significantly.

19. One-Way ANOVA

The example we've discussed is called

It examines one categorical independent variable.

Example

Does teaching method affect exam scores?

Teaching method

• Method A
• Method B
• Method C

Outcome

Exam score

20. Two-Way ANOVA

Two-way ANOVA examines two categorical factors.

Example

Does exam performance depend on both teaching method and study environment?

Factors

Factor 1

Teaching method

• A
• B
• C
• Factor 2

Environment

Online

Classroom

Outcome

Exam score

Two-way ANOVA can examine

• Effect of teaching method
• Effect of environment
• Interaction between them

21. Interaction Effect

An interaction occurs when the effect of one factor depends on the level of another factor.

Example

• Teaching Method A
• → works very well online
• → works poorly in classroom
• Teaching Method B
• → works poorly online
• → works very well in classroom

The effect of teaching method depends on the environment.

That's an interaction effect.

22. ANOVA Assumptions

One-way ANOVA typically assumes

1. Independence

Observations should be independent.

2. Approximately normal residuals

The residuals should be reasonably consistent with normality, especially for small samples.

3. Homogeneity of variance

The groups should have reasonably similar variances.

This is often called

or homoscedasticity.

23. What If Variances Are Not Equal?

Standard one-way ANOVA may not be appropriate when group variances differ substantially.

An alternative is

which is more robust to unequal variances.

This is an important practical point

Don't automatically apply standard ANOVA without checking whether its assumptions are reasonable.

24. ANOVA vs t-Test

A common question is

Why not just perform multiple t-tests?

Suppose we have

• A
• B
• C

We could compare

But performing many separate tests increases the chance of a Type I error somewhere among the comparisons.

ANOVA provides an overall test first.

Then, if appropriate, post-hoc tests can identify which groups differ.

25. ANOVA vs Chi-Square

26. Real-World Applications

• Business
• Compare sales across different regions.
• Marketing

Compare conversion rates or numerical performance metrics across multiple campaigns, when the assumptions and outcome structure are appropriate.

• Healthcare
• Compare a numerical outcome across treatment groups.
• Education
• Compare test scores across teaching methods.
• Manufacturing
• Compare production measurements across machines.
• Data Science

Determine whether a categorical factor is associated with differences in a numerical target.

27. Example: Employee Salaries

Suppose we want to compare average salaries across

• IT
• Finance
• HR
• Sales

We could formulate

[ H_0: \mu_{IT}

\mu_{Finance}

\mu_{HR}

\mu_{Sales} ]

Alternative

Run one-way ANOVA.

If

we conclude

There is statistically significant evidence that average salaries are not equal across all departments.

Then use an appropriate post-hoc method to identify which departments differ.

28. ANOVA and Machine Learning

ANOVA can be useful during Exploratory Data Analysis and feature analysis.

For example

Feature: Customer Segment

↓

┌────────┼────────┐

↓ ↓ ↓

Basic Silver Gold

↓ ↓ ↓

Customer Spend

ANOVA can help determine whether average customer spending differs across segments.

However

A statistically significant ANOVA result does not automatically mean the feature will improve a machine-learning model.

Model validation is still required.

29. Key Formulas

Total variation

Between-group degrees of freedom

Within-group degrees of freedom

Mean square

F-statistic

30. Easy Way to Understand ANOVA

Think of it this way

Total Variation

│

┌────────────┴────────────┐

↓ ↓

Between Groups Within Groups

│ │

Group means differ Individuals differ

│ │

└────────────┬────────────┘

↓

The key question is

Are the groups separated from each other more than we would expect based on the variation inside the groups?

31. Quick Revision

The most important formula

And the most important interpretation

Important: ANOVA tells you that a difference exists somewhere, not which groups differ. For that, use an appropriate post-hoc comparison such as Tukey HSD.

Linear Regression is a statistical and machine-learning technique used to model the relationship between a dependent variable and one or more independent variables.

In simple terms

Linear regression finds the best-fitting line through data and uses that line to understand or predict a numerical outcome.

Examples

• Predict house price from area
• Predict salary from years of experience
• Predict sales from advertising spend
• Predict electricity consumption from temperature
• Predict customer spending from age/income

1. Simple Linear Regression

When we have one independent variable (X) and one dependent variable (Y), it is called Simple Linear Regression.

The basic equation is

Where

• (x) = independent variable
• (y) = actual dependent variable
• (\hat y) = predicted value
• (b_0) = intercept
• (b_1) = slope/coefficient

genui{"learning_viz":{"type_id":"LEAST_SQUARE_REGRESSION"}}

2. Example

Suppose we want to predict salary from years of experience.

A regression model might learn

For someone with 6 years of experience

So the predicted salary is

3. Scatter Plot and Regression Line

The data can be visualized as points

Salary

|

55| *

50| *

45| *

40| *

35| *

30| *

|________________________

Experience

Linear regression finds a line that best represents the relationship.

The line is often called the

Line of best fit

4. Slope

The slope (b_1) tells us how much the predicted value of (Y) changes when (X) increases by one unit.

Suppose

Then

Interpretation

For every additional year of experience, predicted salary increases by 5 units, on average, according to this model.

If salary is measured in ₹ lakhs

assuming the model is appropriate and the relationship is linear.

5. Intercept

The intercept (b_0) is the predicted value of (Y) when

For

the intercept is

So the model predicts

when

Important

The intercept is not always practically meaningful.

For example, if (x) represents years of work experience, predicting salary at exactly zero years may or may not make sense depending on the data and context.

6. Actual vs Predicted Values

Suppose

and the model predicts

But the actual value is

The difference is called the residual.

Therefore

7. Residuals

Residuals represent the prediction errors

For example

A good regression model generally has residuals that are relatively small and don't show problematic patterns.

8. How Does Regression Find the Best Line?

Linear regression commonly uses the Least Squares Method.

It chooses the line that minimizes the sum of squared residuals

This is called the

The model tries to find coefficients that make this quantity as small as possible.

9. Why Square the Errors?

Suppose the errors are

If we simply add them

It would look like there is no error.

Instead, square them

This prevents positive and negative errors from cancelling each other out.

10. Regression Equation

The basic equation is

For example

If

then

11. Multiple Linear Regression

If we have multiple independent variables, we use Multiple Linear Regression.

The equation becomes

For example, house price might depend on

• Area
• Number of bedrooms
• Age of house
• Distance from city center

The model could be

12. Interpreting Multiple Regression Coefficients

Suppose

[ Price= 20+ 0.05(Area)+ 5(Bedrooms)

0.3(Age) ]

The coefficient for Area is

Interpretation

Holding the other variables constant, a one-unit increase in area is associated with a 0.05-unit increase in predicted price.

The phrase

is particularly important in multiple regression.

13. Correlation vs Linear Regression

These concepts are related but different.

Correlation

Measures the strength and direction of linear association.

Regression

Builds an equation that can be used to

• Predict (Y)
• Estimate effects/associations
• Understand relationships

14. R-Squared

One of the most important regression metrics is

It tells us the proportion of variation in the dependent variable explained by the model, under the usual regression interpretation.

For example

means the model explains approximately

of the observed variation in the outcome in the sample/model context.

15. R-Squared Formula

A common formula is

Where

(SS_{Residual}) = unexplained variation

(SS_{Total}) = total variation

Another way to think about it

[ \boxed{ \text{Total variation}

\text{Explained variation} + \text{Unexplained variation} } ]

16. Example of R-Squared

Suppose

We can say

The model explains approximately 75% of the variation in the dependent variable.

But

does not mean

"The model is 75% accurate."

That's incorrect.

(R^2) and prediction accuracy are different concepts.

17. Adjusted R-Squared

In multiple regression, adding more variables can increase or leave (R^2) unchanged, even if those variables aren't genuinely useful.

Adjusted (R^2) accounts for the number of predictors.

Therefore

Adjusted (R^2) is often more useful than ordinary (R^2) when comparing multiple regression models with different numbers of predictors.

18. Regression Assumptions

Classical linear regression relies on several important assumptions.

1. Linearity

The relationship between predictors and the expected outcome should be appropriately modeled as linear.

2. Independence

Observations/errors should generally be independent.

3. Homoscedasticity

Residual variance should be reasonably constant across fitted values.

4. Normality of residuals

For many inferential procedures, residuals being approximately normally distributed can be important, especially with smaller samples.

5. No problematic multicollinearity

In multiple regression, predictors should not be excessively redundant.

19. Residual Analysis

After fitting a regression model, we should inspect the residuals.

A good residual plot might look approximately random

Residual

|

+ | * * *

| * *

0 |-------------------------

| * * *

- | * *

|________________________

Predicted

A problematic pattern might look like

Residual

|

+ | * *

| * *

0 |-----*-----*------------

| * *

- | *

|________________________

Predicted

A visible curve can indicate that a linear model may not adequately capture the relationship.

20. Outliers and Influential Points

Regression can be sensitive to extreme observations.

For example

Y

|

| *

| *

| *

| *

| *

|________________________ X

*

One extreme point can substantially change the fitted line.

Therefore, regression analysis should include

• Scatter plots
• Residual analysis
• Outlier investigation
• Influence diagnostics

21. Linear Regression in Python

Using scikit-learn

from sklearn.linear_model import LinearRegression
X = [[1], [2], [3], [4], [5]]
y = [30, 35, 40, 45, 50]
model = LinearRegression()
model.fit(X, y)
print("Intercept:", model.intercept_)
print("Slope:", model.coef_[0])

The model will learn approximately

22. Making Predictions

Once the model is trained

prediction = model.predict([[6]])
print(prediction)

The predicted value will be approximately

23. Complete Python Example

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score, mean_squared_error
X = np.array([[1], [2], [3], [4], [5]])
y = np.array([30, 35, 40, 45, 50])
model = LinearRegression()
model.fit(X, y)
predictions = model.predict(X)
print("Intercept:", model.intercept_)
print("Slope:", model.coef_[0])
print("R²:", r2_score(y, predictions))
print("MSE:", mean_squared_error(y, predictions))

24. MSE

Another important regression metric is Mean Squared Error (MSE).

Lower MSE generally means the predictions are closer to the observed values, although the metric should always be interpreted in context.

25. RMSE

Root Mean Squared Error (RMSE) is

The advantage is that RMSE is expressed in the same units as the target variable.

For example, if we're predicting house prices in lakhs, RMSE is also expressed in lakhs.

26. MAE

Mean Absolute Error (MAE) is

Example

Actual

100, 200, 300

Predicted

110, 190, 320

Errors

-10, +10, -20

Absolute errors

10, 10, 20

Therefore

27. Regression Metrics

28. Statistical Significance of Regression Coefficients

Regression isn't only about prediction.

We can also test whether a coefficient is statistically different from zero.

For example

versus

If the coefficient's p-value is small, we have evidence that the predictor has a non-zero linear association with the outcome, conditional on the other variables in the model.

Again

Statistical significance does not prove causation.

29. Regression vs ANOVA

These topics are closely connected.

ANOVA

Asks

Are the means of several groups different?

Linear Regression

Asks

How does a numerical outcome change with one or more predictors?

Interestingly, ANOVA can also be expressed within the linear model framework.

For example

Department → Salary

can be analyzed using ANOVA or an equivalent regression model with categorical indicators.

30. Real-World Example

Suppose you're predicting electricity consumption.

Variables

• Temperature
• House Size
• Number of Occupants
• Appliance Usage

Target

Monthly Electricity Consumption

A multiple linear regression model could be

The model can help

• Predict future consumption
• Understand relationships
• Identify important predictors
• Estimate expected changes

31. Key Formulas

Simple linear regression

Residual

Sum of squared errors

Mean squared error

Root mean squared error

Coefficient of determination

32. Quick Revision

Final Mental Model

Think of linear regression as

The three most important concepts to remember are

And remember

Correlation describes a relationship; regression builds a model for that relationship and can be used for prediction.

The next topic in your syllabus is 5.18 Statistical Tests using Python, where we can put the hypothesis-testing concepts from this module into practice using SciPy and Python, including t-tests, chi-square, ANOVA, correlation tests, and p-values.

Python provides powerful libraries for performing statistical tests without manually calculating complex formulas.

The most commonly used library is

scipy.stats

Other useful libraries include

• numpy
• pandas
• statsmodels

The general workflow is

1. Important Statistical Tests

Here are the major tests you should know for Data Science

2. Installing Required Libraries

If SciPy is not installed

pip install scipy

For Data Science work, you will commonly use

pip install numpy pandas scipy statsmodels matplotlib seaborn

Import them

import numpy as np
import pandas as pd
from scipy import stats

3. One-Sample t-Test

A one-sample t-test tests whether a sample mean differs from a specified population/reference mean.

Example

Suppose the claimed average score is

We collect

data = [52, 49, 51, 53, 50, 48, 52, 51, 49, 54]

Test

against

Python

from scipy.stats import ttest_1samp
data = [52, 49, 51, 53, 50, 48, 52, 51, 49, 54]
result = ttest_1samp(data, popmean=50)
print("t-statistic:", result.statistic)
print("p-value:", result.pvalue)

4. Interpreting the Result

Suppose the result gives

t-statistic: 1.95

p-value: 0.083

Using

we compare

Therefore

Conclusion

There is insufficient evidence at the 5% significance level to conclude that the population mean differs from 50.

5. Independent Two-Sample t-Test

Use this when comparing the means of two independent groups.

Example

group_a = [72, 75, 71, 69, 74]
group_b = [80, 82, 78, 81, 79]

Question

Is the average score different between Group A and Group B?

Hypotheses

Python

from scipy.stats import ttest_ind
result = ttest_ind(group_a, group_b)
print("t-statistic:", result.statistic)
print("p-value:", result.pvalue)

6. Welch's t-Test

In real-world data, the two groups may have different variances.

Welch's t-test is often a safer default than assuming equal variances

result = ttest_ind(
    group_a,
    group_b,
    equal_var=False
)

print(result)

This is especially useful when

• Group sizes differ
• Variances differ
• You don't have a strong reason to assume equal variances

7. Paired t-Test

Use a paired t-test when the observations are naturally paired.

Typical examples

• Before vs after training
• Before vs after treatment
• Same customer before vs after a change
• Same employee before vs after training

Example

before = [70, 72, 68, 75, 71]
after = [75, 76, 72, 78, 77]

Python

from scipy.stats import ttest_rel
result = ttest_rel(before, after)
print("t-statistic:", result.statistic)
print("p-value:", result.pvalue)

8. Choosing the Correct t-Test

• Easy memory trick
• One → one-sample
• Two separate groups → independent
• Same subjects twice → paired

9. Chi-Square Test

The Chi-Square test is commonly used for categorical data.

Suppose

Question

Is membership type associated with purchase behavior?

Python

import numpy as np
from scipy.stats import chi2_contingency
observed = np.array([
    [30, 70],

])

chi2, p, dof, expected = chi2_contingency(observed)

print("Chi-square:", chi2)
print("p-value:", p)
print("Degrees of freedom:", dof)
print("Expected:")
print(expected)

10. Interpreting Chi-Square

Suppose

Chi-square = 8.33

p-value = 0.0039

With

we have

Therefore

Conclusion

There is statistically significant evidence of an association between membership type and purchase behavior.

Remember

11. Chi-Square Goodness-of-Fit

Suppose we observe die rolls

observed = [8, 12, 9, 11, 7, 13]

For a fair die

expected = [10, 10, 10, 10, 10, 10]

Run

from scipy.stats import chisquare
result = chisquare(
    f_obs=observed,
    f_exp=expected
)

print("Chi-square:", result.statistic)
print("p-value:", result.pvalue)

This tests whether the observed distribution is consistent with the expected distribution.

12. One-Way ANOVA

ANOVA is used to compare three or more group means.

Suppose

group_a = [10, 12, 11, 9, 10]
group_b = [20, 21, 19, 20, 22]
group_c = [30, 29, 31, 30, 32]

We want to test

Python

from scipy.stats import f_oneway
result = f_oneway(
    group_a,
    group_b,
    group_c
)

print("F-statistic:", result.statistic)
print("p-value:", result.pvalue)

13. ANOVA Interpretation

Suppose

F-statistic = 125.4

p-value = 0.000001

Since

we reject (H_0).

Conclusion

• There is statistically significant evidence that the population means are not all equal.
• But ANOVA doesn't tell us which groups differ.
• For that, use a post-hoc test.

14. Tukey HSD

After a significant ANOVA, we can use Tukey's Honestly Significant Difference test.

from statsmodels.stats.multicomp import pairwise_tukeyhsd
import numpy as np
scores = np.array([
    10, 12, 11, 9, 10,
    20, 21, 19, 20, 22,
    30, 29, 31, 30, 32
])

groups = (
    ["A"] * 5 +
    ["B"] * 5 +
    ["C"] * 5
)

result = pairwise_tukeyhsd(
    scores,
    groups,
    alpha=0.05
)

print(result)

This compares

• A vs B
• A vs C
• B vs C

while accounting for multiple comparisons.

15. Pearson Correlation Test

Pearson correlation measures the strength of a linear relationship between two numerical variables.

Suppose

study_hours = [1, 2, 3, 4, 5]
scores = [45, 50, 60, 70, 80]

Run

from scipy.stats import pearsonr
result = pearsonr(
    study_hours,
    scores
)

print("Correlation:", result.statistic)
print("p-value:", result.pvalue)

16. Interpreting Pearson Correlation

Suppose

Correlation = 0.95

p-value = 0.01

Then

indicates a strong positive linear association.

And

provides evidence against the null hypothesis of zero population correlation.

But remember

17. Spearman Correlation

Spearman correlation works with ranks and is useful for monotonic relationships.

from scipy.stats import spearmanr
x = [1, 2, 3, 4, 5]
y = [10, 20, 25, 40, 100]
result = spearmanr(x, y)
print("Spearman correlation:", result.statistic)
print("p-value:", result.pvalue)

Use it when

• Data is ordinal
• Relationship is monotonic rather than strictly linear
• Pearson's assumptions aren't suitable

18. Shapiro-Wilk Normality Test

Many statistical tests make assumptions about distributions.

The Shapiro-Wilk test can be used to test for evidence against normality.

Hypotheses

Python

from scipy.stats import shapiro
data = [12, 14, 15, 13, 16, 14, 15, 17, 13, 15]
result = shapiro(data)
print("Statistic:", result.statistic)
print("p-value:", result.pvalue)

If

• we reject the normality assumption under this test.
• Important practical point
• Don't rely solely on a normality test.

Also examine

• Histogram
• Q-Q plot
• Sample size
• Domain knowledge

With very large samples, tiny deviations from normality can produce very small p-values.

19. Mann-Whitney U Test

The Mann-Whitney U test is a non-parametric alternative often used for comparing two independent groups when the assumptions of a t-test are questionable.

from scipy.stats import mannwhitneyu
group_a = [10, 12, 11, 9, 10]
group_b = [20, 21, 19, 20, 22]
result = mannwhitneyu(
    group_a,
    group_b,
    alternative="two-sided"
)

print("U statistic:", result.statistic)
print("p-value:", result.pvalue)

It works with the ordering/ranks of observations rather than relying on the same assumptions as the t-test.

20. Wilcoxon Signed-Rank Test

This is a non-parametric alternative for paired data.

Example

before = [70, 72, 68, 75, 71]
after = [75, 76, 72, 78, 77]

Python

from scipy.stats import wilcoxon
result = wilcoxon(before, after)
print("Statistic:", result.statistic)
print("p-value:", result.pvalue)

21. Test Selection Guide

A useful decision tree

What type of question?

│

├── One numerical sample vs reference

│ ↓

│ One-sample t-test

│

├── Two independent numerical groups

│ ↓

│ Independent t-test

│

├── Before vs after

│ ↓

│ Paired t-test

│

├── 3+ numerical groups

│ ↓

│ ANOVA

│

├── Two categorical variables

│ ↓

│ Chi-Square

│

├── Two numerical variables

│ ↓

│ Pearson / Spearman

│

22. Complete Statistical Testing Example

Suppose an e-commerce company wants to determine whether a new website design improves customer spending.

Step 1 — Data

old_site = [100, 110, 95, 105, 98, 102, 108]
new_site = [115, 120, 110, 118, 122, 116, 119]

These are two independent groups.

Step 2 — Hypotheses

Step 3 — Test

Use Welch's t-test

from scipy.stats import ttest_ind
result = ttest_ind(
    old_site,
    new_site,
    equal_var=False
)

print("t-statistic:", result.statistic)
print("p-value:", result.pvalue)

Step 4 — Decision

alpha = 0.05
if result.pvalue < alpha:
print("Reject H0")
else:
print("Fail to reject H0")

23. Don't Just Look at the p-value

A good statistical analysis should consider more than

p = 0.03

Also examine

1. Effect size

How large is the difference?

2. Confidence interval

What range of effects is plausible?

3. Sample size

How much data was collected?

4. Assumptions

Is the selected test appropriate?

5. Practical significance

Does the result matter in the real world?

24. Reporting Statistical Results

Instead of saying

"p = 0.02, so the new system is better."

A better report is

"The new system had a higher average outcome than the old system. The difference was statistically significant at the 5% level ((p=0.02)). The estimated effect size and confidence interval should also be considered when evaluating practical significance."

This is much better statistical communication.

25. Important Python Pattern

Most SciPy statistical tests return something similar to

result.statistic

result.pvalue

For example

result = ttest_1samp(data, 50)
print(result.statistic)
print(result.pvalue)

You can use

if result.pvalue < 0.05:
print("Reject H0")
else:
print("Fail to reject H0")

26. Summary Table

27. Final Statistical Testing Workflow

For Data Science, remember this process

↓

┌──────────┴──────────┐

↓ ↓

Statistic p-value

↓

Compare with α

↓

┌─────────────┴────────────┐

↓ ↓

p < α p ≥ α

↓ ↓

Reject H₀ Fail to reject H₀

↓ ↓

Interpret result Interpret result

Key Takeaways

The most important thing is not memorizing Python functions. Learn to answer

• What type of data do I have?
• What question am I asking?
• What are (H_0) and (H_a)?
• Which statistical test matches the question and assumptions?
• What is the p-value?
• What is the effect size and confidence interval?
• Is the result practically meaningful?

The core mapping to remember

And the universal hypothesis-testing rule

The next topic is 5.19 Exploratory Data Analysis (EDA), where these statistical concepts are combined with Python, Pandas, visualization, distributions, missing-value analysis, outlier detection, and correlation analysis to understand a real dataset before building a machine-learning model.

Exploratory Data Analysis (EDA) is the process of examining, cleaning, summarizing, and visualizing a dataset to understand its structure, patterns, relationships, anomalies, and data-quality problems before performing statistical analysis or building a machine-learning model.

In simple terms

EDA is how we get to know our data before trusting or modeling it.

A typical EDA workflow is

1. Why Is EDA Important?

Imagine you're given a customer dataset with

• Customer_ID
• Age
• Income
• Gender
• City
• Purchase_Amount

Before building a model, you need to know

• How many customers are there?
• Are there missing values?
• Are there duplicate records?
• What is the average income?
• What is the distribution of age?
• Are there unusual values?
• Which variables are correlated?
• Are categorical values consistent?
• Is the target variable balanced?
• Are there data-entry errors?

EDA helps answer these questions.

2. Main Goals of EDA

EDA generally aims to

1. Understand the data

Learn what each column represents.

2. Identify data-quality issues

Find

• Missing values
• Duplicates
• Invalid values
• Incorrect data types

3. Understand distributions

Study

• Mean
• Median
• Standard deviation
• Skewness
• Percentiles

4. Find relationships

Understand how variables relate to each other.

5. Detect outliers

Identify unusually large or small observations.

6. Generate hypotheses

EDA can reveal patterns that deserve further statistical investigation.

7. Prepare data for modeling

EDA helps determine what preprocessing may be necessary.

3. Python Libraries for EDA

The most commonly used tools are

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

• NumPy
• Numerical operations.
• Pandas
• Data manipulation and analysis.
• Matplotlib
• Visualization.
• Seaborn
• Statistical visualization.

4. Load a Dataset

Suppose we have

customers.csv

Load it using Pandas

import pandas as pd
df = pd.read_csv("customers.csv")

Check the first rows

print(df.head())

5. head()

df.head()

Shows the first five rows by default.

You can specify the number

df.head(10)

This is useful for quickly understanding the dataset.

6. tail()

df.tail()

Shows the last five rows.

Useful for checking

• Dataset ending
• Unexpected records
• File loading issues

7. Dataset Shape

Use

df.shape

Suppose the result is

(10000, 8)

This means

8. Column Names

Use

df.columns

Example

Index([

'customer_id',

'age',

• 'income',
• 'gender',
• 'city',
• 'purchase_amount',
• 'membership'

])

This tells us which variables are available.

9. Data Types

Use

df.dtypes

Example

age int64

income float64

gender object

purchase_amount float64

This is important because statistical operations depend on the correct data type.

10. info()

One of the most useful initial EDA commands is

df.info()

It gives information about

• Number of rows
• Columns
• Non-null values
• Data types
• Memory usage

Example

RangeIndex: 10000 entries

Data columns: 7 columns

age 9800 non-null int64

income 9500 non-null float64

gender 10000 non-null object

Immediately we can see that some columns have missing values.

11. Descriptive Statistics

Use

df.describe()

For numerical columns, it provides

• Count
• Mean
• Standard deviation
• Minimum
• 25th percentile
• Median
• 75th percentile
• Maximum

Example

age income

count 9500.0 9500.0

mean 35.4 65000.0

std 9.8 18000.0

min 18.0 20000.0

25% 28.0 52000.0

50% 34.0 64000.0

75% 42.0 77000.0

max 95.0 250000.0

12. Interpreting describe()

Suppose

and

The distribution may be approximately symmetric.

But suppose

and

There may be strong right skewness caused by high values.

For example, a few very wealthy customers can increase average income significantly.

13. Categorical Data

describe() by default focuses on numerical columns.

For categorical columns

df.describe(include="object")

This can show

• Count
• Number of unique values
• Most frequent value
• Frequency of most frequent value

Example

gender

count 10000

unique 3

top Male

freq 5200

14. Unique Values

To inspect a categorical column

df["gender"].unique()

Example

Number of unique values

df["gender"].nunique()

15. Value Counts

Use

df["gender"].value_counts()

Example

Male 5200

Female 4500

Other 300

This is useful for understanding categorical distributions.

16. Missing Values

Missing data is one of the most important EDA checks.

Use

df.isnull().sum()

Example

age 200

income 500

gender 0

purchase_amount 75

This tells us how many missing values exist in each column.

17. Missing Value Percentage

A better approach is often to calculate percentages

missing_pct = (
    df.isnull().mean() * 100
)

print(missing_pct)

Example

age 2.0

income 5.0

gender 0.0

purchase_amount 0.75

Now we can assess the severity of the problem.

18. Handling Missing Values

There are several approaches.

Remove rows

df.dropna()

Remove a specific column

df.drop(columns=["income"])

• Fill numerical values with median
• df["income"] = df["income"].fillna(
• df["income"].median()

)

• Fill categorical values with mode
• df["gender"] = df["gender"].fillna(
• df["gender"].mode()[0]

)

The correct approach depends on

• Why values are missing
• How much data is missing
• The analysis/model
• Whether missingness itself carries information

19. Duplicate Records

Check duplicates

df.duplicated().sum()

Remove duplicates

df = df.drop_duplicates()

But don't blindly remove duplicates.

Sometimes repeated records are legitimate.

20. Univariate Analysis

Univariate analysis means analyzing one variable at a time.

Examples

• Age
• Income
• Gender
• Sales

Questions include

• What is the distribution?
• What is the center?
• How much variation exists?
• Are there outliers?

21. Histogram

A histogram shows the distribution of a numerical variable.

import matplotlib.pyplot as plt
plt.hist(df["age"], bins=20)
plt.xlabel("Age")
plt.ylabel("Frequency")
plt.title("Age Distribution")
plt.show()

A histogram helps identify

• Center
• Spread
• Skewness
• Multiple peaks
• Potential outliers

22. Box Plot

A box plot is excellent for detecting outliers.

import seaborn as sns
import matplotlib.pyplot as plt

sns.boxplot(x=df["income"])

plt.show()

Conceptually

┌───────────────┐

──────│ │──────

└───────────────┘

↑ ↑

Q1 Q3

Points far beyond the whiskers may be potential outliers.

23. Box Plot Components

A box plot generally shows

• Minimum/non-outlier lower value
• Q1
• Median
• Q3
• Maximum/non-outlier upper value
• Potential outliers

The Interquartile Range is

A common outlier rule is

Values beyond these boundaries are often flagged as potential outliers.

24. Bar Chart

For categorical data

sns.countplot(

data=df,
x="gender"

)

plt.show()

This shows the number of observations in each category.

25. Pie Charts

Pie charts can show proportions, but for many categories, bar charts are usually easier to compare.

For example

df["gender"].value_counts().plot(

kind="bar"

)

plt.show()

26. Bivariate Analysis

Bivariate analysis examines the relationship between two variables.

Examples

• Age vs income
• Advertising vs sales
• Gender vs purchase
• Experience vs salary

27. Scatter Plot

For two numerical variables

sns.scatterplot(

data=df,
x="age",
y="income"

)

plt.show()

A scatter plot can reveal

• Positive relationships
• Negative relationships
• Non-linear relationships
• Clusters
• Outliers

28. Correlation Matrix

Calculate

corr = df.corr(numeric_only=True)
print(corr)

Example

age income purchase

age 1.00 0.70 0.40

income 0.70 1.00 0.65

purchase 0.40 0.65 1.00

This shows relationships between numerical variables.

29. Correlation Heatmap

sns.heatmap(

df.corr(numeric_only=True),
annot=True

)

plt.show()

This makes strong relationships easier to identify visually.

30. Important Warning About Correlation

Suppose

This indicates strong linear association.

It does not prove

causally.

Remember

31. Multivariate Analysis

Multivariate analysis examines multiple variables simultaneously.

For example

• Age
• Income
• Education
• Experience
• Purchase Amount
• We might investigate how several variables relate to purchase amount.
• A common visualization is a pair plot.
• sns.pairplot(
• df[

[

"age",

• "income",
• "experience",
• "purchase_amount"

]

]

)

plt.show()

32. Pair Plot

A pair plot gives multiple scatter plots and distributions.

It can help identify

• Correlations
• Clusters
• Outliers
• Non-linear patterns
• Distribution shapes

It can become difficult to interpret with many columns, so it is best used selectively.

33. Detecting Outliers Using IQR

We can programmatically identify potential outliers.

Q1 = df["income"].quantile(0.25)
Q3 = df["income"].quantile(0.75)
IQR = Q3 - Q1
lower = Q1 - 1.5 * IQR
upper = Q3 + 1.5 * IQR
outliers = df[
    (df["income"] < lower) |
    (df["income"] > upper)
]

print(outliers)

34. Z-Score Outlier Detection

Another approach is the z-score.

For example

from scipy.stats import zscore

df["income_z"] = zscore(df["income"])

outliers = df[
    df["income_z"].abs() > 3
]

A common rule of thumb is

may indicate an unusual observation.

However, this rule is not universally appropriate.

35. Skewness

• EDA should also examine the shape of distributions.
• Positive skew
• Long tail toward the right.

/\

/ \

/ \

____/ \___________

Typically

Negative skew

Long tail toward the left.

/\

/ \

_________/ \____

Typically

36. Checking Skewness in Python

df["income"].skew()

A rough interpretation

• Near 0 → approximately symmetric
• Positive → right-skewed
• Negative → left-skewed

These are descriptive guidelines, not strict cutoffs.

37. Data Type Problems

Sometimes data appears numeric but is stored as text.

Example

• "1000"
• "2000"
• "3000"

Check

df.dtypes

Convert

df["salary"] = pd.to_numeric(

df["salary"],

errors="coerce"

)

Invalid values become NaN, which can then be investigated.

38. Inconsistent Categories

Suppose a city column contains

• Hyderabad
• hyderabad
• HYDERABAD
• Hyd
• These may represent the same city.

Check

df["city"].value_counts()

Standardize where appropriate

df["city"] = (

• df["city"]
• .str.strip()
• .str.lower()

)

Now

hyderabad

will be represented consistently.

39. Date-Time EDA

Dates often require special handling.

Convert

df["order_date"] = pd.to_datetime(

df["order_date"]

)

Then extract

• df["year"] = df["order_date"].dt.year
• df["month"] = df["order_date"].dt.month
• df["day"] = df["order_date"].dt.day
• df["weekday"] = df["order_date"].dt.day_name()

Now you can analyze

• Monthly sales
• Weekly trends
• Seasonality
• Day-of-week behavior

40. Time-Series EDA

Suppose we have

Date

Sales

We can plot

plt.plot(
    df["date"],
    df["sales"]
)

plt.xlabel("Date")
plt.ylabel("Sales")
plt.title("Sales Trend")
plt.show()

This can reveal

• Trends
• Seasonality
• Sudden changes
• Missing periods
• Anomalies

41. Target Variable Analysis

In machine learning, one of the most important EDA steps is analyzing the target variable.

For regression

df["price"].describe()

For classification

df["churn"].value_counts()

If

No 9500

Yes 500

then

This can have important implications for model evaluation and training.

42. EDA for Classification

Suppose

Target = Customer Churn

We might examine

sns.countplot(

data=df,
x="churn"

)

plt.show()

Then investigate relationships

sns.boxplot(

data=df,
x="churn",
y="monthly_spend"

)

plt.show()

This helps determine whether spending patterns differ between churned and non-churned customers.

43. EDA for Regression

Suppose the target is

House Price

We might examine

sns.scatterplot(

data=df,
x="area",
y="price"

)

plt.show()

Then

df.corr(numeric_only=True)

This can help identify potentially useful numerical predictors.

44. EDA and Feature Engineering

EDA often suggests useful new features.

Suppose we have

date_of_birth

Instead of directly using the date, we might create

df["age"] = (

pd.Timestamp.today().year -
df["date_of_birth"].dt.year

)

Or from transaction data

order_date

we could derive

• Month
• Quarter
• Day of week
• Weekend indicator

This process is called

45. Complete EDA Example

Let's create a small dataset

import pandas as pd
df = pd.DataFrame({
    "age": [22, 25, 28, 35, 40, 45, 50, 55],
    "income": [
        30000, 35000, 42000, 50000,
        60000, 70000, 85000, 120000
    ],
    "purchase": [
        200, 300, 350, 500,
        600, 700, 800, 1200
    ]
})

Step 1 — Structure

print(df.shape)
print(df.info())

Step 2 — Statistics

print(df.describe())

Step 3 — Missing values

print(df.isnull().sum())

Step 4 — Correlation

print(df.corr())

Step 5 — Visualization

import seaborn as sns
import matplotlib.pyplot as plt

sns.pairplot(df)

plt.show()

Now we have a basic understanding of

• Distribution
• Missing values
• Relationships
• Correlations
• Potential outliers

46. A Practical EDA Checklist

When you receive a new dataset, follow this checklist.

Step 1 — Load

df = pd.read_csv("data.csv")

Step 2 — Understand structure

df.head()
df.shape
df.columns
df.info()

Step 3 — Statistics

df.describe()

Step 4 — Missing values

df.isnull().sum()

Step 5 — Duplicates

df.duplicated().sum()

Step 6 — Categorical variables

df.select_dtypes(
    include="object"
).nunique()

Step 7 — Numerical distributions

df.hist(figsize=(12, 8))
plt.show()

Step 8 — Outliers

Use

• Box plots
• IQR
• Z-scores
• Domain knowledge
• Step 9 — Correlations

df.corr(numeric_only=True)

Step 10 — Relationships

Use

• Scatter plots
• Box plots
• Bar charts
• Heatmaps
• Step 11 — Target analysis

Understand the target distribution and class balance.

Step 12 — Document findings

Write down

• Important patterns
• Data-quality issues
• Outliers
• Relationships
• Assumptions
• Possible feature engineering ideas

47. EDA vs Data Cleaning

These are related but not exactly the same.

EDA

Focuses on

Understanding what is in the data.

Data Cleaning

Focuses on

Correcting or handling data-quality problems.

They often happen iteratively

48. EDA vs Statistical Testing

EDA is primarily exploratory.

Statistical testing is used to formally evaluate hypotheses.

For example

• EDA
• Customer age and spending appear positively related.
• Statistical test

Pearson correlation

r = 0.72
p = 0.003

The statistical test provides formal evidence under its assumptions.

49. EDA vs Machine Learning

A common workflow is

EDA should generally be performed carefully to avoid data leakage.

For example, when building a predictive model, transformations that learn parameters from the data should be fitted using the training data and then applied to validation/test data appropriately.

50. Common EDA Mistakes

• Mistake 1: Jumping directly to ML

Don't train a model before understanding the dataset.

• Mistake 2: Ignoring missing values

Missing data can significantly affect analysis.

• Mistake 3: Automatically deleting outliers

An outlier could be

• Data error
• Genuine rare event
• Important business case
• Investigate before removing it.
• Mistake 4: Assuming correlation means causation

Always remember

• Mistake 5: Looking only at averages

Mean alone doesn't tell you the complete distribution.

Use

• Median
• Standard deviation
• Percentiles
• Histograms
• Box plots
• Mistake 6: Ignoring categorical variables

Categorical variables often contain valuable information.

51. EDA and Your Previous Topics

You have now covered many concepts that come together in EDA

EDA is where you begin applying these concepts to real datasets.

52. EDA Cheat Sheet

53. Final EDA Mental Model

Think of EDA as being a data detective

DATASET

↓

┌─────────────────────┐

│ What do I have? │

└──────────┬──────────┘

↓

┌─────────────────────┐

│ Is data complete? │

└──────────┬──────────┘

↓

┌─────────────────────┐

│ Is data correct? │

└──────────┬──────────┘

↓

┌─────────────────────┐

│ How is it distributed│

└──────────┬──────────┘

↓

┌─────────────────────┐

│ What relates to what│

└──────────┬──────────┘

↓

┌─────────────────────┐

│ Any outliers? │

└──────────┬──────────┘

↓

┌─────────────────────┐

│ What patterns exist?│

└──────────┬──────────┘

↓

Key Takeaway

EDA is not just creating charts. It is a systematic process of understanding data, identifying quality issues, discovering patterns, testing assumptions, and preparing the dataset for reliable statistical analysis or machine learning.

For a Data Scientist, a strong EDA process should answer

This completes 5.19 Exploratory Data Analysis (EDA). The final topic in your Module 5 syllabus is 5.20 Statistics Project, where all these concepts can be combined into one end-to-end real-world project.

End-to-End Customer Purchase Analysis Using Python

This project brings together the major concepts from Module 5 – Statistics

• Mean
• Median
• Mode
• Variance
• Standard Deviation
• Probability
• Normal Distribution
• Sampling
• Central Limit Theorem
• Hypothesis Testing
• p-value
• Confidence Intervals
• Correlation
• Covariance
• Chi-Square Test
• ANOVA
• Linear Regression
• Statistical Tests using Python
• Exploratory Data Analysis (EDA)

1. Project Objective

We will analyze an e-commerce customer dataset to answer questions such as

• What is the average customer spending?
• How widely does customer spending vary?
• Are customers from different membership groups spending differently?
• Is income related to purchase amount?
• Is membership type associated with purchasing behavior?
• Is there a statistically significant difference between customer groups?
• Can we predict purchase amount from customer characteristics?

The overall objective is

Use statistical analysis to understand customer behavior and build a simple model for predicting purchase amount.

2. Business Scenario

Imagine an e-commerce company called

ShopSmart

The company stores information about its customers.

Example data

Our objective is to discover patterns in this data.

3. Dataset Structure

Let's assume our CSV contains

• customer_id
• age
• gender
• income
• membership
• purchase_count
• purchase_amount
• satisfaction
• city
• Variable types

4. Project Structure

A professional project can be organized as

statistics_project/

│

├── data/

│ └── customers.csv

│

├── notebooks/

│ └── statistics_analysis.ipynb

│

├── src/

│ └── analysis.py

│

├── reports/

│ └── statistics_report.md

│

5. Required Python Libraries

Install

pip install numpy pandas scipy matplotlib seaborn statsmodels scikit-learn

Import

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from scipy import stats

6. Load the Dataset

import pandas as pd
df = pd.read_csv("data/customers.csv")
print(df.head())

Check the size

print(df.shape)

Example

(10000, 9)

This means

7. Initial EDA

Check the dataset structure

df.info()

Check statistics

df.describe()

Check categorical variables

df.describe(include="object")

Check missing values

df.isnull().sum()

Check duplicates

df.duplicated().sum()

8. Data Quality Analysis

We should investigate

Missing values

missing = df.isnull().sum()
print(missing)

Percentage

missing_percentage = (
    df.isnull().mean() * 100
)

print(missing_percentage)

Duplicates

print("Duplicates:", df.duplicated().sum())

Data types

print(df.dtypes)

9. Descriptive Statistics

Let's analyze

df["purchase_amount"].describe()

Suppose we get

We can conclude

The average purchase amount is approximately ₹650, while the median is ₹620.

Since

the distribution may be somewhat right-skewed.

10. Mean, Median and Mode

Calculate

mean = df["purchase_amount"].mean()
median = df["purchase_amount"].median()
mode = df["purchase_amount"].mode()[0]
print("Mean:", mean)
print("Median:", median)
print("Mode:", mode)

These provide different views of the center of the data.

Mean

• Average purchase amount.
• Median
• Middle value.
• Mode
• Most frequently occurring value.

11. Variance and Standard Deviation

Calculate

variance = df["purchase_amount"].var()
std = df["purchase_amount"].std()
print("Variance:", variance)
print("Standard deviation:", std)

If

then customer purchase amounts typically vary around the mean by roughly 220 units, although standard deviation is not simply the average absolute distance.

12. Distribution of Purchase Amount

Create a histogram

plt.figure(figsize=(10, 6))

sns.histplot(

df["purchase_amount"],

kde=True

)

plt.title("Purchase Amount Distribution")
plt.xlabel("Purchase Amount")
plt.ylabel("Frequency")
plt.show()

Look for

• Symmetry
• Skewness
• Multiple peaks
• Outliers
• Approximate normality

13. Box Plot

plt.figure(figsize=(10, 4))

sns.boxplot(

x=df["purchase_amount"]

)

plt.title("Purchase Amount Box Plot")
plt.show()

This helps identify potential outliers.

14. Outlier Detection Using IQR

Calculate

Q1 = df["purchase_amount"].quantile(0.25)
Q3 = df["purchase_amount"].quantile(0.75)
IQR = Q3 - Q1
lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQR
print("Lower:", lower_bound)
print("Upper:", upper_bound)

Find potential outliers

outliers = df[
    (df["purchase_amount"] < lower_bound) |
    (df["purchase_amount"] > upper_bound)
]

print(outliers)

15. Probability Analysis

Suppose we define a high-value customer as

We can estimate

using

probability = (
    (df["purchase_amount"] > 1000).mean()
)

print(probability)

If the result is

0.12

then

or

16. Sampling

Suppose the full dataset contains

customers.

Take a random sample

sample = df.sample(
    n=100,
    random_state=42
)

print(sample.head())

This allows us to study how sample statistics relate to population statistics.

17. Central Limit Theorem Demonstration

We can repeatedly sample the data and calculate the mean.

sample_means = []
for i in range(1000):
    sample = df["purchase_amount"].sample(
        n=100,
        replace=True
    )

sample_means.append(

sample.mean()

)

Plot the sampling distribution

plt.figure(figsize=(10, 6))

sns.histplot(

sample_means,

kde=True

)

plt.title("Sampling Distribution of Mean")
plt.xlabel("Sample Mean")
plt.ylabel("Frequency")
plt.show()

Even if the original distribution is not perfectly normal, the distribution of sample means tends toward normality as sample size increases under the conditions of the CLT.

This demonstrates the Central Limit Theorem.

18. Confidence Interval for Mean Purchase

Suppose

data = df["purchase_amount"].dropna()
n = len(data)
mean = data.mean()
std = data.std()
confidence = 0.95
alpha = 1 - confidence
t_critical = stats.t.ppf(
    1 - alpha / 2,
    df=n - 1
)

margin = (
    t_critical *
    std /
    np.sqrt(n)
)

lower = mean - margin
upper = mean + margin
print("Mean:", mean)
print("95% CI:", lower, upper)

We might obtain

Interpretation

Using this confidence-interval method, the population mean purchase amount is estimated to lie between approximately ₹646 and ₹654.

19. Hypothesis Test

Suppose the company believes

Average customer spending is ₹600.

We want to test

Run

result = stats.ttest_1samp(
    df["purchase_amount"].dropna(),
    popmean=600
)

print("t-statistic:", result.statistic)
print("p-value:", result.pvalue)

Suppose

Then

Conclusion

There is statistically significant evidence that the average customer purchase amount differs from ₹600.

20. Correlation Analysis

Let's investigate

Is customer income related to purchase amount?

Calculate

correlation = df[

].corr()

print(correlation)

Or

• r, p = stats.pearsonr(
• df["income"],
• df["purchase_amount"]

)

print("Correlation:", r)
print("p-value:", p)

Suppose

Then income and purchase amount have a strong positive linear association.

21. Visualize Correlation

sns.scatterplot(

data=df,
x="income",
y="purchase_amount"

)

plt.title(
    "Income vs Purchase Amount"
)

plt.show()

A scatter plot helps us determine whether the relationship actually looks linear.

22. Covariance

Calculate

covariance = df[
    "income"
].cov(
    df["purchase_amount"]
)

print("Covariance:", covariance)

If covariance is positive

then the variables tend to move in the same direction.

Remember

Covariance is affected by the units of measurement, whereas correlation is standardized.

23. Chi-Square Test

Now consider two categorical variables

Membership

Purchase Status

Create a contingency table

table = pd.crosstab(
    df["membership"],
    df["purchase_status"]
)

print(table)

Example

Now perform the chi-square test

chi2, p, dof, expected = (

stats.chi2_contingency(table)

)

print("Chi-square:", chi2)
print("p-value:", p)
print("Degrees of freedom:", dof)

24. Chi-Square Interpretation

Hypotheses

Suppose

Then

Conclusion

There is statistically significant evidence of an association between membership type and purchase status.

Again

25. ANOVA

Now ask

Does average purchase amount differ among membership groups?

Groups

• Basic
• Premium
• Gold

Run

basic = df.loc[
    df["membership"] == "Basic",
    "purchase_amount"
].dropna()

premium = df.loc[
    df["membership"] == "Premium",
    "purchase_amount"
].dropna()

gold = df.loc[
    df["membership"] == "Gold",
    "purchase_amount"
].dropna()

Run ANOVA

result = stats.f_oneway(
    basic,
    premium,
    gold
)

print("F-statistic:", result.statistic)
print("p-value:", result.pvalue)

26. ANOVA Interpretation

Hypotheses

[ H_0: \mu_{Basic}

\mu_{Premium}

\mu_{Gold} ]

Alternative

If

we reject (H_0).

Conclusion

There is statistically significant evidence that average purchase amounts are not all equal across membership groups.

To determine which groups differ, use a post-hoc test such as Tukey HSD.

27. Tukey HSD

from statsmodels.stats.multicomp import pairwise_tukeyhsd
data = df[

].dropna()

tukey = pairwise_tukeyhsd(
    endog=data["purchase_amount"],
    groups=data["membership"],
    alpha=0.05
)

print(tukey)

This can tell us whether

• Basic vs Premium
• Basic vs Gold
• Premium vs Gold
• show statistically significant differences.