Data Analysis & Visualization

NumPy (Numerical Python) is the fundamental numerical-computing library in the Python data-analysis ecosystem. While basic NumPy focuses on creating arrays and performing simple operations, Advanced NumPy focuses on efficient computation, multidimensional data manipulation, broadcasting, vectorization, memory management, linear algebra, and statistical operations.

6.1.1Learning Objectives

By the end of this topic, you should be able to

• Work confidently with multidimensional NumPy arrays.
• Understand NumPy's memory model and array dimensions.
• Use broadcasting effectively.
• Perform vectorized calculations without Python loops.
• Apply advanced indexing and Boolean filtering.
• Reshape, transpose, split, stack, and combine arrays.
• Perform statistical calculations efficiently.
• Work with matrices and linear algebra.
• Generate random data for analysis and simulations.
• Understand views versus copies.
• Optimize numerical Python code.

6.1.2NumPy Array Fundamentals

A NumPy array is an object of type numpy.ndarray.

import numpy as np
data = np.array([10, 20, 30, 40, 50])
print(data)
print(type(data))
print(data.ndim)
print(data.shape)
print(data.size)
print(data.dtype)

Output

<class 'numpy.ndarray'>

1

(5,)

5

int64

Important properties

Example

data = np.array([
    [10, 20, 30],

])

print(data.ndim)
print(data.shape)
print(data.size)
print(data.dtype)
print(data.nbytes)

The shape is

(2, 3)

This means

• 2 rows
• 3 columns
• 6 total elements

6.1.3Multidimensional Arrays

NumPy supports arrays with multiple dimensions.

1D

a = np.array([10, 20, 30])

Shape

(3,)

2D

b = np.array([
    [10, 20, 30],

])

Shape

(2, 3)

3D

c = np.array([
    [
        [1, 2],

],

[

[5, 6],

]

])

Shape

(2, 2, 2)

You can think of a 3D array as a collection of 2D tables.

6.1.4Array Data Types

NumPy supports specialized numerical data types.

a = np.array([1, 2, 3], dtype=np.int32)
print(a.dtype)

Common types include

• int8
• int16
• int32
• int64
• float32
• float64
• bool
• complex64
• complex128

You can convert an array using astype()

prices = np.array([100, 200, 300])
prices_float = prices.astype(float)
print(prices_float)
print(prices_float.dtype)

Why is this important?

Data type affects

• memory usage
• computation speed
• precision
• compatibility with other libraries

For example

data = np.array([1, 2, 3, 4], dtype=np.int8)

uses significantly less memory than

data = np.array([1, 2, 3, 4], dtype=np.int64)

This becomes important when processing millions of records.

6.1.5Advanced Array Creation

NumPy provides several methods for creating arrays.

arange()

numbers = np.arange(1, 11)
print(numbers)

Output

With a step

numbers = np.arange(0, 20, 2)
print(numbers)

Output

linspace()

linspace() generates a specified number of equally spaced values.

numbers = np.linspace(0, 100, 11)
print(numbers)

Output

This is especially useful for mathematical calculations and plotting.

zeros()

a = np.zeros((3, 4))
print(a)

Creates a 3 × 4 array containing zeros.

ones()

a = np.ones((2, 3))
print(a)

full()

a = np.full((3, 3), 100)
print(a)

Output

Identity Matrix

identity = np.eye(4)
print(identity)

Output

6.1.6Advanced Indexing

Basic indexing

data = np.array([10, 20, 30, 40, 50])
print(data[0])
print(data[-1])

Output

• 10
• 50
• Slicing

print(data[1:4])

Output

You can also use a step

print(data[::2])

Output

6.1.72D Array Indexing

data = np.array([
    [10, 20, 30],
    [40, 50, 60],

])

Access a specific element

print(data[1, 2])

Output

• 60
• The first number represents the row and the second represents the column.
• data[row, column]
• Selecting a row

print(data[1])

Output

Selecting a column

print(data[:, 1])

Output

The : means "all rows".

6.1.8Boolean Indexing

Boolean indexing is one of the most useful NumPy techniques in data analysis.

sales = np.array([100, 250, 80, 450, 300])
high_sales = sales[sales > 200]
print(high_sales)

Output

You can create the Boolean condition separately

condition = sales > 200
print(condition)

Output

Then

print(sales[condition])

Multiple Conditions

Use & for AND

sales = np.array([100, 250, 80, 450, 300])
result = sales[(sales > 200) & (sales < 400)]
print(result)

Output

Use | for OR

result = sales[(sales < 100) | (sales > 400)]
print(result)

Output

6.1.9Fancy Indexing

Fancy indexing allows you to select specific positions.

data = np.array([10, 20, 30, 40, 50])
indexes = [0, 2, 4]
print(data[indexes])

Output

For a 2D array

data = np.array([
    [10, 20, 30],
    [40, 50, 60],

])

rows = [0, 2]
cols = [1, 2]
print(data[rows, cols])

Output

6.1.10Broadcasting

Broadcasting allows NumPy to perform operations between arrays with compatible shapes.

Consider

prices = np.array([100, 200, 300])
discount = 10
result = prices - discount
print(result)

Output

NumPy automatically applies 10 to every element.

This is broadcasting.

Broadcasting with 2D Arrays

sales = np.array([
    [100, 200, 300],

])

tax = np.array([10, 20, 30])
result = sales + tax
print(result)

Output

The tax array is effectively applied to every row.

6.1.11Broadcasting Rules

When NumPy performs operations between arrays, dimensions are compared from right to left.

Two dimensions are compatible when

They are equal, or

One of them is 1.

For example

Array A: (3, 4)

Array B: (4,)

These are compatible.

Another example

Array A: (3, 1)

Array B: (1, 4)

These are also compatible.

The result becomes

(3, 4)

Understanding broadcasting is extremely important for efficient numerical programming.

6.1.12Vectorization

Vectorization means performing operations on entire arrays instead of processing elements individually using Python loops.

Traditional Python

numbers = [1, 2, 3, 4, 5]
result = []
for number in numbers:
    result.append(number * 2)

print(result)

NumPy

numbers = np.array([1, 2, 3, 4, 5])
result = numbers * 2
print(result)

Output

NumPy's approach is generally faster and more concise for numerical operations.

6.1.13Universal Functions — ufuncs

NumPy provides optimized functions called universal functions, or ufuncs.

Examples

numbers = np.array([1, 4, 9, 16])
print(np.sqrt(numbers))
print(np.square(numbers))
print(np.exp(numbers))
print(np.log(numbers))

Other useful functions include

np.abs()
np.sin()
np.cos()
np.tan()
np.ceil()
np.floor()
np.round()

Example

prices = np.array([100.4, 200.6, 300.2])
print(np.round(prices))

Output

6.1.14Aggregation Functions

NumPy provides efficient statistical functions.

sales = np.array([100, 200, 300, 400, 500])
print(np.sum(sales))
print(np.mean(sales))
print(np.median(sales))
print(np.min(sales))
print(np.max(sales))
print(np.std(sales))
print(np.var(sales))

Common functions

6.1.15Axis Operations

Consider

sales = np.array([
    [100, 200, 300],

])

Sum of all values

print(np.sum(sales))

Output

2100

Sum by column

print(np.sum(sales, axis=0))

Output

Sum by row

print(np.sum(sales, axis=1))

Output

A useful way to remember

axis=0 → operate down rows → result for each column
axis=1 → operate across columns → result for each row

6.1.16Reshaping Arrays

Suppose

data = np.arange(1, 13)
print(data)

Output

Convert it into a 3 × 4 matrix

matrix = data.reshape(3, 4)
print(matrix)

Output

You can use -1 when NumPy should calculate one dimension

matrix = data.reshape(3, -1)
print(matrix)

6.1.17Flattening Arrays

Convert a multidimensional array into one dimension.

data = np.array([
    [1, 2, 3],

])

flat = data.flatten()
print(flat)

Output

Another option

flat = data.ravel()

There is an important difference

flatten() generally returns a copy.

ravel() attempts to return a view where possible.

6.1.18Transpose

Transpose converts rows into columns.

data = np.array([
    [1, 2, 3],

])

print(data.T)

Output

You can also use

np.transpose(data)

6.1.19Stacking Arrays

Vertical stacking

a = np.array([1, 2, 3])
b = np.array([4, 5, 6])
result = np.vstack((a, b))
print(result)

Output

Horizontal stacking

result = np.hstack((a, b))
print(result)

Output

For multidimensional arrays

a = np.array([[1, 2], [3, 4]])
b = np.array([[5, 6], [7, 8]])
result = np.vstack((a, b))
print(result)

6.1.20Concatenation

a = np.array([1, 2, 3])
b = np.array([4, 5, 6])
result = np.concatenate((a, b))
print(result)

Output

For 2D arrays

a = np.array([
    [1, 2],

])

b = np.array([
    [5, 6],

])

result = np.concatenate((a, b), axis=0)
print(result)

6.1.21Splitting Arrays

data = np.arange(1, 10)
parts = np.split(data, 3)
for part in parts:
print(part)

Output

Other functions

np.hsplit()
np.vsplit()
np.array_split()

array_split() is useful when the array cannot be divided equally.

6.1.22Sorting

data = np.array([50, 10, 40, 20, 30])
print(np.sort(data))

Output

To get the indexes that would sort the array

indexes = np.argsort(data)
print(indexes)

This is especially useful when working with related arrays.

6.1.23Searching

Find the index of the maximum value

sales = np.array([100, 500, 200, 800, 300])
print(np.argmax(sales))

Output

3

Find the minimum

print(np.argmin(sales))

Find all values satisfying a condition

indexes = np.where(sales > 300)
print(indexes)

Output

(array([1, 3]),)

6.1.24np.where()

np.where() is extremely useful for conditional transformations.
sales = np.array([100, 500, 200, 800, 300])
status = np.where(sales >= 500, "High", "Low")
print(status)

Output

This is similar to a SQL CASE WHEN.

For example

CASE

• WHEN sales >= 500 THEN 'High'
• ELSE 'Low'
• END

This makes NumPy particularly useful for people working with SQL and analytics.

6.1.25Unique Values

customers = np.array([
    "A", "B", "A", "C", "B", "D"
])

print(np.unique(customers))

Output

You can also get counts

values, counts = np.unique(

customers,

return_counts=True

)

print(values)
print(counts)

Output

6.1.26Views vs Copies

This is an important advanced NumPy concept.

Consider

data = np.array([10, 20, 30, 40])
view = data[1:3]

view[0] = 999

print(data)

The original array may now contain

because slicing generally produces a view into the original array.

To create an independent copy

copy_data = data[1:3].copy()

copy_data[0] = 1000

Now changes to copy_data do not modify the original array.

Key distinction

View → shares underlying data

Copy → independent data

This is important when working with large datasets because unnecessary copies can consume significant memory.

6.1.27Memory Layout

NumPy arrays store data efficiently in memory.

You can inspect memory information

data = np.array([1, 2, 3, 4, 5])
print(data.itemsize)
print(data.nbytes)

For example, if each integer occupies 8 bytes

itemsize = 8
nbytes = 40

For large-scale analytics, memory-efficient data types can make a significant difference.

6.1.28Linear Algebra

NumPy provides a dedicated linear algebra module

np.linalg

Matrix multiplication

A = np.array([
    [1, 2],

])

B = np.array([
    [5, 6],

])

result = A @ B
print(result)

Output

You can also use

np.matmul(A, B)

Matrix inverse

A = np.array([
    [1, 2],

])

inverse = np.linalg.inv(A)
print(inverse)

Determinant

det = np.linalg.det(A)
print(det)

Eigenvalues

values, vectors = np.linalg.eig(A)

print(values)
print(vectors)

Linear algebra is particularly important in machine learning, statistics, optimization, and scientific computing.

6.1.29Random Number Generation

NumPy provides a modern random-number API.

rng = np.random.default_rng(42)
numbers = rng.integers(1, 101, size=10)
print(numbers)

The 42 is a seed, which makes the result reproducible.

Random floating-point numbers

numbers = rng.random(5)
print(numbers)

Normal distribution

values = rng.normal(
    loc=100,
    scale=15,
    size=1000
)

Here

loc = mean
scale = standard deviation
size = number of observations

6.1.30Statistical Analysis

Consider a sales dataset

sales = np.array([
    1200, 1500, 1800, 1100,
    2200, 2500, 1700, 1900
])

Calculate statistics

print("Mean:", np.mean(sales))
print("Median:", np.median(sales))
print("Minimum:", np.min(sales))
print("Maximum:", np.max(sales))
print("Standard Deviation:", np.std(sales))

Percentiles

print(np.percentile(sales, 25))
print(np.percentile(sales, 50))
print(np.percentile(sales, 75))

These statistics are frequently used during Exploratory Data Analysis (EDA).

6.1.31Handling NaN Values

NumPy also provides functions specifically for missing numerical values represented as NaN.

data = np.array([
    10,
    20,
    np.nan,
    40,
    50
])

Regular mean

print(np.mean(data))

This produces nan.

Instead

print(np.nanmean(data))

Output

30.0

Other useful functions

np.nansum()
np.nanmean()
np.nanmedian()
np.nanmin()
np.nanmax()
np.nanstd()

6.1.32Practical Example — Sales Analysis

Suppose a company has monthly sales

sales = np.array([
    [12000, 15000, 14000, 16000],
    [18000, 21000, 19000, 22000],

])

Each row represents a region and each column represents a month.

Total sales

total_sales = np.sum(sales)
print(total_sales)

Sales by region

region_sales = np.sum(sales, axis=1)
print(region_sales)

Sales by month

monthly_sales = np.sum(sales, axis=0)
print(monthly_sales)

Average monthly sales

average_sales = np.mean(sales, axis=0)
print(average_sales)

Highest-performing region

best_region = np.argmax(region_sales)
print(best_region)

This demonstrates how NumPy can perform large-scale calculations without writing explicit loops.

6.1.33NumPy vs Python Lists

Consider

numbers = [1, 2, 3, 4, 5]

Python lists are general-purpose containers.

NumPy arrays are optimized for numerical computation.

6.1.34NumPy and SQL Comparison

Since data analysts frequently work with SQL, several NumPy concepts can be related to SQL.

For example, SQL

SELECT *
FROM sales

WHERE amount > 500;

Equivalent NumPy filtering

result = sales[sales > 500]

This comparison helps bridge the gap between SQL-based analytics and Python-based analytics.

6.1.35Performance: Loop vs Vectorization

Consider

numbers = np.arange(1_000_000)

A Python loop

result = []
for number in numbers:
    result.append(number * 2)

Vectorized NumPy

result = numbers * 2

The second approach is generally preferable because the operation is performed using NumPy's optimized numerical machinery.

General principle

Instead of

for row in data:
    calculate(row)

look for opportunities to perform

result = operation(data)

This is one of the most important habits in effective NumPy programming.

6.1.36Common NumPy Mistakes

Mistake 1 — Using and instead of &

Incorrect

data[(data > 10) and (data < 50)]

Correct

data[(data > 10) & (data < 50)]

Mistake 2 — Forgetting parentheses

Incorrect

data > 10 & data < 50

Correct

(data > 10) & (data < 50)

Mistake 3 — Confusing shape

a = np.array([1, 2, 3])
print(a.shape)

Output

(3,)

This is not

(1, 3)

If you need a row vector

a.reshape(1, 3)

Mistake 4 — Unexpected modification through views

subset = data[1:4]

Remember that slicing can create a view.

Use

subset = data[1:4].copy()

when an independent array is required.

6.1.37Mini Project — Regional Sales Analysis

Dataset

import numpy as np
sales = np.array([
    [12000, 15000, 14000, 16000, 18000, 19000],
    [18000, 21000, 19000, 22000, 23000, 25000],
    [10000, 12000, 13000, 14000, 15000, 16000],

])

Assume

Rows → Regions

• Columns → Months
• Tasks
• Task 1: Calculate total company sales.

total_sales = np.sum(sales)

Task 2: Calculate total sales for each region.

region_sales = np.sum(sales, axis=1)

Task 3: Calculate sales for each month.

monthly_sales = np.sum(sales, axis=0)

Task 4: Find the highest-performing region.

best_region = np.argmax(region_sales)

Task 5: Find the month with the highest sales.

best_month = np.argmax(monthly_sales)

Task 6: Calculate average sales.

average_sales = np.mean(sales)

Task 7: Find all sales values greater than ₹20,000.

high_sales = sales[sales > 20000]

6.1.38Interview Questions

• Beginner/Intermediate
• What is NumPy?
• What is an ndarray?
• What is the difference between shape, size, and ndim?
• What is vectorization?
• What is broadcasting?
• What is Boolean indexing?
• What is fancy indexing?
• What is the difference between reshape() and resize()?
• What is the difference between flatten() and ravel()?
• What does axis=0 mean?

Advanced

• What are NumPy views and copies?
• How does broadcasting work?
• Why is NumPy faster than Python loops?
• What are ufuncs?
• What is the difference between np.array() and Python lists?
• What is np.where()?
• What does np.argsort() return?
• How do you handle NaN values in NumPy?
• How can you reduce NumPy memory consumption?
• What is the difference between @, np.dot(), and np.matmul()?
• How does NumPy store multidimensional arrays?
• What are strides?
• How does NumPy avoid unnecessary data copies?
• How would you optimize a NumPy calculation involving millions of records?

Explain broadcasting with an example.

6.1.39Key Takeaways

The most important concepts to master are

For data analysis, the five concepts you should become especially strong at are Boolean indexing, broadcasting, vectorization, axis-based aggregation, and views vs copies. These form the foundation for advanced Pandas and efficient Python data-processing workflows.

Pandas is one of the most important Python libraries for data analysis. While basic Pandas teaches you how to create DataFrames, select columns, filter rows, and calculate simple statistics, Advanced Pandas focuses on complex data manipulation, aggregation, joining datasets, reshaping, hierarchical indexing, time-series operations, window functions, and performance optimization.

6.2.1Learning Objectives

By the end of this topic, you should be able to

• Work efficiently with large DataFrames.
• Perform advanced filtering and indexing.
• Use groupby() for complex aggregations.
• Merge and join multiple datasets.
• Reshape data using pivot(), pivot_table(), melt(), and stack().
• Work with MultiIndex DataFrames.
• Apply transformations using apply(), map(), and transform().
• Perform rolling and expanding calculations.
• Work with dates and time-series data.
• Handle duplicate and inconsistent data.
• Optimize DataFrame memory usage.
• Build reusable data-processing pipelines.

6.2.2Pandas Data Structures

Pandas provides two primary data structures

Series

A one-dimensional labeled structure.

import pandas as pd
sales = pd.Series(
    [100, 200, 300],
    index=["Jan", "Feb", "Mar"]
)

print(sales)

DataFrame

A two-dimensional tabular structure.

data = {
    "Product": ["Laptop", "Phone", "Tablet"],
    "Sales": [50000, 30000, 20000]
}

df = pd.DataFrame(data)
print(df)

A DataFrame can be thought of as a table containing

• rows
• columns
• indexes
• values
• data types

6.2.3Understanding DataFrame Metadata

Before performing analysis, inspect the DataFrame.

print(df.shape)
print(df.columns)
print(df.index)
print(df.dtypes)

Useful methods

df.head()
df.tail()
df.info()
df.describe()
df.memory_usage()

For example

print(df.info())

This helps identify

• number of records
• column names
• missing values
• data types
• memory usage

6.2.4Advanced Selection

Suppose

df = pd.DataFrame({
    "Employee": ["A", "B", "C", "D"],
    "Department": ["IT", "HR", "IT", "Finance"],
    "Salary": [60000, 50000, 75000, 65000],
    "Experience": [3, 2, 6, 5]
})

Select multiple columns

result = df[["Employee", "Salary"]]

Select rows using loc

result = df.loc[0:2, ["Employee", "Salary"]]

Select rows using iloc

result = df.iloc[0:3, [0, 2]]

Remember

loc → label-based selection

iloc → position-based selection

6.2.5Advanced Filtering

Simple condition

result = df[df["Salary"] > 60000]

Multiple conditions

result = df[
    (df["Salary"] > 60000) &
    (df["Experience"] >= 5)
]

OR condition

result = df[
    (df["Department"] == "IT") |
    (df["Department"] == "Finance")
]

Using isin()

Instead of

• df[
• (df["Department"] == "IT") |
• (df["Department"] == "Finance")

]

you can write

df[

df["Department"].isin(["IT", "Finance"])

]

This is cleaner when working with multiple values.

6.2.6Using query()

Pandas provides a SQL-like filtering syntax.

result = df.query("Salary > 60000")

Multiple conditions

result = df.query(
    "Salary > 60000 and Experience >= 5"
)

String values

result = df.query(
    "Department == 'IT'"
)

This can make complex filtering easier to read.

6.2.7Sorting Data

Sort by one column

df.sort_values("Salary")

Descending

df.sort_values(
    "Salary",
    ascending=False
)

Sort by multiple columns

df.sort_values(
    ["Department", "Salary"],
    ascending=[True, False]
)

Sort by index

df.sort_index()

6.2.8groupby() — The Most Important Advanced Concept

groupby() is one of the most powerful Pandas features.

Consider

sales = pd.DataFrame({
    "Region": [
        "South", "South", "North",
        "North", "West", "West"
    ],
    "Product": [
        "Laptop", "Phone", "Laptop",
        "Phone", "Laptop", "Phone"
    ],
    "Sales": [
        50000, 30000, 45000,
        35000, 60000, 40000
    ]
})

Calculate total sales by region

result = sales.groupby("Region")["Sales"].sum()
print(result)

This is conceptually similar to

• SELECT
• Region,
• SUM(Sales)
• FROM sales
• GROUP BY Region;

6.2.9Multiple Aggregations

result = sales.groupby("Region")["Sales"].agg(

)

print(result)

You can also give custom names

result = sales.groupby("Region").agg(
    Total_Sales=("Sales", "sum"),
    Average_Sales=("Sales", "mean"),
    Maximum_Sales=("Sales", "max")
)

This produces a clean analytical result.

6.2.10Grouping by Multiple Columns

result = sales.groupby(

)["Sales"].sum()

print(result)

This provides sales at

Region + Product

level.

For example

South Laptop

South Phone

North Laptop

North Phone

West Laptop

West Phone

This is equivalent to grouping by multiple dimensions in SQL.

6.2.11transform()

transform() is different from groupby().agg().

Suppose

sales = pd.DataFrame({
    "Region": ["South", "South", "North", "North"],
    "Sales": [100, 300, 200, 400]
})

Calculate regional average

• sales["Region_Avg"] = (
• sales.groupby("Region")["Sales"]
• .transform("mean")

)

print(sales)

Result

Region Sales Region_Avg

0 South 100 200

1 South 300 200

2 North 200 300

3 North 400 300

The important point is that transform() returns a result aligned with the original rows.

This is extremely useful for

• percentage of group total
• group averages
• normalization
• ranking
• comparison against group-level metrics

6.2.12Percentage of Group Total

• sales["Region_Total"] = (
• sales.groupby("Region")["Sales"]
• .transform("sum")

)

• sales["Percentage"] = (
• sales["Sales"] /
• sales["Region_Total"] * 100

)

This is a common business-analysis calculation.

6.2.13apply()

apply() allows you to execute a custom function.

def classify_salary(salary):
if salary >= 70000:
return "High"
elif salary >= 50000:
return "Medium"
else:
return "Low"

Apply it

df["Salary_Level"] = df["Salary"].apply(

classify_salary

)

You can also use a lambda

df["Salary_Level"] = df["Salary"].apply(

lambda x: "High" if x >= 70000 else "Low"

)

Important distinction

Use

• vectorized operations when possible
• map() for element-wise Series mapping
• apply() when custom logic is necessary
• transform() when the result must align with the original DataFrame

6.2.14map()

map() is commonly used with Series.

mapping = {
    "IT": "Technology",
    "HR": "Human Resources",
    "Finance": "Finance"
}

df["Department_Name"] = (

df["Department"].map(mapping)

)

This is useful for lookup-style transformations.

6.2.15Merging DataFrames

Suppose we have

customers = pd.DataFrame({
    "CustomerID": [1, 2, 3],
    "CustomerName": ["Alice", "Bob", "Charlie"]
})

orders = pd.DataFrame({
    "OrderID": [101, 102, 103, 104],
    "CustomerID": [1, 2, 1, 3],
    "Amount": [500, 700, 300, 900]
})

Merge

result = pd.merge(
    orders,
    customers,
    on="CustomerID"
)

This is equivalent to an SQL

INNER JOIN

6.2.16Types of Joins

Pandas supports

pd.merge(
    left,
    right,
    how="inner",
    on="CustomerID"
)

Available join types

• inner
• left
• right
• outer
• cross
• Inner join
• Only matching records.

pd.merge(
    orders,
    customers,
    how="inner",
    on="CustomerID"
)

Left join

Keep all records from the left DataFrame.

pd.merge(
    orders,
    customers,
    how="left",
    on="CustomerID"
)

Outer join

Keep all records from both DataFrames.

pd.merge(
    orders,
    customers,
    how="outer",
    on="CustomerID"
)

6.2.17Joining on Different Column Names

Suppose

orders = pd.DataFrame({
    "Customer_ID": [1, 2, 3],
    "Amount": [500, 700, 900]
})

customers = pd.DataFrame({
    "ID": [1, 2, 3],
    "Name": ["Alice", "Bob", "Charlie"]
})

Use

result = pd.merge(
    orders,
    customers,
    left_on="Customer_ID",
    right_on="ID"
)

6.2.18Concatenation

concat() is useful for combining DataFrames vertically or horizontally.

df1 = pd.DataFrame({
    "Name": ["A", "B"],
    "Sales": [100, 200]
})

df2 = pd.DataFrame({
    "Name": ["C", "D"],
    "Sales": [300, 400]
})

Vertical concatenation

result = pd.concat(
    [df1, df2],
    ignore_index=True
)

Result

Name Sales

0 A 100

1 B 200

2 C 300

3 D 400

6.2.19Pivot Tables

A PivotTable summarizes data across multiple dimensions.

sales = pd.DataFrame({
    "Region": [
        "South", "South",
        "North", "North"
    ],
    "Product": [
        "Laptop", "Phone",
        "Laptop", "Phone"
    ],
    "Sales": [
        50000, 30000,
        45000, 35000
    ]
})

Create a pivot table

pivot = pd.pivot_table(
    sales,
    values="Sales",
    index="Region",
    columns="Product",
    aggfunc="sum"
)

print(pivot)

This is conceptually similar to an Excel PivotTable.

6.2.20pivot() vs pivot_table()

pivot()

Used when combinations of index and columns are unique.

df.pivot(
    index="Region",
    columns="Product",
    values="Sales"
)

pivot_table()

Can aggregate duplicate combinations.

df.pivot_table(
    index="Region",
    columns="Product",
    values="Sales",
    aggfunc="sum"
)

For business analytics, pivot_table() is often more flexible.

6.2.21Melting Data

melt() converts wide data into long format.

Suppose

df = pd.DataFrame({
    "Product": ["Laptop", "Phone"],
    "Jan": [100, 200],
    "Feb": [150, 250],
    "Mar": [200, 300]
})

Use

long_df = df.melt(
    id_vars="Product",
    var_name="Month",
    value_name="Sales"
)

Result

Product Month Sales

Laptop Jan 100

Laptop Feb 150

Laptop Mar 200

Phone Jan 200

Phone Feb 250

Phone Mar 300

This structure is especially useful for visualization.

6.2.22Stack and Unstack

These methods are commonly used with hierarchical indexes.

df = pd.DataFrame({
    "A": [10, 20],
    "B": [30, 40]
})

Stack

stacked = df.stack()

Unstack

unstacked = stacked.unstack()

They are useful when changing between different dimensional representations.

6.2.23MultiIndex

A MultiIndex allows multiple levels of indexing.

data = pd.DataFrame({
    "Region": ["South", "South", "North", "North"],
    "Product": ["Laptop", "Phone", "Laptop", "Phone"],
    "Sales": [50000, 30000, 45000, 35000]
})

multi = data.set_index(

)

print(multi)

The resulting index contains

Region

Product

This is useful for hierarchical analytical data.

6.2.24Resetting Index

To convert the index back into normal columns

df = multi.reset_index()

This is frequently used after

groupby()

For example

result = (
    sales.groupby("Region")["Sales"]
    .sum()
    .reset_index()
)

This produces a regular DataFrame.

6.2.25Duplicate Handling

Find duplicate rows

df.duplicated()

Count duplicates

df.duplicated().sum()

Remove duplicates

df = df.drop_duplicates()

Duplicates based on specific columns

df = df.drop_duplicates(
    subset=["CustomerID"]
)

Keep the last occurrence

df = df.drop_duplicates(
    subset=["CustomerID"],
    keep="last"
)

6.2.26Missing Values

Check missing values

df.isna().sum()

Remove rows

df.dropna()

Fill with a value

df["Sales"] = df["Sales"].fillna(0)

Fill with mean

df["Salary"] = df["Salary"].fillna(

df["Salary"].mean()

)

Forward fill

df["Sales"] = df["Sales"].ffill()

Backward fill

df["Sales"] = df["Sales"].bfill()

6.2.27String Operations

Pandas provides vectorized string operations through .str.

df["Name"] = df["Name"].str.upper()

Other examples

• df["Name"].str.lower()
• df["Name"].str.title()
• df["Name"].str.strip()
• df["Name"].str.len()

Search for text

• df[
• df["Name"].str.contains(
• "john",

case=False,
na=False

)

]

Extract text

df["Email"].str.extract(

r"@(.*)"

)

This is extremely useful for cleaning real-world datasets.

6.2.28Date and Time Operations

Convert a column to datetime

df["OrderDate"] = pd.to_datetime(

df["OrderDate"]

)

Extract year

df["Year"] = df["OrderDate"].dt.year

Month

df["Month"] = df["OrderDate"].dt.month

Month name

df["Month_Name"] = (

df["OrderDate"].dt.month_name()

)

Day

df["Day"] = df["OrderDate"].dt.day

Day of week

df["DayOfWeek"] = (

df["OrderDate"].dt.day_name()

)

6.2.29Time Difference

Suppose

df["StartDate"] = pd.to_datetime(

df["StartDate"]

)

df["EndDate"] = pd.to_datetime(

df["EndDate"]

)

Calculate duration

• df["Duration"] = (
• df["EndDate"] -
• df["StartDate"]

)

Number of days

df["Duration_Days"] = (

df["Duration"].dt.days

)

6.2.30Rolling Window Calculations

Rolling calculations are extremely important in time-series analysis.

Suppose

sales = pd.Series(

)

Three-period moving average

moving_avg = sales.rolling(3).mean()
print(moving_avg)

Conceptually

• 100
• 120
• (100 + 120 + 150) / 3
• (120 + 150 + 130) / 3

...

Rolling calculations are useful for

• moving averages
• smoothing
• trend analysis
• monitoring
• anomaly detection

6.2.31Expanding Calculations

An expanding calculation uses all observations available up to the current row.

sales = pd.Series(

)

cumulative_avg = sales.expanding().mean()
print(cumulative_avg)

Cumulative sum

cumulative_sales = sales.cumsum()

6.2.32Ranking

Suppose

df = pd.DataFrame({
    "Employee": ["A", "B", "C", "D"],
    "Sales": [500, 800, 300, 700]
})

Rank

df["Rank"] = df["Sales"].rank(

ascending=False

)

Result

Employee Sales Rank

A 500 3

B 800 1

C 300 4

D 700 2

Ranking is frequently used in business reporting.

6.2.33Top-N Analysis

Top 3 records

top3 = df.nlargest(
    3,
    "Sales"
)

Bottom 3

bottom3 = df.nsmallest(
    3,
    "Sales"
)

This is simpler than manually sorting and slicing.

6.2.34Cumulative Calculations

Cumulative sum

df["CumulativeSales"] = (

df["Sales"].cumsum()

)

Cumulative maximum

df["RunningMax"] = (

df["Sales"].cummax()

)

Cumulative minimum

df["RunningMin"] = (

df["Sales"].cummin()

)

6.2.35cut() and qcut()

These functions are useful for creating categories.

cut()

Create fixed ranges

ages = pd.Series(

)

groups = pd.cut(
    ages,
    bins=[0, 20, 40, 60, 100],
    labels=[
        "Young",
        "Adult",
        "Middle",
        "Senior"
    ]
)

print(groups)

qcut()

Create approximately equal-sized groups

groups = pd.qcut(
    ages,
    q=4,
    labels=[
        "Q1", "Q2", "Q3", "Q4"
    ]
)

This is useful for

• customer segmentation
• income bands
• sales quartiles
• risk categories

6.2.36Categorical Data

• For columns with a small set of repeated values, the category dtype can reduce memory usage.
• df["Department"] = (
• df["Department"].astype("category")

)

For example

• IT
• IT
• IT
• HR
• HR
• Finance

Instead of storing each string independently, Pandas can represent the values using categorical codes.

This can improve memory efficiency for large datasets.

6.2.37Reading Large Datasets in Chunks

When a CSV is too large to fit comfortably into memory

for chunk in pd.read_csv(
    "large_file.csv",
    chunksize=100000

)

process(chunk)

For example

total_sales = 0
for chunk in pd.read_csv(
    "sales.csv",
    chunksize=100000

)

total_sales += chunk["Sales"].sum()

print(total_sales)

This allows large files to be processed incrementally.

6.2.38Memory Optimization

Check memory usage

df.info(memory_usage="deep")

Convert suitable columns

df["Department"] = (

df["Department"].astype("category")

)

Use smaller numerical types where appropriate

df["Quantity"] = (

df["Quantity"].astype("int32")

)

Avoid unnecessary copies

df2 = df.copy()

only when an independent DataFrame is actually required.

6.2.39Method Chaining

Instead of writing many intermediate variables

df = df.dropna()
df = df[df["Sales"] > 1000]
df = df.sort_values("Sales")

you can use method chaining

result = (
    df
    .dropna()
    .query("Sales > 1000")
    .sort_values("Sales", ascending=False)
)

This makes data-processing pipelines easier to follow.

6.2.40assign()

assign() is useful in method chains.

result = (
    df
    .assign(
        Revenue=lambda x:
        x["Quantity"] * x["Price"]
    )
)

You can create multiple columns

result = (
    df
    .assign(
        Revenue=lambda x:
        x["Quantity"] * x["Price"],
        Discount=lambda x:
        x["Revenue"] * 0.10
    )
)

6.2.41Advanced Business Example

Consider an order dataset

orders = pd.DataFrame({
    "OrderID": [1, 2, 3, 4, 5, 6],
    "Region": [
        "South", "South", "North",
        "North", "West", "West"
    ],
    "Product": [
        "Laptop", "Phone", "Laptop",
        "Phone", "Laptop", "Phone"
    ],
    "Quantity": [2, 5, 3, 4, 1, 6],
    "Price": [
        50000, 30000, 50000,
        30000, 50000, 30000
    ]
})

Calculate revenue

• orders["Revenue"] = (
• orders["Quantity"] *
• orders["Price"]

)

Regional revenue

regional = (
    orders
    .groupby("Region")

.sum()

.reset_index()

)

Product revenue

product_sales = (
    orders
    .groupby("Product")

.sum()

.reset_index()

)

Find the top product

top_product = (
    orders
    .groupby("Product")["Revenue"]
    .sum()
    .nlargest(1)
)

Calculate each order's percentage of regional revenue

• orders["Region_Total"] = (
• orders
• .groupby("Region")["Revenue"]
• .transform("sum")

)

• orders["Region_Percentage"] = (
• orders["Revenue"] /
• orders["Region_Total"] *
• 100

)

This is a realistic analytical workflow.

6.2.42Pandas and SQL Comparison

If you already know SQL, the following mapping is useful

For example, SQL

• SELECT
• Region,
• SUM(Revenue) AS TotalRevenue
• FROM Orders
• GROUP BY Region
• ORDER BY TotalRevenue DESC;

Pandas

result = (
    orders
    .groupby("Region")["Revenue"]
    .sum()
    .sort_values(ascending=False)
    .reset_index(name="TotalRevenue")
)

6.2.43Advanced Window-Function Concept

SQL users often use

SUM(Revenue) OVER (

PARTITION BY Region

)

A Pandas equivalent can be

• orders["Region_Total"] = (
• orders
• .groupby("Region")["Revenue"]
• .transform("sum")

)

For ranking within groups

• orders["Region_Rank"] = (
• orders
• .groupby("Region")["Revenue"]
• .rank(ascending=False)

)

This is a very important bridge between SQL analytics and Pandas analytics.

6.2.44Practical Project — E-Commerce Sales Analysis

Create a DataFrame containing

• OrderID
• OrderDate
• CustomerID
• Region
• Category
• Product
• Quantity
• UnitPrice
• Discount
• Step 1 — Calculate revenue
• df["Revenue"] = (
• df["Quantity"] *
• df["UnitPrice"] *
• (1 - df["Discount"])

)

• Step 2 — Calculate monthly sales
• df["OrderDate"] = pd.to_datetime(
• df["OrderDate"]

)

monthly_sales = (
    df.groupby(
        df["OrderDate"].dt.to_period("M")
    )["Revenue"]
    .sum()
)

Step 3 — Regional performance

regional_sales = (
    df.groupby("Region")["Revenue"]
    .agg(["sum", "mean", "count"])
)

Step 4 — Product performance

product_sales = (
    df.groupby("Product")["Revenue"]
    .sum()
    .sort_values(ascending=False)
)

Step 5 — Top customers

top_customers = (
    df.groupby("CustomerID")["Revenue"]
    .sum()
    .nlargest(10)
)

Step 6 — Monthly moving average

monthly = (
    df.groupby(
        df["OrderDate"].dt.to_period("M")
    )["Revenue"]
    .sum()
)

moving_average = monthly.rolling(3).mean()

This project brings together most of the advanced Pandas concepts.

6.2.45Common Pandas Mistakes

Mistake 1 — Chained assignment

Avoid patterns such as

df[df["Sales"] > 1000]["Status"] = "High"

Prefer

df.loc[
    df["Sales"] > 1000,
    "Status"
] = "High"

This is clearer and avoids problems associated with modifying an intermediate object.

Mistake 2 — Using apply() unnecessarily

Instead of

df["Revenue"] = df["Sales"].apply(

lambda x: x * 1.18

)

prefer a vectorized operation

• df["Revenue"] = df["Sales"] * 1.18
• Vectorized operations are generally clearer and faster.
• Mistake 3 — Forgetting reset_index()

After

result = df.groupby("Region")["Sales"].sum()

Region becomes the index.

If you need a normal DataFrame

result = result.reset_index()

Mistake 4 — Incorrect datetime handling

Don't treat date columns as ordinary strings when you need date calculations.

Use

df["OrderDate"] = pd.to_datetime(

df["OrderDate"]

)

6.2.46Advanced Pandas Cheat Sheet

# Selection

df["Column"]

df[["A", "B"]]

df.loc[rows, columns]
df.iloc[rows, columns]

# Filtering

df[df["Sales"] > 1000]

df.query("Sales > 1000")

df[df["Region"].isin(["South", "North"])]

# Sorting

df.sort_values("Sales")
df.sort_values("Sales", ascending=False)

# Grouping

df.groupby("Region")["Sales"].sum()
df.groupby("Region")["Sales"].mean()
df.groupby("Region").agg({
    "Sales": ["sum", "mean", "max"]
})

# Transformation

df.groupby("Region")["Sales"].transform("sum")

# Joining

pd.merge(df1, df2, on="ID", how="left")

# Combining

pd.concat([df1, df2])

# Reshaping

df.pivot()
df.pivot_table()
df.melt()
df.stack()
df.unstack()

# Missing data

df.isna()
df.dropna()
df.fillna()
df.ffill()
df.bfill()

# Duplicates

df.duplicated()
df.drop_duplicates()

# Dates

pd.to_datetime(df["Date"])

• df["Date"].dt.year
• df["Date"].dt.month
• df["Date"].dt.day
• # Ranking
• df["Rank"] = df["Sales"].rank()
• # Rolling
• df["Sales"].rolling(3).mean()
• # Cumulative
• df["Sales"].cumsum()
• # Top N

df.nlargest(10, "Sales")
df.nsmallest(10, "Sales")

# Categorization

pd.cut()
pd.qcut()

# Memory

df.info(memory_usage="deep")

6.2.47Interview Questions

Intermediate

• What is the difference between a Series and a DataFrame?
• What is the difference between loc and iloc?
• What is groupby()?
• What is the difference between merge() and concat()?
• What is a PivotTable?
• What is the difference between pivot() and pivot_table()?
• What does reset_index() do?
• What is melt()?
• How do you handle duplicate records?
• How do you handle missing values?

Advanced

• Explain groupby().agg() versus groupby().transform().
• When would you use apply()?
• Why should vectorized operations generally be preferred over apply()?
• Explain Pandas MultiIndex.
• How do you perform a left join in Pandas?
• How do you calculate a rolling average?
• How would you implement a SQL window function in Pandas?
• How can you optimize Pandas memory usage?
• How do you process a CSV file that is larger than available memory?
• What is method chaining?
• What is the difference between map(), apply(), and transform()?
• How do you calculate a group's percentage of total?
• How would you rank records within each group?
• How would you perform time-series aggregation?
• How would you optimize a slow Pandas transformation?

6.2.48Key Takeaways

The most important advanced Pandas concepts are

• The five concepts to master first

If your goal is Data Analyst / Data Engineer / BI Developer work, prioritize

• groupby() + agg()
• merge() / joins
• transform() and window-style calculations
• pivot_table() / melt()
• Datetime + rolling operations

Once these are strong, you can move naturally from SQL → Pandas → Visualization → Power BI, which is the core workflow for practical data analysis.

Data cleaning is the process of identifying, correcting, removing, or transforming inaccurate, incomplete, inconsistent, duplicated, and improperly formatted data before analysis.

In real-world data projects, data is rarely ready for immediate analysis. It may contain

• Missing values
• Duplicate records
• Incorrect data types
• Invalid dates
• Spelling variations
• Extra spaces
• Incorrect numerical values
• Outliers
• Inconsistent categories
• Invalid business rules

A good data analyst spends significant time making the dataset accurate, consistent, complete, and analysis-ready.

6.3.1Learning Objectives

By the end of this topic, you should be able to

• Understand data-quality problems.
• Inspect a raw dataset systematically.
• Identify duplicate records.
• Detect missing and invalid values.
• Standardize text and categorical data.
• Convert incorrect data types.
• Validate dates and numerical values.
• Detect inconsistent business values.
• Build reusable data-cleaning pipelines.
• Maintain data-quality checks.
• Produce a clean analytical dataset.

6.3.2What Is Data Cleaning?

Suppose you receive this customer dataset

There are several problems

• Ravi/RAVI may represent the same customer.
• Hyderabad and hyderabad are inconsistent.
• Age -5 is invalid.
• Email is missing.
• Bangalore and Bengaluru may represent the same location.
• Duplicate customer information may exist.
• Cleaning converts this raw dataset into a reliable analytical dataset.

6.3.3Data Quality Dimensions

Data quality can be evaluated using several dimensions.

1. Accuracy

Does the value represent reality?

Example

Age = -10

is not accurate.

2. Completeness

Are required values present?

Example

Email = NULL

3. Consistency

Are values represented consistently?

Example

• Hyderabad
• hyderabad
• HYDERABAD

4. Validity

Does the value follow the required format?

Example

Email = "hello"

may not be a valid email address.

5. Uniqueness

Are duplicate records present?

6. Timeliness

Is the data current enough for the business purpose?

6.3.4Typical Data-Cleaning Workflow

A practical workflow is

The order can vary depending on the dataset.

6.3.5Loading Data

Example CSV

import pandas as pd
df = pd.read_csv("customers.csv")
print(df.head())

Inspect the dataset

print(df.shape)
print(df.columns)
print(df.dtypes)

6.3.6Initial Data Inspection

First records

df.head()

Last records

df.tail()

Dataset information

df.info()

Statistical summary

df.describe()

For categorical columns

df.describe(include="object")

For all columns

df.describe(include="all")

6.3.7Understanding Data Types

Suppose

print(df.dtypes)

Output

Customer_ID int64

Name object

Age object

Join_Date object

Sales float64

The Age and Join_Date columns may require conversion.

6.3.8Converting Data Types

Convert age to numeric

df["Age"] = pd.to_numeric(

df["Age"],

errors="coerce"

)

errors="coerce" converts invalid values to NaN.

For example

"35" → 35

"42" → 42

"unknown" → NaN

6.3.9Date Conversion

Suppose

• Join_Date
• 01-01-2025
• 15-02-2025
• invalid

Convert

df["Join_Date"] = pd.to_datetime(

df["Join_Date"],

errors="coerce"

)

Invalid dates become missing values.

Then inspect

df[df["Join_Date"].isna()]

6.3.10Detecting Missing Values

Check whether values are missing

df.isna()

Count missing values

df.isna().sum()

Percentage of missing values

missing_percentage = (
    df.isna().mean() * 100
)

print(missing_percentage)

This is much more useful than simply knowing the number of missing records.

6.3.11Missing Value Patterns

Suppose

missing = pd.DataFrame({
    "Name": [0, 0, 1, 0],
    "Email": [0, 1, 1, 0],
    "Age": [0, 0, 0, 1]
})

You may discover

Email → 50% missing

Age → 25% missing

Name → 25% missing

The correct treatment depends on the business context.

For example

• Missing email may be acceptable.
• Missing customer ID may be unacceptable.
• Missing transaction amount may require investigation.

6.3.12Detecting Duplicate Records

Check duplicates

df.duplicated()

Count them

df.duplicated().sum()

View duplicate records

df[df.duplicated()]

Remove duplicates

df = df.drop_duplicates()

6.3.13Duplicates Based on Selected Columns

Sometimes the entire row is not duplicated, but a business key is duplicated.

Example

df.duplicated(
    subset=["Customer_ID"]
)

Remove duplicate customers

df = df.drop_duplicates(
    subset=["Customer_ID"]
)

Keep the latest record

df = df.sort_values("Updated_Date")
df = df.drop_duplicates(
    subset=["Customer_ID"],
    keep="last"
)

This is common in customer master and transaction datasets.

6.3.14Standardizing Text

Real-world text often contains inconsistent capitalization.

Example

• Hyderabad
• hyderabad
• HYDERABAD
• HyDeRaBaD

Convert to lowercase

df["City"] = df["City"].str.lower()

Or uppercase

df["City"] = df["City"].str.upper()

Title case

df["City"] = df["City"].str.title()

6.3.15Removing Extra Spaces

Consider

• " Hyderabad"
• "Hyderabad "
• " Hyderabad "

Use

df["City"] = df["City"].str.strip()

For multiple text columns

text_columns = ["Name", "City", "Department"]
for column in text_columns:
    df[column] = df[column].str.strip()

6.3.16Internal Whitespace

Suppose

"New Delhi"

You can normalize repeated spaces

df["City"] = (

• df["City"]
• .str.replace(r"\s+", " ", regex=True)
• .str.strip()

)

Result

New Delhi

6.3.17Standardizing Categories

Suppose a department column contains

• IT
• it
• Information Technology

I.T.

Technology

These may represent the same business category.

Create a mapping

department_mapping = {
    "IT": "Information Technology",
    "it": "Information Technology",
    "I.T.": "Information Technology",
    "Technology": "Information Technology"
}

• df["Department"] = (
• df["Department"]
• .replace(department_mapping)

)

6.3.18Standardizing Locations

Suppose

• Bangalore
• Bengaluru
• BANGALORE

Use a mapping

city_mapping = {
    "Bangalore": "Bengaluru",
    "BANGALORE": "Bengaluru"
}

df["City"] = df["City"].replace(city_mapping)

The important lesson is

Standardization should be based on a defined business rule, not arbitrary assumptions.

6.3.19Handling Invalid Numerical Values

Suppose

df["Age"]

contains

25

32

-5

150

40

A reasonable business rule might be

Age must be between 0 and 120

Find invalid values

invalid_age = df[
    (df["Age"] < 0) |
    (df["Age"] > 120)
]

Convert invalid values to missing

df.loc[
    (df["Age"] < 0) |
    (df["Age"] > 120),
    "Age"
] = pd.NA

6.3.20Business Rule Validation

Data cleaning should not only check technical correctness.

Suppose an order dataset has

• Quantity
• UnitPrice
• Discount
• Revenue

Possible business rules

• Quantity >= 0
• UnitPrice >= 0
• Discount between 0 and 1
• Revenue >= 0

Check quantity

invalid_quantity = df[
    df["Quantity"] < 0
]

Check discount

invalid_discount = df[
    (df["Discount"] < 0) |
    (df["Discount"] > 1)
]

6.3.21Cleaning Email Addresses

Suppose

" Ravi@Gmail.com "

"PRIYA@YAHOO.COM"

Normalize

df["Email"] = (

• df["Email"]
• .str.strip()
• .str.lower()

)

Basic validation

email_pattern = r"^[^@\s]+@[^@\s]+\.[^@\s]+$"
invalid_email = df[
    ~df["Email"].str.match(
        email_pattern,
        na=False
    )
]

This identifies values that don't follow a basic email pattern.

6.3.22Replacing Invalid Values

Suppose the dataset contains

• N/A
• NA
• Unknown
• unknown

-

These should potentially be treated as missing.

missing_values = [
    "N/A",
    "NA",
    "Unknown",
    "unknown",
    "-"
]

df = df.replace(
    missing_values,
    pd.NA
)

This is useful before performing missing-value analysis.

6.3.23Replacing Specific Values

Use replace()

df["Gender"] = df["Gender"].replace({

"M": "Male",

"F": "Female"

})

This creates consistent categories.

6.3.24Detecting Impossible Dates

Suppose an employee's joining date is

2035-01-01

but today is earlier than that.

You can validate

future_dates = df[
    df["Join_Date"] > pd.Timestamp.now()
]

Similarly, business rules may require

OrderDate >= CustomerRegistrationDate

Check

invalid = df[
    df["OrderDate"] <
    df["CustomerRegistrationDate"]
]

6.3.25Data Validation with assert

Assertions can be used for automated quality checks.

For example

assert df["Customer_ID"].notna().all()

Check that sales are non-negative

assert (df["Sales"] >= 0).all()

Check that age is within a valid range

assert df["Age"].between(

0, 120

).all()

Assertions are particularly useful in production pipelines.

6.3.26Creating a Data Quality Report

A simple quality report

quality_report = pd.DataFrame({
    "Column": df.columns,
    "Missing_Count": [
        df[column].isna().sum()
        for column in df.columns
    ],
    "Missing_Percentage": [
        df[column].isna().mean() * 100
        for column in df.columns
    ],
    "Unique_Count": [
        df[column].nunique()
        for column in df.columns
    ],
    "Data_Type": [
        df[column].dtype
        for column in df.columns
    ]
})

print(quality_report)

This gives a quick overview of dataset quality.

6.3.27Detecting Constant Columns

A column containing the same value for every row often provides little analytical value.

constant_columns = [
    column
    for column in df.columns
    if df[column].nunique(dropna=False) <= 1
]

print(constant_columns)

Such columns can potentially be removed after confirming they have no business purpose.

6.3.28Detecting High-Cardinality Columns

Check the number of unique values

df.nunique().sort_values(
    ascending=False
)

High-cardinality columns may include

• Transaction_ID
• Email
• Customer_ID

These are not necessarily bad, but they should be treated differently from low-cardinality categorical columns.

6.3.29Handling Outliers

Data cleaning often includes identifying unusual numerical values.

Example

Q1 = df["Sales"].quantile(0.25)
Q3 = df["Sales"].quantile(0.75)
IQR = Q3 - Q1
lower = Q1 - 1.5 * IQR
upper = Q3 + 1.5 * IQR
outliers = df[
    (df["Sales"] < lower) |
    (df["Sales"] > upper)
]

Do not automatically delete outliers.

An outlier may represent

• data-entry error
• fraud
• genuine large transaction
• exceptional business event
• The correct action requires business understanding.

6.3.30Cleaning Numerical Formatting

Suppose sales values are stored as

• ₹50,000
• ₹1,20,000
• ₹75,000

These are strings, not numbers.

Clean them

df["Sales"] = (

• df["Sales"]
• .str.replace("₹", "", regex=False)
• .str.replace(",", "", regex=False)
• .astype(float)

)

Now

₹50,000

becomes

50000.0

6.3.31Cleaning Percentage Values

Suppose

• 10%
• 20%
• 15%

Clean them

• df["Discount"] = (
• df["Discount"]
• .str.replace("%", "", regex=False)
• .astype(float)
• / 100

)

Now

10%

becomes

0.10

6.3.32Cleaning Phone Numbers

Suppose phone numbers appear as

• +91-9876543210
• 98765 43210
• 9876543210

Remove unwanted characters

df["Phone"] = (

df["Phone"]

.str.replace(r"\D", "", regex=True)

)

This keeps only digits.

Further validation can check length and country-specific business rules.

6.3.33Handling Duplicate Customers

Suppose

customers = pd.DataFrame({
    "CustomerID": [101, 102, 102, 103],
    "Name": ["Ravi", "Priya", "Priya", "Arun"],
    "UpdatedDate": [
        "2025-01-01",
        "2025-01-05",
        "2025-02-01",
        "2025-01-10"
    ]
})

Convert the date

customers["UpdatedDate"] = pd.to_datetime(

customers["UpdatedDate"]

)

Sort

customers = customers.sort_values(
    "UpdatedDate"
)

Keep the latest customer record

customers = customers.drop_duplicates(
    subset=["CustomerID"],
    keep="last"
)

This is a common master-data cleaning pattern.

6.3.34Cleaning a Complete Dataset

Consider

df = pd.DataFrame({
    "Customer_ID": [101, 102, 102, 103],
    "Name": [
        " Ravi ",
        "PRIYA",
        "PRIYA",
        " Arun"
    ],
    "City": [
        "hyderabad",
        "Bangalore ",
        "Bengaluru",
        "HYDERABAD"
    ],
    "Age": [30, 25, 25, -5],
    "Sales": [
        "₹50,000",
        "₹70,000",
        "₹70,000",
        "₹30,000"
    ]
})

Step 1 — Clean text

df["Name"] = (

• df["Name"]
• .str.strip()
• .str.title()

)

Step 2 — Standardize city

df["City"] = (

• df["City"]
• .str.strip()
• .str.title()

)

df["City"] = df["City"].replace({

"Bangalore": "Bengaluru"

})

Step 3 — Convert age

df["Age"] = pd.to_numeric(

df["Age"],

errors="coerce"

)

Step 4 — Validate age

df.loc[
    ~df["Age"].between(0, 120),
    "Age"
] = pd.NA

Step 5 — Clean sales

df["Sales"] = (

• df["Sales"]
• .str.replace("₹", "", regex=False)
• .str.replace(",", "", regex=False)
• .astype(float)

)

Step 6 — Remove duplicate customer IDs

df = df.drop_duplicates(
    subset=["Customer_ID"]
)

The dataset is now much more consistent and suitable for analysis.

6.3.35Building a Reusable Cleaning Function

Instead of repeating cleaning logic

def clean_customer_data(df):
df = df.copy()

# Standardize names

df["Name"] = (

• df["Name"]
• .str.strip()
• .str.title()

)

# Standardize city

df["City"] = (

• df["City"]
• .str.strip()
• .str.title()

)

• # Standardize city names
• df["City"] = df["City"].replace({
• "Bangalore": "Bengaluru"

})

# Convert age

df["Age"] = pd.to_numeric(

df["Age"],

errors="coerce"

)

# Validate age

df.loc[
    ~df["Age"].between(0, 120),
    "Age"
] = pd.NA

return df

Then

clean_df = clean_customer_data(df)

This makes the process reusable.

6.3.36Data Cleaning Pipeline

A production-style pipeline may look like

def clean_data(df):
    return (
        df
        .drop_duplicates()
        .assign(
            Name=lambda x:
            x["Name"].str.strip().str.title(),
            City=lambda x:
            x["City"].str.strip().str.title()
        )
    )

For complex projects, separate functions are often better

6.3.37Column Name Cleaning

Raw datasets often contain

• Customer ID
• Customer Name
• Order Date
• Total Sales ($)

Standardize them

df.columns = (
    df.columns
    .str.strip()
    .str.lower()
    .str.replace(" ", "_")
    .str.replace(r"[^\w]", "", regex=True)
)

A cleaner approach when special characters need to be preserved carefully is to explicitly map important columns.

For example

df = df.rename(columns={
    "Customer ID": "customer_id",
    "Order Date": "order_date",
    "Total Sales ($)": "total_sales"
})

Explicit mappings are often safer for production pipelines.

6.3.38Data Cleaning and ETL

Data cleaning is a major part of ETL/ELT.

For example

• SQL Server
• Oracle
• Excel
• CSV

This is why data-cleaning knowledge is highly valuable for Data Analysts, Data Engineers, BI Developers, and ML Engineers.

6.3.39Data Cleaning vs Data Transformation

• These concepts are related but different.
• Data Cleaning
• Focuses on correcting data-quality problems.

Examples

• Remove duplicates
• Fix invalid dates
• Handle missing values
• Correct spelling
• Remove invalid records
• Data Transformation
• Changes the representation of valid data.

Examples

• Create revenue
• Calculate profit margin
• Convert currencies
• Create customer segments
• Normalize numerical values

Typical workflow

6.3.40Data Cleaning Checklist

Before considering a dataset ready for analysis, check

• Correct column names
• Correct data types
• Missing values identified
• Missing values handled appropriately
• Duplicate records checked
• Duplicate business keys checked
• Text standardized
• Leading/trailing spaces removed
• Categories standardized
• Numerical values validated
• Dates validated
• Impossible values identified
• Outliers investigated
• Business rules validated
• Data-quality report generated
• Final dataset validated

6.3.41Real-World Data Cleaning Example

Imagine an e-commerce dataset

• OrderID
• CustomerID
• OrderDate
• CustomerName
• City
• Product
• Quantity
• UnitPrice
• Discount
• Revenue

A professional cleaning process might be

1. Check row count

↓

2. Check column names

↓

3. Check data types

↓

4. Standardize column names

↓

5. Identify missing values

↓

6. Check duplicate OrderIDs

↓

7. Validate CustomerIDs

↓

8. Clean customer names

↓

9. Standardize cities

↓

10. Convert OrderDate

↓

11. Validate Quantity

↓

12. Validate UnitPrice

↓

13. Validate Discount

↓

14. Recalculate Revenue

↓

15. Compare calculated vs source Revenue

↓

16. Investigate anomalies

↓

17. Generate quality report

↓

18. Save clean dataset

This is much closer to how data cleaning works in real projects than simply calling dropna().

6.3.42Important Principle: Never Delete Data Blindly

A common beginner mistake is

df = df.dropna()

This may remove thousands of valuable records.

Similarly

df = df.drop_duplicates()

may remove legitimate repeated transactions.

And

df = df[df["Sales"] < upper_limit]

may remove genuine high-value transactions.

The correct process is

Every major cleaning decision should have a business or technical reason.

6.3.43Interview Questions

• Basic
• What is data cleaning?
• Why is data cleaning important?
• What are common data-quality problems?
• How do you identify missing values?
• How do you identify duplicate records?
• How do you remove duplicates?
• How do you convert a string column to numeric?
• How do you convert a column to datetime?
• How do you standardize text?
• What is data validation?

Advanced

• How would you clean a dataset with millions of rows?
• How do you distinguish a duplicate transaction from a legitimate repeated transaction?
• How would you handle an invalid age such as -10?
• How would you standardize inconsistent city names?
• How would you handle invalid dates?
• How would you validate business rules using Pandas?
• How would you create an automated data-quality report?
• How would you build a reusable data-cleaning pipeline?
• What is the difference between data cleaning and data transformation?
• Why shouldn't all missing values simply be removed?
• Why shouldn't all outliers be removed?
• How would you compare source revenue with calculated revenue?
• How would you clean a dataset containing ₹, commas, and percentage symbols?
• How would you process a CSV larger than available memory?
• How would you monitor data quality in a production ETL pipeline?

6.3.44Mini Project — Customer Data Quality Pipeline

Create a project using a messy customer dataset containing

• Customer_ID
• Customer_Name
• Email
• Phone
• City
• Age
• Registration_Date
• Total_Purchases

Your pipeline should perform

Expected output

Produce three datasets

1. Clean Data

Contains records suitable for analysis.

2. Rejected Data

Contains records that failed critical validation rules.

3. Data Quality Report

Contains

• Column
• Data Type
• Total Records
• Missing Count
• Missing %
• Unique Count
• Invalid Count
• Duplicate Count

This approach is much closer to a production data-quality pipeline than simply cleaning a DataFrame interactively.

6.3.45Key Takeaways

The complete data-cleaning process can be summarized as

RAW DATA

│

▼

┌───────────────┐

│ PROFILE │

│ DATA │

└───────┬───────┘

│

┌─────────┼─────────┐

▼ ▼ ▼

Missing Duplicate Invalid

Values Data Values

│ │ │

└─────────┼─────────┘

▼

STANDARDIZATION

│

▼

TYPE CONVERSION

│

▼

BUSINESS VALIDATION

│

▼

QUALITY CHECKS

│

▼

CLEAN DATA

│

▼

ANALYSIS / BI / ML

• Five concepts to master

For practical Data Analyst and Data Engineer work, focus especially on

• Missing-value detection and treatment
• Duplicate and business-key validation
• Data-type and date conversion
• Text/category standardization
• Business-rule and data-quality validation

The ultimate goal of data cleaning is not simply to make a dataset look clean. The goal is to make the data trustworthy enough that the decisions made from it can also be trusted.

Missing values are observations where a particular field does not contain a usable value. They are one of the most common data-quality problems in real-world datasets.

Missing data can occur because

• A customer did not provide information.
• A system failed to capture a value.
• A field was not applicable.
• Data was lost during ETL.
• Different source systems use different representations.
• A value was intentionally hidden or removed.
• A data-entry error occurred.

Handling missing values correctly is important because careless treatment can introduce bias, incorrect statistics, and misleading business conclusions.

6.4.1Learning Objectives

By the end of this topic, you should be able to

• Identify missing values using Pandas and NumPy.
• Understand different representations of missing data.
• Measure missing-value percentages.
• Distinguish different types of missingness.
• Remove missing records appropriately.
• Fill missing values using statistical methods.
• Use forward and backward filling.
• Interpolate missing time-series values.
• Handle missing categorical data.
• Handle missing dates.
• Create missing-value indicators.
• Validate imputed data.
• Choose an appropriate missing-value strategy.

6.4.2What Is a Missing Value?

Consider

Here

Arun's Age is missing.

Meena's City is missing.

In Pandas, missing numerical values are commonly represented using NaN, while modern Pandas also supports pd.NA.

6.4.3Common Representations of Missing Data

Missing data may appear as

NaN

None

pd.NA

• NaT
• NULL
• N/A
• NA
• Unknown

?

-

Blank

Empty string

Not all of these are automatically recognized as missing.

For example

data = pd.Series([
    "Ravi",
    "Priya",
    "NA",
    "Arun"
])

print(data.isna())

The string "NA" may be treated as ordinary text depending on how the data was loaded.

6.4.4Creating Missing Values

Using NumPy

import numpy as np
import pandas as pd
data = pd.Series([
    100,
    200,
    np.nan,
    400
])

print(data)

Using Pandas

data = pd.Series([
    100,
    200,
    pd.NA,
    400
])

For datetime

dates = pd.Series([
    "2025-01-01",
    None,
    "2025-03-01"
])

dates = pd.to_datetime(dates)

The missing date becomes NaT (Not a Time).

6.4.5Detecting Missing Values

The most important function is

df.isna()

Example

df = pd.DataFrame({
    "Name": ["Ravi", "Priya", "Arun"],
    "Age": [30, np.nan, 35],
    "City": ["Hyderabad", "Bengaluru", np.nan]
})

print(df.isna())

6.4.6Counting Missing Values

Use

df.isna().sum()

Example result

Name 0

Age 1

City 1

This gives the number of missing values per column.

6.4.7Missing Value Percentage

Count alone isn't enough.

For example

missing_percentage = (
    df.isna().mean() * 100
)

print(missing_percentage)

You might get

Name 0.0

Age 33.3

City 33.3

This allows you to compare missingness across columns.

6.4.8Total Missing Values

total_missing = df.isna().sum().sum()
print(total_missing)

This returns the total number of missing cells in the DataFrame.

6.4.9Finding Rows Containing Missing Values

missing_rows = df[
    df.isna().any(axis=1)
]

print(missing_rows)

This returns every row containing at least one missing value.

6.4.10Finding Rows Without Missing Values

complete_rows = df[
    df.notna().all(axis=1)
]

This returns rows where every column contains a value.

6.4.11Checking a Specific Column

df[df["Age"].isna()]

This returns records where Age is missing.

For non-missing values

df[df["Age"].notna()]

6.4.12Missing Values During CSV Loading

Pandas can recognize common missing-value representations when reading files.

df = pd.read_csv(
    "customers.csv",
    na_values=[
        "NA",
        "N/A",
        "Unknown",
        "-"
    ]
)

Now those values can be treated as missing.

You can also specify multiple values

missing_values = [
    "",
    " ",
    "NA",
    "N/A",
    "NULL",
    "Unknown",
    "-"
]

df = pd.read_csv(
    "customers.csv",
    na_values=missing_values
)

6.4.13Missingness Patterns

Not all missing data occurs for the same reason.

Three important concepts are

MCAR — Missing Completely At Random

Missingness has no relationship with observed or unobserved data.

Example

• A sensor randomly fails occasionally.
• MAR — Missing At Random
• Missingness is related to other observed variables.

Example

• Income is more frequently missing for customers in a particular age group.
• MNAR — Missing Not At Random
• Missingness is related to the missing value itself.

Example

People with very high incomes may be less willing to report their income.

These concepts matter because the appropriate treatment can differ depending on why the data is missing.

6.4.14Removing Missing Rows

The simplest approach

df_clean = df.dropna()

This removes any row containing a missing value.

Example

Original

After

df.dropna()

Only Ravi remains.

Problem

You may lose a large amount of useful information.

Therefore, dropna() should not automatically be the default solution.

6.4.15Removing Rows Based on Specific Columns

Suppose Customer_ID and Order_ID are mandatory.

df = df.dropna(
    subset=["Customer_ID", "Order_ID"]
)

This is usually better than removing a row because an unrelated optional field is missing.

6.4.16Removing Columns with Too Many Missing Values

Suppose

df.isna().mean() * 100

returns

Name 2%

Age 5%

Phone 10%

Comments 85%

If Comments is not important and 85% of its values are missing, you may decide to remove it:

df = df.drop(
    columns=["Comments"]
)

This should be a business decision, not an automatic rule.

6.4.17Using thresh

thresh allows you to retain rows containing a minimum number of non-missing values.

df = df.dropna(
    thresh=5
)

This keeps rows with at least 5 non-null values.

For example, if a dataset has 7 columns, a row with only 4 valid values would be removed.

6.4.18Filling Missing Values

Instead of deleting missing values, you can replace them.

The function is

df.fillna(value)

Example

df["Age"] = df["Age"].fillna(0)

But replacing age with 0 may be inappropriate.

The correct replacement depends on the meaning of the data.

6.4.19Mean Imputation

Suppose

ages = pd.Series([
    25, 30, 35, np.nan, 40
])

Calculate mean

mean_age = ages.mean()
print(mean_age)

Fill

ages = ages.fillna(mean_age)

Advantages

• Simple
• Fast
• Easy to understand

Disadvantages

• Reduces natural variation
• Can distort distributions
• Sensitive to outliers

Mean imputation is generally more appropriate when the distribution is reasonably symmetric and the missingness mechanism supports the assumption.

6.4.20Median Imputation

Median is often preferable when data is skewed.

median_salary = df["Salary"].median()

df["Salary"] = df["Salary"].fillna(

median_salary

)

For example, salaries may contain a few very high values.

The median is less affected by extreme values than the mean.

6.4.21Mean vs Median

Suppose salaries are

• 30,000
• 35,000
• 40,000
• 45,000
• 500,000

The mean is heavily influenced by 500,000.

The median is much more representative of the typical salary.

Therefore

Symmetric distribution → Mean may work well

Skewed distribution → Median is often safer

6.4.22Mode Imputation

For categorical data

mode_city = df["City"].mode()[0]

df["City"] = df["City"].fillna(

mode_city

)

Example

• Hyderabad
• Hyderabad
• Bengaluru
• Hyderabad
• NaN

The mode is

Hyderabad

So the missing city would be filled with Hyderabad.

Limitation

If missingness is not representative of the most common category, mode imputation can introduce bias.

6.4.23Filling with a Constant

• Sometimes a meaningful category can be used.
• df["Department"] = (
• df["Department"]
• .fillna("Unknown")

)

For a numeric field

• df["Sales"] = df["Sales"].fillna(0)
• But 0 should only be used if zero has a valid business meaning.
• Missing ≠ zero.

This is an extremely important rule.

6.4.24Forward Fill

Forward fill uses the previous available value.

df["Sales"] = df["Sales"].ffill()

Example

• 100
• 120
• NaN
• 150
• NaN
• 200

After forward fill

• 100
• 120
• 120
• 150
• 150
• 200

This is particularly useful for

• time-series data
• sensor readings
• configuration values
• status information

when carrying the previous value forward is logically valid.

6.4.25Backward Fill

Backward fill uses the next available value.

df["Sales"] = df["Sales"].bfill()

Example

• 100
• NaN
• 120
• NaN
• 150

After backward fill

• 100
• 120
• 120
• 150
• 150

6.4.26Forward vs Backward Fill

For time series

January 100

February 120

March NaN

April 160

Forward fill

March = 120

Backward fill

March = 160

Neither is automatically correct. The business meaning determines the appropriate approach.

6.4.27Filling Within Groups

Suppose sales data contains multiple stores

df = pd.DataFrame({
    "Store": [
        "A", "A", "A",
        "B", "B", "B"
    ],
    "Sales": [
        100, np.nan, 150,
        200, np.nan, 300
    ]
})

Forward fill within each store

df["Sales"] = (

df.groupby("Store")["Sales"]

.ffill()

)

This is much better than filling across unrelated stores.

6.4.28Group-Based Mean Imputation

Suppose salary varies significantly by department.

Instead of using one overall mean

df["Salary"] = df["Salary"].fillna(

df["Salary"].mean()

)

use department-specific means

df["Salary"] = (

df.groupby("Department")["Salary"]

.transform(

lambda x: x.fillna(x.mean())

)

)

This can preserve differences between groups.

6.4.29Group-Based Median Imputation

df["Salary"] = (

df.groupby("Department")["Salary"]

.transform(

lambda x: x.fillna(x.median())

)

)

This can be particularly useful when salary distributions differ across departments.

6.4.30Interpolation

Interpolation estimates missing values based on surrounding observations.

Example

sales = pd.Series([
    100,
    120,
    np.nan,
    160,
    180
])

sales = sales.interpolate()
print(sales)

The missing value may become

140

because it lies between 120 and 160.

6.4.31Linear Interpolation

Linear interpolation is the default method in many simple cases

df["Sales"] = df["Sales"].interpolate(

method="linear"

)

This is useful for continuous measurements such as

• temperature
• electricity consumption
• stock-related measurements
• sensor readings

provided interpolation is meaningful for the underlying process.

6.4.32Time-Based Interpolation

For time-series data

df["Date"] = pd.to_datetime(df["Date"])

df = df.sort_values("Date")
df = df.set_index("Date")

df["Sales"] = df["Sales"].interpolate(

method="time"

)

Time-based interpolation considers the actual spacing between dates.

6.4.33Limit Interpolation

You may not want to fill an unlimited number of consecutive missing values.

df["Sales"] = df["Sales"].interpolate(

limit=2

)

This limits interpolation to a maximum number of consecutive missing values.

6.4.34Missing Value Indicators

Sometimes the fact that a value was missing is itself meaningful.

Create an indicator

df["Age_Missing"] = (

df["Age"].isna().astype(int)

)

Result

Age Age_Missing

30 0

NaN 1

40 0

This tells a model or analyst

0 → originally present

1 → originally missing

This can be useful in machine-learning workflows.

6.4.35Missing Values in Categorical Columns

Suppose

• Department
• IT
• HR
• NaN
• Finance

Possible strategies

• Strategy 1 — Use "Unknown"
• df["Department"] = (
• df["Department"]
• .fillna("Unknown")

)

• Strategy 2 — Use the mode
• df["Department"] = (
• df["Department"]
• .fillna(
• df["Department"].mode()[0]

)

)

Strategy 3 — Investigate the source

If department is a mandatory field, missing values may indicate a source-system problem rather than something that should simply be imputed.

6.4.36Missing Dates

Suppose

df["OrderDate"].isna().sum()

returns 100 missing dates.

Possible options

• Retrieve the date from the source system.
• Use another related date.
• Remove the record if date is mandatory.
• Create an "Unknown" date category for reporting.
• Impute only when there is a defensible business rule.
• Do not automatically replace missing dates with today's date.
• That can create completely incorrect analytics.

6.4.37Missing Values in Financial Data

Consider

Customer | Revenue

A | 10000

B | NaN

C | 15000

Should NaN become zero?

Not necessarily.

Possible interpretations

Interpretation 1

Customer had zero revenue.

Then

• fillna(0)
• may be valid.
• Interpretation 2
• Revenue was not captured.

Then

Missing ≠ Zero

You should investigate the source.

This distinction is extremely important in financial and business reporting.

6.4.38Missing Values in Time Series

Consider

df = pd.DataFrame({
    "Date": pd.date_range(
        "2025-01-01",
        periods=7
    ),
    "Power": [
        100, 110, np.nan,
        130, 140, np.nan, 160
    ]
})

A simple interpolation

df["Power"] = df["Power"].interpolate()

However, for real operational data, you should consider

• sensor behavior
• seasonality
• operating hours
• shutdown periods
• expected business patterns
• data-collection failures

Blind interpolation can hide a genuine operational problem.

6.4.39Missing Values and Data Leakage

In machine learning, imputation must be performed carefully.

Suppose you split your data

Training Data

Testing Data

You should calculate the training-set mean or median only from the training data.

Incorrect

This allows information from the test set to influence the training process.

Better

↓

Apply the same values to test data

This is called avoiding data leakage.

6.4.40Using Scikit-Learn for Imputation

For machine-learning pipelines, Scikit-learn provides imputers.

Example

from sklearn.impute import SimpleImputer
imputer = SimpleImputer(
    strategy="median"
)

X = imputer.fit_transform(X)

Strategies include

• mean
• median
• most_frequent
• constant

For example

imputer = SimpleImputer(
    strategy="most_frequent"
)

is suitable for some categorical features.

6.4.41Advanced Imputation Methods

For more complex problems, you may encounter

KNN Imputation

Uses nearby observations.

from sklearn.impute import KNNImputer
imputer = KNNImputer(
    n_neighbors=5
)

Iterative Imputation

Estimates missing values using relationships among other features.

These approaches can be more sophisticated but also introduce assumptions and computational cost.

6.4.42Comparing Missing-Value Strategies

6.4.43Missing Data Quality Report

A professional data-quality report should include

report = pd.DataFrame({
    "Column": df.columns,
    "Total_Rows": len(df),
    "Missing_Count": [
        df[col].isna().sum()
        for col in df.columns
    ],
    "Missing_Percentage": [
        df[col].isna().mean() * 100
        for col in df.columns
    ],
    "Non_Missing_Count": [
        df[col].notna().sum()
        for col in df.columns
    ]
})

print(report)

Example output

This immediately tells the data team where the biggest problems are.

6.4.44Practical Example — Customer Dataset

Consider

import pandas as pd
import numpy as np
df = pd.DataFrame({
    "Customer": [
        "Ravi", "Priya", "Arun",
        "Meena", "John"
    ],
    "Age": [
        30, np.nan, 35,
        40, np.nan
    ],
    "City": [
        "Hyderabad",
        "Bengaluru",
        np.nan,
        "Hyderabad",
        "Hyderabad"
    ],
    "Salary": [
        50000,
        60000,
        np.nan,
        70000,
        80000
    ]
})

Step 1 — Profile missing values

print(df.isna().sum())

Step 2 — Calculate percentages

print(
    df.isna().mean() * 100
)

• Step 3 — Fill Age using median
• df["Age"] = df["Age"].fillna(
• df["Age"].median()

)

• Step 4 — Fill Salary using median
• df["Salary"] = df["Salary"].fillna(
• df["Salary"].median()

)

• Step 5 — Fill City with "Unknown"
• df["City"] = df["City"].fillna(
• "Unknown"

)

Step 6 — Verify

print(df.isna().sum())

Expected

Customer 0

Age 0

City 0

Salary 0

But remember: zero missing values does not automatically mean high-quality data. The imputation strategy itself must be appropriate.

6.4.45Advanced Example — Group-Based Imputation

Suppose

df = pd.DataFrame({
    "Department": [
        "IT", "IT", "IT",
        "HR", "HR", "HR"
    ],
    "Salary": [
        80000, np.nan, 90000,
        50000, np.nan, 55000
    ]
})

Using the overall median

df["Salary"] = df["Salary"].fillna(

df["Salary"].median()

)

may not preserve the difference between IT and HR salaries.

Instead

df["Salary"] = (

df.groupby("Department")["Salary"]

.transform(

lambda x: x.fillna(x.median())

)

)

Now missing salaries are estimated using the employee's department.

This is often a more meaningful strategy.

6.4.46Advanced Example — Time Series

Consider electricity consumption

df = pd.DataFrame({
    "Date": pd.date_range(
        "2025-01-01",
        periods=10
    ),
    "Consumption": [
        100, 110, 120,
        np.nan, 140, 150,
        np.nan, 170, 180, 190
    ]
})

Convert the date

df["Date"] = pd.to_datetime(

df["Date"]

)

Sort

df = df.sort_values("Date")

Interpolate

• df["Consumption"] = (
• df["Consumption"]
• .interpolate()

)

Now the missing observations are estimated based on neighboring values.

6.4.47Best Practices

1. Understand why data is missing

Don't immediately fill it.

2. Analyze missingness before cleaning

Calculate

df.isna().sum()

and

df.isna().mean() * 100

3. Use domain knowledge

A missing Revenue field may mean something very different from a missing Middle_Name.

4. Don't confuse missing with zero

NaN ≠ 0

unless the business definition explicitly says so.

5. Preserve an audit trail

Record

• which values were missing
• how they were handled
• how many records were affected
• why the chosen method was used

6. Validate after imputation

Always compare

Check whether the distribution changed dramatically.

6.4.48Interview Questions

• Basic
• What is a missing value?
• How do you identify missing values in Pandas?
• What is the difference between NaN, None, pd.NA, and NaT?
• How do you count missing values?
• How do you calculate missing-value percentages?
• How do you remove rows containing missing values?
• How do you fill missing values?
• What is forward fill?
• What is backward fill?
• What is interpolation?

Advanced

• What is the difference between MCAR, MAR, and MNAR?
• When would you use mean versus median imputation?
• Why can mean imputation be problematic?
• When should you use group-based imputation?
• How would you handle missing values in a time series?
• Why shouldn't missing values always be replaced with zero?
• How can missingness itself be used as a feature?
• What is data leakage during imputation?
• How does SimpleImputer work?
• What is KNN imputation?
• How would you handle a column with 90% missing values?
• How would you validate an imputation strategy?
• How would you handle missing values differently for numerical and categorical columns?
• How would you handle missing values in a production ETL pipeline?
• How would you determine whether missing data indicates a source-system defect?

6.4.49Mini Project — Missing Value Analysis

Create a customer dataset containing

• Customer_ID
• Age
• Gender
• City
• Income
• Order_Count
• Last_Order_Date
• Intentionally introduce missing values.

Your project should

Step 1

Generate a missing-value report.

Step 2

Calculate missing percentage for every column.

Step 3

Identify columns with more than 30% missing values.

Step 4

Remove records where Customer_ID is missing.

Step 5

Use median imputation for Income.

Step 6

Use group-based median imputation for Income by City.

Step 7

Use mode or "Unknown" for Gender.

Step 8

Use a business-appropriate strategy for missing Last_Order_Date.

Step 9

Create missing-value indicator columns.

Step 10

Compare the dataset before and after treatment.

6.4.50Key Takeaways

The missing-value workflow is

RAW DATA

│

▼

IDENTIFY MISSING VALUES

│

▼

MEASURE MISSINGNESS

│

▼

UNDERSTAND THE CAUSE

│

┌─────────┼─────────┐

▼ ▼ ▼

REMOVE IMPUTE RETAIN

│ │ │

└─────────┼─────────┘

▼

VALIDATE RESULT

│

▼

CLEAN DATA

• Most important concepts

Focus especially on

• isna() / notna()
• Missing-value percentages
• dropna() vs fillna()
• Mean, median, and mode imputation
• ffill() / bfill()
• Interpolation
• Group-based imputation
• Missing-value indicators
• MCAR, MAR, and MNAR
• Avoiding data leakage during imputation

The most important principle is

There is no universally correct way to handle missing values. The right approach depends on why the value is missing, what the field represents, and how the cleaned data will be used.

Outlier detection is the process of identifying observations that are unusually different from the majority of the data.

For example, suppose monthly customer purchases are

• ₹1,200
• ₹1,500
• ₹1,800
• ₹1,700
• ₹1,600
• ₹95,000

₹95,000 is an unusual observation compared with the other values.

However, an outlier is not automatically an error. It could represent

• Data-entry error
• Fraud
• System malfunction
• Exceptional customer
• Large business transaction
• Seasonal event
• Genuine extreme observation

Therefore, the objective is not simply to remove outliers, but to detect, investigate, and decide how they should be treated.

6.5.1Learning Objectives

By the end of this topic, you should be able to

• Understand what an outlier is.
• Distinguish outliers from normal variation.
• Detect outliers using statistical methods.
• Use the IQR method.
• Use Z-score detection.
• Use percentile-based detection.
• Use Boolean filtering in Pandas.
• Visualize outliers with box plots and scatter plots.
• Detect outliers within groups.
• Understand the effect of outliers on statistics.
• Decide whether to remove, cap, transform, or retain outliers.
• Build an outlier-detection workflow.

6.5.2What Is an Outlier?

An outlier is an observation that lies unusually far from the typical pattern of a dataset.

Example

• 10
• 12
• 11
• 13
• 15
• 14
• 12
• 100

Here

100

is substantially different from the other values.

But consider

• 10
• 12
• 11
• 13
• 15
• 14
• 12
• 20

20 is larger, but it may not be sufficiently unusual to be considered an outlier.

Therefore, outlier detection requires a defined method.

6.5.3Types of Outliers

There are several ways to classify outliers.

1. Global Outlier

An observation that is unusual compared with the entire dataset.

Example

Sales = ₹10,000,000

when almost all other sales are below ₹100,000.

2. Local Outlier

An observation may be normal globally but unusual within a specific group.

Example

100 may not look extreme globally, but it is unusual within the North region.

3. Contextual Outlier

An observation is unusual only in a particular context.

Example

Electricity consumption

Monday 2 PM → 500 KW

Monday 3 PM → 510 KW

Monday 4 PM → 505 KW

But

Sunday 2 AM → 500 KW

may be unusual because the expected consumption at that time is much lower.

4. Multivariate Outlier

A value may look normal individually but be unusual when multiple variables are considered together.

Example

Age = 25
Income = ₹20 lakh

Neither value may be extreme individually, but their combination may be unusual for the population being analyzed.

6.5.4Why Outliers Matter

Outliers can strongly influence

• Mean
• Standard deviation
• Regression models
• Correlation
• Machine-learning algorithms
• Forecasting
• Business KPIs

Consider

• 10
• 20
• 30
• 40
• 50
• 1000

The mean becomes much larger because of 1000.

The median is much less affected.

This is why analysts often compare

df["Sales"].mean()

df["Sales"].median()

6.5.5Basic Data Inspection

Start with

import pandas as pd
import numpy as np
df = pd.DataFrame({
    "Sales": [
        100, 120, 130, 110,
        125, 140, 135, 5000
    ]
})

print(df.describe())

Look at

• minimum
• maximum
• mean
• median
• standard deviation
• quartiles

A very large difference between the median and maximum can indicate potential outliers.

6.5.6Visual Detection

Visualization is one of the easiest ways to understand outliers.

Important charts include

• Box plot
• Histogram
• Scatter plot
• Time-series plot

Using Matplotlib

import matplotlib.pyplot as plt
plt.boxplot(df["Sales"])
plt.title("Sales Distribution")
plt.show()

A box plot displays

Minimum

│

├── Q1

│

├── Median

│

├── Q3

│

└── Maximum

Values far outside the whiskers are potential outliers.

6.5.7Box Plot and IQR

The Interquartile Range (IQR) is one of the most widely used methods for outlier detection.

Define

IQR = Q3 - Q1

where

Q1 = 25th percentile
Q3 = 75th percentile

The standard boundaries are

• Lower Bound = Q1 - 1.5 × IQR
• Upper Bound = Q3 + 1.5 × IQR
• Values outside these boundaries are considered potential outliers.

6.5.8IQR Method in Python

Suppose

sales = pd.Series([
    100, 120, 130, 110,
    125, 140, 135, 5000
])

Calculate Q1

Q1 = sales.quantile(0.25)

Calculate Q3

Q3 = sales.quantile(0.75)

Calculate IQR

IQR = Q3 - Q1

Calculate boundaries

lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQR

Find outliers

outliers = sales[
    (sales < lower_bound) |
    (sales > upper_bound)
]

print(outliers)

6.5.9Complete IQR Function

Instead of repeating the calculation, create a function

def detect_iqr_outliers(series):
    Q1 = series.quantile(0.25)
    Q3 = series.quantile(0.75)
    IQR = Q3 - Q1
    lower = Q1 - 1.5 * IQR
    upper = Q3 + 1.5 * IQR
    return series[
        (series < lower) |
        (series > upper)
    ]

Use it

outliers = detect_iqr_outliers(
    df["Sales"]
)

print(outliers)

6.5.10Understanding Quartiles

Suppose sorted data is

• 10
• 20
• 30
• 40
• 50
• 60
• 70
• 80
• 90
• 100

Quartiles divide the data approximately into four sections

Q1 Q2 Q3

│ │ │

--------|-----------|-----------|--------

25% 50% 75%

Where

Q1 = 25th percentile
Q2 = Median
Q3 = 75th percentile

The IQR covers the middle 50% of the observations.

6.5.11Why IQR Is Useful

IQR is popular because it is relatively robust against extreme values.

For example

• 10
• 11
• 12
• 13
• 14
• 10000

The extreme value 10000 has a strong impact on the mean and standard deviation, but the quartiles are much less affected.

Therefore, IQR is often a good first method for skewed business data.

6.5.12Z-Score Method

Another common method is the Z-score.

The Z-score measures how many standard deviations an observation is from the mean.

Formula

Z = (X - μ) / σ

where

X = observation

μ = mean

σ = standard deviation

For example

Z = 3

means the observation is approximately 3 standard deviations above the mean.

6.5.13Z-Score in Python

Using SciPy

from scipy.stats import zscore

df["Z_Score"] = zscore(

df["Sales"]

)

Find potential outliers

outliers = df[
    df["Z_Score"].abs() > 3
]

A common rule is

|Z| > 3 → potential outlier

But the threshold is not universal.

6.5.14Manual Z-Score Calculation

You can calculate it without SciPy

mean = df["Sales"].mean()
std = df["Sales"].std()

df["Z_Score"] = (

(df["Sales"] - mean) / std

)

Then

outliers = df[
    df["Z_Score"].abs() > 3
]

6.5.15IQR vs Z-Score

General guideline

• Skewed / non-normal data → IQR
• Approximately normal data → Z-score
• But always consider the business context.

6.5.16Percentile-Based Detection

Another simple method is to define acceptable percentiles.

For example

lower = df["Sales"].quantile(0.01)
upper = df["Sales"].quantile(0.99)

Find observations outside the range

outliers = df[
    (df["Sales"] < lower) |
    (df["Sales"] > upper)
]

This identifies approximately the lowest 1% and highest 1%.

This approach is useful when business rules naturally focus on extreme tails.

6.5.17Winsorization

Instead of removing extreme values, you can cap them.

Suppose

lower = df["Sales"].quantile(0.01)
upper = df["Sales"].quantile(0.99)

Then

• df["Sales_Capped"] = (
• df["Sales"]
• .clip(lower, upper)

)

Values below the lower limit become the lower limit, and values above the upper limit become the upper limit.

This is called winsorization/capping.

6.5.18Removing Outliers

If an outlier is confirmed to be erroneous

df_clean = df[
    (df["Sales"] >= lower_bound) &
    (df["Sales"] <= upper_bound)
]

But don't do this automatically.

For example

• ₹10,000
• ₹12,000
• ₹11,000
• ₹15,000
• ₹1,000,000

The ₹1,000,000 transaction could be a genuine enterprise customer order.

Removing it would distort the business analysis.

6.5.19Outlier Treatment Strategies

• Once an outlier is detected, you have several options.
• Option 1 — Keep it
• Use this when it represents a genuine observation.
• Option 2 — Remove it
• Use this when it is confirmed to be invalid.
• Option 3 — Correct it
• If the source value is incorrect and the correct value can be established.
• Option 4 — Cap it
• Replace extreme values with percentile limits.
• Option 5 — Transform it

For example, use

np.log1p(df["Sales"])

Option 6 — Analyze separately

For example

Normal customers

VIP customers

Instead of treating them as one population.

6.5.20Log Transformation

Highly skewed data can sometimes be transformed using logarithms.

Example

df["Log_Sales"] = np.log1p(

df["Sales"]

)

log1p(x) calculates

log(1 + x)

and is useful when zero values are present.

Log transformation can reduce the influence of extremely large values.

6.5.21Outlier Detection by Group

This is an important advanced concept.

Suppose

df = pd.DataFrame({
    "Region": [
        "South", "South", "South",
        "North", "North", "North"
    ],
    "Sales": [
        100, 110, 105,
        10000, 11000, 50
    ]
})

• 50 might be a local outlier in the North region.
• A global analysis might not identify it correctly.
• Use group-specific IQR rules.

6.5.22Group-Based IQR

def mark_outliers(group):
Q1 = group["Sales"].quantile(0.25)
Q3 = group["Sales"].quantile(0.75)
IQR = Q3 - Q1
lower = Q1 - 1.5 * IQR
upper = Q3 + 1.5 * IQR

• group["Outlier"] = (
• (group["Sales"] < lower) |
• (group["Sales"] > upper)

)

return group

Apply

df = (
    df.groupby("Region", group_keys=False)
    .apply(mark_outliers)
)

This detects outliers within each region.

6.5.23Outliers in Time Series

Time-series outliers require special attention.

Consider

• Jan → 100
• Feb → 105
• Mar → 110
• Apr → 108
• May → 500
• Jun → 115
• May could be an outlier.

But perhaps May represents

• Festival demand
• Marketing campaign
• Product launch
• System issue
• Seasonal event

A simple IQR calculation may detect May but cannot explain why it occurred.

Time-series analysis should consider

• trend
• seasonality
• calendar events
• business events
• historical patterns

6.5.24Rolling Outlier Detection

• For time-series data, rolling statistics can be useful.
• df["RollingMean"] = (
• df["Sales"]
• .rolling(7)
• .mean()

)

• df["RollingStd"] = (
• df["Sales"]
• .rolling(7)

.std()

)

Calculate a rolling upper threshold

df["Upper"] = (

df["RollingMean"] +

3 * df["RollingStd"]

)

Then

df["Outlier"] = (

df["Sales"] > df["Upper"]

)

This adapts the threshold based on recent observations.

6.5.25Scatter Plot for Multivariate Outliers

Suppose

plt.scatter(
    df["Income"],
    df["Spending"]
)

plt.xlabel("Income")
plt.ylabel("Spending")
plt.title("Income vs Spending")
plt.show()

An observation far away from the main cluster may be a potential multivariate outlier.

6.5.26Correlation and Outliers

Outliers can dramatically affect correlation.

Suppose

• X: 10, 20, 30, 40, 50
• Y: 12, 22, 32, 42, 52
• The relationship is strongly positive.

But one extreme point

X = 1000
Y = 10

can dramatically change the correlation.

Therefore, always inspect scatter plots when analyzing relationships between variables.

6.5.27Effect on Mean and Median

Consider

data = pd.Series([
    10, 12, 14, 15, 16, 1000
])

print("Mean:", data.mean())
print("Median:", data.median())

• The mean is pulled toward the extreme value.
• The median remains much more stable.
• This demonstrates why median is often preferred for highly skewed data.

6.5.28Effect on Standard Deviation

Standard deviation is also highly affected by extreme observations.

Without outlier

• 10
• 12
• 14
• 15
• 16

With outlier

• 10
• 12
• 14
• 15
• 16
• 1000
• The standard deviation increases substantially.

This is one reason why the Z-score method can be affected by extreme observations.

6.5.29Robust Statistics

Robust statistics are less sensitive to extreme observations.

Examples

• Median
• IQR
• MAD
• MAD means Median Absolute Deviation.

Conceptually

MAD = median(
    |x - median(x)|
)

A robust z-score can be based on MAD rather than the standard deviation.

This can be useful when the dataset itself contains strong outliers.

6.5.30MAD Example

median = df["Sales"].median()
mad = (
    df["Sales"] - median
).abs().median()

robust_z = (
    0.6745 *
    (df["Sales"] - median) /
    mad
)

Potential outliers can then be identified using a threshold such as

outliers = df[
    robust_z.abs() > 3.5
]

The exact threshold should be chosen based on the analytical context.

6.5.31Outlier Detection with Isolation Forest

For more advanced machine-learning workflows, Isolation Forest can detect unusual observations.

from sklearn.ensemble import IsolationForest
model = IsolationForest(
    contamination=0.05,
    random_state=42
)

df["Outlier"] = model.fit_predict(

df[["Sales"]]

)

The result is typically

1 → normal

-1 → potential outlier

For multiple variables

features = [
    "Income",
    "Spending",
    "Purchase_Count"
]

df["Outlier"] = model.fit_predict(

df[features]

)

This can identify multivariate anomalies that simple one-variable rules may miss.

6.5.32IQR vs Z-Score vs Isolation Forest

No single method is universally best.

6.5.33Practical Example — Employee Salary

Suppose

df = pd.DataFrame({
    "Employee": [
        "A", "B", "C", "D",
        "E", "F", "G"
    ],
    "Salary": [
        50000, 55000, 60000,
        58000, 62000, 65000,
        500000
    ]
})

Step 1 — Calculate quartiles

Q1 = df["Salary"].quantile(0.25)
Q3 = df["Salary"].quantile(0.75)

Step 2 — Calculate IQR

IQR = Q3 - Q1

Step 3 — Calculate bounds

lower = Q1 - 1.5 * IQR
upper = Q3 + 1.5 * IQR

• Step 4 — Mark outliers
• df["Outlier"] = (
• (df["Salary"] < lower) |
• (df["Salary"] > upper)

)

Step 5 — Investigate

print(
    df[df["Outlier"]]
)

If employee G is actually a CEO, removing the record would be incorrect.

6.5.34Practical Example — Sales Transactions

Suppose a company has

df = pd.DataFrame({
    "OrderID": range(1, 11),
    "Sales": [
        1000, 1200, 900,
        1500, 1300, 1100,
        1250, 1400, 10000,
        1150
    ]
})

Detect IQR outliers

Q1 = df["Sales"].quantile(0.25)
Q3 = df["Sales"].quantile(0.75)
IQR = Q3 - Q1
lower = Q1 - 1.5 * IQR
upper = Q3 + 1.5 * IQR

• df["Outlier"] = (
• (df["Sales"] < lower) |
• (df["Sales"] > upper)

)

Then investigate

print(
    df[df["Outlier"]]
)

Instead of immediately deleting the transaction, check the source system.

6.5.35Outlier Investigation Workflow

A professional process looks like

RAW DATA

│

▼

Detect Potential

Outliers

│

▼

Statistical Method

│

▼

Business Validation

│

┌──────────┼──────────┐

▼ ▼ ▼

Genuine Error Unknown

│ │ │

▼ ▼ ▼

Keep Correct Investigate

│

▼

Validate

│

▼

Final Dataset

6.5.36Never Automatically Delete Outliers

This is perhaps the most important lesson.

Suppose your sales data contains

• ₹1,000
• ₹1,200
• ₹900
• ₹1,100
• ₹1,300
• ₹500,000

The last value may be flagged as an outlier.

But it could be

• a bulk corporate order
• an annual contract
• a genuine high-value customer

If you remove it, you could significantly understate total revenue.

Therefore

Outlier detection identifies observations for investigation; it does not automatically prove that they are wrong.

6.5.37Data Quality Rules for Outliers

A production pipeline can define rules such as

Sales < 0 → Invalid

Quantity < 0 → Invalid

Discount > 100% → Invalid

Age < 0 → Invalid

Age > 120 → Suspicious

Transaction > P99 → Investigate

Then classify records

• VALID
• SUSPICIOUS
• INVALID

This is often better than simply deleting all extreme values.

6.5.38Outlier Report

A useful outlier report can contain

You can create this programmatically

def iqr_summary(df, column):
    Q1 = df[column].quantile(0.25)
    Q3 = df[column].quantile(0.75)
    IQR = Q3 - Q1
    lower = Q1 - 1.5 * IQR
    upper = Q3 + 1.5 * IQR
    mask = (
        (df[column] < lower) |
        (df[column] > upper)
    )

return {
    "Column": column,
    "Outlier_Count": mask.sum(),
    "Outlier_Percentage": mask.mean() * 100,
    "Lower_Bound": lower,
    "Upper_Bound": upper
}

6.5.39Mini Project — Customer Transaction Outlier Detection

Create a dataset

• Transaction_ID
• Customer_ID
• Transaction_Date
• Region
• Product
• Quantity
• Unit_Price
• Total_Amount
• Perform the following analysis.
• Step 1 — Data profiling

Inspect

df.info()
df.describe()

• Step 2 — Detect negative values
• df[
• df["Total_Amount"] < 0

]

Step 3 — Detect IQR outliers

Apply the IQR method to

• Quantity
• Unit_Price
• Total_Amount
• Step 4 — Detect Z-score outliers

Calculate

• Z-score > 3
• Z-score < -3
• Step 5 — Compare methods

Determine which transactions are identified by

• IQR only
• Z-score only
• Both
• Step 6 — Visualize

Create

• Box plot
• Histogram
• Scatter plot
• Step 7 — Investigate

Classify each potential outlier

• Genuine
• Data Error
• Business Exception
• Unknown
• Step 8 — Treatment

Apply the appropriate action

• Keep
• Correct
• Remove
• Cap
• Transform
• Investigate
• Step 9 — Generate a final report

Include

• Total Records
• Potential Outliers
• Confirmed Errors
• Genuine Extreme Values
• Removed Records
• Corrected Records
• Remaining Outliers

6.5.40Interview Questions

• Basic
• What is an outlier?
• Why are outliers important?
• What is the difference between an outlier and an error?
• What is the IQR?
• What are Q1 and Q3?
• How is the IQR calculated?
• What is a box plot?
• What is a Z-score?
• What is the common Z-score threshold for outlier detection?
• How do you find outliers using Pandas?

Advanced

• Why is IQR often preferred for skewed data?
• Why can Z-score detection be affected by outliers?
• What is the difference between global and local outliers?
• What is a contextual outlier?
• What is a multivariate outlier?
• How would you detect outliers within groups?
• How would you detect outliers in a time series?
• What is winsorization?
• When should you remove an outlier?
• When should you keep an outlier?
• What is MAD?
• How does Isolation Forest detect anomalies?
• How can outliers affect correlation?
• How can outliers affect machine-learning models?
• How would you design an automated outlier-monitoring process?

6.5.41Key Takeaways

The complete outlier-detection workflow is

DATA

│

▼

PROFILE

│

▼

VISUALIZE DATA

│

▼

DETECT POTENTIAL

OUTLIERS

│

┌─────────────┼─────────────┐

▼ ▼ ▼

IQR Z-SCORE ML METHODS

│ │ │

└─────────────┼─────────────┘

▼

BUSINESS REVIEW

│

┌────────┼────────┐

▼ ▼ ▼

Keep Correct Remove

│ │ │

└────────┼────────┘

▼

VALIDATE

│

▼

CLEAN DATA

• Concepts to master

Focus especially on

• IQR method
• Z-score method
• Box plots
• Percentile-based detection
• Outlier treatment strategies
• Group-level outlier detection
• Time-series outliers
• Multivariate outliers
• Robust statistics
• Business validation

The central principle is

An outlier is a signal to investigate, not a command to delete.

In professional data analysis, the strongest workflow is detect → investigate → classify → treat → validate.

Feature engineering is the process of creating, modifying, selecting, and transforming variables so that data contains useful information for analysis, visualization, statistical modeling, and machine learning.

A feature is simply an input variable.

For example, from

• Order_Date
• Quantity
• Unit_Price
• Discount

we can create

• Year
• Month
• Day_of_Week
• Revenue
• Discount_Amount
• Net_Revenue

These derived features can make patterns much easier to analyze.

6.6.1Learning Objectives

By the end of this topic, you should be able to

• Understand what features are.
• Create new numerical features.
• Create categorical features.
• Extract features from dates.
• Create ratio and percentage features.
• Create aggregation-based features.
• Encode categorical variables.
• Create bins and segments.
• Create interaction features.
• Handle skewed variables.
• Select useful features.
• Avoid data leakage.
• Build reusable feature-engineering pipelines.

6.6.2What Is a Feature?

Consider a customer dataset

Existing features are

• Age
• Income
• Orders

We can create

• Income_Per_Order
• Customer_Segment
• Age_Group

For example

• df["Income_Per_Order"] = (
• df["Income"] /
• df["Orders"]

)

This new feature may provide more analytical value than either variable alone.

6.6.3Why Feature Engineering Matters

Raw data is not always in the most useful form.

Consider

OrderDate = 2026-08-23

The date itself may not directly tell us whether the order occurred on

• a weekend
• a holiday
• a month-end
• a quarter-end

We can create

Year = 2026
Month = 8
Quarter = 3
DayOfWeek = Sunday
IsWeekend = True
IsMonthEnd = False

These features can reveal patterns that aren't obvious in the raw data.

6.6.4Types of Feature Engineering

Common categories include

6.6.5Creating Simple Numerical Features

Suppose

df = pd.DataFrame({
    "Quantity": [2, 5, 3, 10],
    "Unit_Price": [100, 200, 150, 50]
})

Create revenue

• df["Revenue"] = (
• df["Quantity"] *
• df["Unit_Price"]

)

Result

This is one of the simplest forms of feature engineering.

6.6.6Business Features

Suppose an order contains

Revenue

Cost

Create profit

• df["Profit"] = (
• df["Revenue"] -
• df["Cost"]

)

Profit margin

• df["Profit_Margin"] = (
• df["Profit"] /
• df["Revenue"] * 100

)

This converts raw financial data into a meaningful business KPI.

6.6.7Ratio Features

Ratios can reveal relationships between variables.

Example

• df["Revenue_Per_Order"] = (
• df["Revenue"] /
• df["Order_Count"]

)

Customer value

• df["Revenue_Per_Customer"] = (
• df["Revenue"] /
• df["Customer_Count"]

)

Conversion rate

• df["Conversion_Rate"] = (
• df["Conversions"] /
• df["Visitors"] * 100

)

Always handle division by zero

• df["Conversion_Rate"] = np.where(
• df["Visitors"] > 0,
• df["Conversions"] /
• df["Visitors"] * 100,
• 0

)

6.6.8Percentage Change Features

Suppose monthly revenue is

• df["Revenue_Growth"] = (
• df["Revenue"]
• .pct_change() * 100

)

Example

January → 10000

February → 12000

Growth

20%

This is useful for

• revenue growth
• customer growth
• traffic growth
• operational metrics

6.6.9Difference Features

Create absolute change

• df["Revenue_Change"] = (
• df["Revenue"]
• .diff()

)

For example

• 10000
• 12000
• 15000

becomes

• NaN
• 2000
• 3000

6.6.10Date-Based Feature Engineering

Date features are extremely important.

Suppose

df["OrderDate"] = pd.to_datetime(

df["OrderDate"]

)

Create year

df["Year"] = (

df["OrderDate"].dt.year

)

Month

df["Month"] = (

df["OrderDate"].dt.month

)

Quarter

df["Quarter"] = (

df["OrderDate"].dt.quarter

)

Day

df["Day"] = (

df["OrderDate"].dt.day

)

6.6.11Day-of-Week Features

df["DayOfWeek"] = (

df["OrderDate"].dt.dayofweek

)

Values

Monday → 0

Tuesday → 1

Wednesday → 2

Thursday → 3

Friday → 4

Saturday → 5

Sunday → 6

Get the name

df["DayName"] = (

df["OrderDate"].dt.day_name()

)

6.6.12Weekend Feature

Create a Boolean feature

df["IsWeekend"] = (

df["OrderDate"].dt.dayofweek >= 5

)

Or numeric

• df["IsWeekend"] = (
• df["OrderDate"].dt.dayofweek >= 5
• ).astype(int)

Now

0 → Weekday

1 → Weekend

6.6.13Month-End and Month-Start Features

df["IsMonthEnd"] = (

df["OrderDate"].dt.is_month_end

)

df["IsMonthStart"] = (

df["OrderDate"].dt.is_month_start

)

These can be useful for financial and operational analysis.

6.6.14Customer Age from Date of Birth

Suppose

df["DOB"] = pd.to_datetime(

df["DOB"]

)

Calculate approximate age

today = pd.Timestamp.today()

df["Age"] = (

today.year -

df["DOB"].dt.year

)

For more accurate age calculation

df["Age"] = (

today.year -

df["DOB"].dt.year -

(

(today.month, today.day) <

(df["DOB"].dt.month, df["DOB"].dt.day)

)

)

In production code, date handling should be carefully validated around birthdays and missing dates.

6.6.15Customer Tenure

Suppose

Registration_Date

Create tenure in days

df["Tenure_Days"] = (

pd.Timestamp.today() -
df["Registration_Date"]

).dt.days

Tenure in years

df["Tenure_Years"] = (

df["Tenure_Days"] / 365.25

)

This can be useful for customer segmentation and retention analysis.

6.6.16Age Grouping

Suppose

df["Age"]

contains individual ages.

Create groups

bins = [
    0, 18, 30, 45, 60, 100
]

labels = [
    "Under 18",
    "18-30",
    "31-45",
    "46-60",
    "60+"
]

df["Age_Group"] = pd.cut(

df["Age"],

bins=bins,
labels=labels

)

This transforms continuous age into a categorical feature.

6.6.17Income Segmentation

df["Income_Group"] = pd.cut(

df["Income"],

bins=[
    0,
    30000,
    60000,
    100000,
    float("inf")
],
labels=[
    "Low",
    "Medium",
    "High",
    "Very High"
]

)

This can make business reports easier to understand.

6.6.18Quantile-Based Segmentation

Instead of fixed ranges, use qcut()

df["Income_Quartile"] = pd.qcut(

df["Income"],

q=4,
labels=[
    "Q1",
    "Q2",
    "Q3",
    "Q4"
]

)

This creates groups containing approximately equal numbers of records.

6.6.19Categorical Encoding

Machine-learning algorithms often require numerical representations of categories.

Suppose

• df["Gender"] = [
• "Male",
• "Female",
• "Female",
• "Male"

]

Label encoding

from sklearn.preprocessing import LabelEncoder
encoder = LabelEncoder()

• df["Gender_Encoded"] = (
• encoder.fit_transform(
• df["Gender"]

)

)

Potential result

Female → 0

Male → 1

However, label encoding can incorrectly imply an ordering when categories are nominal.

6.6.20One-Hot Encoding

For nominal categories

df = pd.get_dummies(
    df,
    columns=["Gender"]
)

Example

• Gender
• Male
• Female
• Female

becomes something like

• Gender_Female
• Gender_Male
• with binary values.

6.6.21Avoiding the Dummy Variable Trap

For some modeling algorithms, you may drop one category

df = pd.get_dummies(
    df,
    columns=["Gender"],
    drop_first=True
)

This reduces redundant information when using models where perfect multicollinearity is a concern.

6.6.22Frequency Encoding

For high-cardinality categorical variables, frequency encoding can be useful.

Suppose

• City
• Hyderabad
• Hyderabad
• Bengaluru
• Chennai

Calculate frequency

frequency = (
    df["City"]
    .value_counts(normalize=True)
)

df["City_Frequency"] = (

df["City"].map(frequency)

)

Example

Hyderabad → 0.50

Bengaluru → 0.25

Chennai → 0.25

6.6.23Target Encoding

Target encoding replaces categories with statistics based on the target variable.

Example

City Average Revenue

Hyderabad 50000

Bengaluru 65000

Chennai 45000

This can be powerful but has a major risk

Target encoding can cause data leakage if calculated using information from validation/test records.

For machine learning, target encoding must be performed carefully, usually using training-only information or cross-validation schemes.

6.6.24Aggregation Features

Aggregation is one of the most powerful feature-engineering techniques.

Suppose transaction data contains

• Customer_ID
• Order_ID
• Revenue

Calculate total customer revenue

customer_revenue = (
    df.groupby("Customer_ID")["Revenue"]
    .transform("sum")
)

df["Customer_Total_Revenue"] = (

customer_revenue

)

6.6.25Customer-Level Features

From transaction data, create

• Total Revenue
• Number of Orders
• Average Order Value
• Maximum Order Value
• Minimum Order Value
• Last Purchase Date
• Days Since Last Purchase

Example

customer_features = (
    df.groupby("Customer_ID")
    .agg(
        Total_Revenue=("Revenue", "sum"),
        Order_Count=("Order_ID", "nunique"),
        Average_Order_Value=("Revenue", "mean"),
        Max_Order_Value=("Revenue", "max")
    )
    .reset_index()
)

These are highly valuable for customer analytics.

6.6.26RFM Features

A classic customer feature-engineering approach is RFM

• R → Recency
• F → Frequency
• M → Monetary Value
• Recency
• How recently did the customer purchase?

reference_date = df["OrderDate"].max()
recency = (
    df.groupby("Customer_ID")["OrderDate"]
    .max()
    .apply(

lambda x

(reference_date - x).days

)

)

Frequency

Number of orders

frequency = (
df.groupby("Customer_ID")

.nunique()

)

Monetary

Total revenue

monetary = (
df.groupby("Customer_ID")

.sum()

)

RFM is widely used in

• customer segmentation
• marketing
• churn analysis
• customer-value analysis

6.6.27Rolling Features

For time-series data, rolling statistics can become features.

Suppose

• df["Sales_7D_Avg"] = (
• df["Sales"]
• .rolling(7)
• .mean()

)

Other features

• df["Sales_7D_Max"] = (
• df["Sales"]
• .rolling(7)

.max()

)

• df["Sales_7D_Min"] = (
• df["Sales"]
• .rolling(7)

.min()

)

These features capture recent behavior.

6.6.28Lag Features

• Lag features use previous observations.
• df["Sales_Lag_1"] = (
• df["Sales"].shift(1)

)

Two-period lag

df["Sales_Lag_2"] = (

df["Sales"].shift(2)

)

These are extremely useful for forecasting.

6.6.29Lead Features

A lead feature looks into the next observation

df["Sales_Next"] = (

df["Sales"].shift(-1)

)

Be careful with lead features in predictive models because future information can easily cause data leakage.

6.6.30Interaction Features

Sometimes the combination of two features is more useful than either alone.

Suppose

df["Age_Income"] = (

df["Age"] *

df["Income"]

)

Or

• df["Price_Quantity"] = (
• df["Price"] *
• df["Quantity"]

)

For categorical variables, interaction terms can represent combinations such as

Region × Product

These can capture relationships that individual features cannot.

6.6.31Polynomial Features

For numerical variables, you can create powers

df["Income_Squared"] = (

df["Income"] ** 2

)

Using Scikit-learn

from sklearn.preprocessing import PolynomialFeatures
poly = PolynomialFeatures(
    degree=2,
    include_bias=False
)

X_poly = poly.fit_transform(X)

Polynomial features can help models capture nonlinear relationships.

However, they can dramatically increase the number of features.

6.6.32Log Transformation

For highly skewed variables

df["Log_Income"] = np.log1p(

df["Income"]

)

This can reduce the influence of extremely large values.

Common candidates

• Income
• Revenue
• Population
• Transaction amount
• Customer lifetime value

6.6.33Binning

Binning converts continuous variables into groups.

Example

df["Sales_Band"] = pd.cut(

df["Sales"],

bins=[
    0, 1000, 5000,
    10000, float("inf")
],
labels=[
    "Low",
    "Medium",
    "High",
    "Very High"
]

)

This is useful for

• customer segmentation
• reporting
• visualization
• rule-based classification

6.6.34Text Feature Engineering

Text can also be converted into useful numerical features.

Suppose

• df["Review"] = [
• "Excellent product",
• "Good product",
• "Very poor service"

]

Create text length

df["Review_Length"] = (

df["Review"].str.len()

)

Word count

• df["Word_Count"] = (
• df["Review"]
• .str.split()
• .str.len()

)

Contains a keyword

• df["Contains_Poor"] = (
• df["Review"]
• .str.contains(
• "poor",

case=False,
na=False

)

)

More advanced NLP feature engineering can use

• TF-IDF
• word embeddings
• sentence embeddings
• sentiment scores

6.6.35Feature Engineering from File or System Metadata

Sometimes useful features don't come directly from business columns.

Examples

• Source_System
• Load_Date
• File_Name
• Batch_ID
• Data_Source
• Record_Created_Date

For example

df["Load_Date"] = pd.Timestamp.today()

These features are useful for

• data lineage
• data-quality monitoring
• ETL troubleshooting
• auditability

6.6.36Handling Division by Zero

Suppose

• df["Revenue_Per_Order"] = (
• df["Revenue"] /
• df["Order_Count"]

)

If Order_Count = 0, you may get infinity or invalid results.

Better

• df["Revenue_Per_Order"] = np.where(
• df["Order_Count"] > 0,
• df["Revenue"] /
• df["Order_Count"],

np.nan

)

Then handle the resulting missing values appropriately.

6.6.37Feature Selection

Creating hundreds of features does not necessarily improve a model.

You may need to remove

• irrelevant features
• duplicate features
• constant features
• highly correlated features
• leakage features
• unstable features

Check correlation

correlation = (
    df.select_dtypes(
        include="number"
    ).corr()
)

Highly correlated features may provide redundant information depending on the model and objective.

6.6.38Removing Constant Features

constant_features = [
    column
    for column in df.columns
    if df[column].nunique() <= 1
]

These features contain no variation and generally provide little predictive value.

6.6.39Feature Leakage

Data leakage occurs when a feature contains information that would not actually be available at prediction time.

Example

Suppose you're predicting whether an order will be refunded.

This would be dangerous

• Order_Date
• Customer_ID
• Payment_Method
• Refund_Date

Refund_Date is only known after the refund occurs.

Using it to predict the refund would leak future information.

6.6.40Another Leakage Example

Suppose you want to predict customer churn.

You create

Last_30_Days_Orders

If the prediction is made at the beginning of the month, but your feature includes orders from the future, you've introduced leakage.

Correct approach

Not

6.6.41Train/Test Feature Engineering

Correct machine-learning workflow

For example, don't calculate a global mean using the entire dataset before splitting if that mean will be used to construct a predictive feature.

6.6.42Feature Engineering Pipeline with Scikit-Learn

For reproducible ML workflows, use pipelines.

from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
numeric_pipeline = Pipeline([
    (
        "imputer",
        SimpleImputer(strategy="median")
    ),
    (
        "scaler",
        StandardScaler()
    )
])

Categorical pipeline

from sklearn.preprocessing import OneHotEncoder
categorical_pipeline = Pipeline([
    (
        "imputer",
        SimpleImputer(
            strategy="most_frequent"
        )
    ),
    (
        "encoder",
        OneHotEncoder(
            handle_unknown="ignore"
        )
    )
])

These pipelines help prevent inconsistent preprocessing and data leakage.

6.6.43ColumnTransformer

For datasets containing both numerical and categorical variables

from sklearn.compose import ColumnTransformer
numeric_features = [
    "Age",
    "Income",
    "Order_Count"
]

categorical_features = [
    "Gender",
    "City"
]

preprocessor = ColumnTransformer([
    (
        "num",
        numeric_pipeline,
        numeric_features
    ),
    (
        "cat",
        categorical_pipeline,
        categorical_features
    )
])

This allows different feature-engineering strategies for different data types.

6.6.44Complete Business Example

Consider an e-commerce dataset

df = pd.DataFrame({
    "Customer_ID": [1, 1, 2, 2, 3],
    "Order_Date": [
        "2026-01-10",
        "2026-02-15",
        "2026-02-20",
        "2026-03-10",
        "2026-03-15"
    ],
    "Quantity": [2, 3, 1, 5, 2],
    "Unit_Price": [
        500, 400, 1000, 200, 800
    ]
})

Convert date

df["Order_Date"] = pd.to_datetime(

df["Order_Date"]

)

Create revenue

• df["Revenue"] = (
• df["Quantity"] *
• df["Unit_Price"]

)

Create date features

• df["Year"] = df["Order_Date"].dt.year
• df["Month"] = df["Order_Date"].dt.month
• df["DayOfWeek"] = (
• df["Order_Date"].dt.dayofweek

)

df["IsWeekend"] = (

df["Order_Date"].dt.dayofweek >= 5

)

Customer-level features

df["Customer_Total_Revenue"] = (

df.groupby("Customer_ID")

.transform("sum")

)

Order count

df["Customer_Order_Count"] = (

df.groupby("Customer_ID")

.transform("count")

)

Average order value

• df["Customer_AOV"] = (
• df["Customer_Total_Revenue"] /
• df["Customer_Order_Count"]

)

This transforms a basic transaction table into a much richer analytical dataset.

6.6.45Feature Engineering Workflow

A professional workflow is

RAW DATA

│

▼

DATA CLEANING

│

▼

UNDERSTAND BUSINESS

OBJECTIVE

│

▼

CREATE FEATURES

│

┌────────────┼────────────┐

▼ ▼ ▼

Numerical Date Categorical

│ │ │

└────────────┼────────────┘

▼

AGGREGATION

│

▼

TRANSFORMATION

│

▼

FEATURE SELECTION

│

▼

LEAKAGE VALIDATION

│

▼

FINAL FEATURES

│

▼

MODEL / EDA / BI

6.6.46Feature Engineering vs Data Cleaning

• These concepts are closely related but different.
• Data Cleaning
• Fixes problems.
• Invalid age → missing
• "hyderabad" → "Hyderabad"
• Duplicate → remove
• "₹50,000" → 50000
• Feature Engineering
• Creates useful information.
• Age → Age_Group
• Quantity × Price → Revenue
• OrderDate → Month
• Revenue / Orders → AOV
• Last_Order_Date → Recency

A typical workflow is

6.6.47Common Feature Engineering Mistakes

• Mistake 1 — Creating features without a business purpose
• More features do not automatically mean better analysis.
• Mistake 2 — Data leakage
• Using future information creates unrealistic model performance.
• Mistake 3 — Ignoring missing values
• Feature calculations can propagate missing values.

For example

• df["Revenue"] = (
• df["Quantity"] *
• df["Price"]

)

• If either input is missing, revenue may also become missing.
• Mistake 4 — Division by zero
• Revenue / Orders
• must handle zero orders.
• Mistake 5 — Creating features before sorting time-series data

For lag and rolling calculations

df = df.sort_values(

)

should generally happen before applying temporal calculations.

Mistake 6 — Using future information in lag/rolling features

Always define the exact prediction time and ensure every feature was available at that time.

6.6.48Feature Engineering Cheat Sheet

• # Arithmetic
• df["Revenue"] = df["Quantity"] * df["Price"]
• # Ratios
• df["AOV"] = df["Revenue"] / df["Orders"]
• # Difference
• df["Change"] = df["Sales"].diff()
• # Percentage change
• df["Growth"] = df["Sales"].pct_change()
• # Date
• df["Year"] = df["Date"].dt.year
• df["Month"] = df["Date"].dt.month
• df["Quarter"] = df["Date"].dt.quarter
• df["Day"] = df["Date"].dt.day
• df["DayOfWeek"] = df["Date"].dt.dayofweek
• # Weekend
• df["IsWeekend"] = (
• df["Date"].dt.dayofweek >= 5

)

# Binning

df["Age_Group"] = pd.cut(

df["Age"],

bins=[0, 18, 30, 45, 60, 100]

)

# One-hot encoding

pd.get_dummies(df, columns=["City"])

# Aggregation

df["Customer_Revenue"] = (

df.groupby("Customer_ID")

.transform("sum")

)

# Ranking

df["Rank"] = df["Revenue"].rank(

ascending=False

)

• # Lag
• df["Previous_Sales"] = (
• df["Sales"].shift(1)

)

• # Rolling
• df["Rolling_Avg"] = (
• df["Sales"]
• .rolling(7)
• .mean()

)

• # Log transformation
• df["Log_Revenue"] = np.log1p(
• df["Revenue"]

)

6.6.49Interview Questions

• Basic
• What is feature engineering?
• What is a feature?
• Why is feature engineering important?
• Give examples of derived features.
• How can you create features from dates?
• How do you create categorical bins?
• What is one-hot encoding?
• What is label encoding?
• What is a ratio feature?
• What is an interaction feature?

Advanced

• What is feature leakage?
• How do you create customer-level aggregation features?
• What are RFM features?
• How do you create lag features?
• How do rolling features work?
• What is target encoding?
• What problems can high-cardinality categorical variables cause?
• When would you use frequency encoding?
• How do you handle division by zero during feature creation?
• Why should feature engineering be performed carefully with time-series data?
• How do you prevent leakage when creating target-based features?
• What is the difference between feature selection and feature engineering?
• How can feature engineering improve machine-learning performance?
• How would you design a reusable feature-engineering pipeline?
• How would you decide whether a newly engineered feature is actually useful?

6.6.50Mini Project — E-Commerce Customer Feature Engineering

Build a feature-engineering pipeline using

• Customer_ID
• Order_ID
• Order_Date
• Product
• Category
• Quantity
• Unit_Price
• Discount

Create the following features

• Transaction Features
• Revenue
• Discount_Amount
• Net_Revenue
• Date Features
• Year
• Month
• Quarter
• DayOfWeek
• IsWeekend
• IsMonthEnd
• Customer Features
• Total_Revenue
• Order_Count
• Average_Order_Value
• Maximum_Order_Value
• Minimum_Order_Value
• RFM Features
• Recency
• Frequency
• Monetary
• Behavioral Features
• Average_Quantity
• Total_Quantity
• Unique_Product_Count
• Unique_Category_Count
• Time-Series Features
• Previous_Order_Revenue
• Revenue_7_Day_Average
• Revenue_30_Day_Average

Finally, create a customer-level feature table

• Customer_ID
• Recency
• Frequency
• Monetary
• Average_Order_Value
• Total_Quantity
• Unique_Product_Count
• Unique_Category_Count

This becomes the foundation for

• Customer segmentation
• Churn prediction
• Customer lifetime value
• Recommendation systems
• Marketing analytics
• Power BI dashboards

6.6.51Key Takeaways

The feature-engineering workflow is

RAW DATA

│

▼

CLEAN DATA

│

▼

UNDERSTAND BUSINESS

OBJECTIVE

│

▼

CREATE FEATURES

│

┌──────────────┼──────────────┐

▼ ▼ ▼

Numerical Date/Time Categorical

│ │ │

└──────────────┼──────────────┘

▼

Aggregation

│

▼

Ratios / KPIs

│

▼

Lag / Rolling

│

▼

Encoding / Binning

│

▼

Feature Selection

│

▼

Leakage Validation

│

▼

FINAL FEATURES

• The most important concepts to master
• Date/time feature extraction
• Ratio and KPI features
• Group-based aggregation
• RFM/customer features
• Categorical encoding
• Binning and segmentation
• Lag and rolling features
• Interaction features
• Feature selection
• Data-leakage prevention

The central principle is

Good feature engineering converts raw data into information that directly represents the business problem.

For example, instead of giving a model only Order_Date, Quantity, and Price, feature engineering can turn them into Revenue, Month, Weekend Flag, Customer Frequency, Average Order Value, Recency, and Growth, making the dataset far more useful for analytics and predictive modeling.

Data transformation is the process of converting data from one structure, format, scale, or representation into another form that is more suitable for analysis, visualization, reporting, statistical modeling, or machine learning.

For example, raw sales data might contain

Quantity = 5
Unit_Price = ₹2,000
Discount = 10%

After transformation

Quantity        = 5
Unit_Price      = 2000
Discount        = 0.10
Gross_Revenue   = 10000
Discount_Value  = 1000
Net_Revenue     = 9000

Data transformation is closely related to feature engineering, but transformation generally focuses on changing the representation or structure of existing data, whereas feature engineering emphasizes creating useful new variables/features.

6.7.1Learning Objectives

By the end of this topic, you should be able to

• Understand data transformation.
• Transform numerical data.
• Transform categorical data.
• Change data types.
• Normalize and standardize values.
• Apply logarithmic transformations.
• Transform dates and times.
• Reshape DataFrames.
• Aggregate and summarize data.
• Merge and join datasets.
• Pivot and melt data.
• Apply custom transformations.
• Build transformation pipelines.
• Understand transformation leakage.

6.7.2Why Data Transformation Is Important

Raw data often isn't directly usable.

For example

• Sales
• ₹10,000
• ₹20,000
• ₹50,000

This is text, not numeric data.

After transformation

• 10000
• 20000
• 50000

we can calculate

• Mean
• Median
• Total
• Growth
• Correlation

Similarly, machine-learning algorithms often perform better when numerical variables are transformed to comparable scales.

6.7.3Data Transformation Workflow

A typical workflow is

6.7.4Common Types of Data Transformation

Important transformation categories include

• 1. Data Type Conversion
• 2. Numerical Transformation
• 3. Scaling
• 4. Normalization
• 5. Standardization
• 6. Log Transformation
• 7. Categorical Transformation
• 8. Date Transformation
• 9. Aggregation
• 10. Reshaping
• 11. Pivoting
• 12. Melting
• 13. Merging
• 14. Joining
• 15. Mapping
• 16. Custom Transformation
• 6.7.5Data Type Conversion

Suppose

df["Age"].dtype

returns

object

Convert to numeric

df["Age"] = pd.to_numeric(

df["Age"],

errors="coerce"

)

6.7.6String to Numeric

Suppose

• "100"
• "200"
• "300"

Convert

df["Sales"] = pd.to_numeric(

df["Sales"]

)

For invalid values

df["Sales"] = pd.to_numeric(

df["Sales"],

errors="coerce"

)

Invalid values become missing.

6.7.7Currency Transformation

Suppose

• ₹10,000
• ₹25,000
• ₹50,000

Transform

df["Sales"] = (

• df["Sales"]
• .str.replace("₹", "", regex=False)
• .str.replace(",", "", regex=False)

)

df["Sales"] = pd.to_numeric(

df["Sales"],

errors="coerce"

)

Now

• 10000
• 25000
• 50000

6.7.8Percentage Transformation

Suppose

• 10%
• 20%
• 35%

Transform

• df["Discount"] = (
• df["Discount"]
• .str.replace("%", "", regex=False)
• .astype(float)
• / 100

)

Now

• 0.10
• 0.20
• 0.35
• This representation is easier for calculations.

6.7.9Date Transformation

Suppose

df["Order_Date"] = pd.to_datetime(

df["Order_Date"]

)

Extract year

df["Year"] = (

df["Order_Date"].dt.year

)

Month

df["Month"] = (

df["Order_Date"].dt.month

)

Quarter

df["Quarter"] = (

df["Order_Date"].dt.quarter

)

Day

df["Day"] = (

df["Order_Date"].dt.day

)

6.7.10Date Formatting

Suppose

df["Order_Date"]

contains datetime values.

Convert to a formatted string

df["Order_Date_String"] = (

df["Order_Date"]

.dt.strftime("%Y-%m-%d")

)

Example

2026-08-23

Other formats

df["Order_Date"].dt.strftime(

"%d-%m-%Y"

)

or

df["Order_Date"].dt.strftime(

"%B %Y"

)

6.7.11Numerical Transformation

Numerical transformation changes the mathematical representation of a variable.

Common techniques

• Min-Max Scaling
• Standardization
• Robust Scaling
• Log Transformation
• Power Transformation
• Quantile Transformation

6.7.12Min-Max Scaling

Min-Max scaling transforms values to a fixed range, usually

0 to 1

Formula

X_scaled =

(X - X_min) /

(X_max - X_min)

Example

from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler()

• df["Income_Scaled"] = (
• scaler.fit_transform(
• df[["Income"]]

)

)

If the original values are

• 100
• 200
• 300

they become approximately

• 0
• 0.5
• 1

6.7.13When to Use Min-Max Scaling

Min-Max scaling is useful when

• A fixed range is desirable.
• Features have different units.
• Algorithms are sensitive to feature magnitude.

Common applications include

• Neural networks
• Distance-based algorithms
• KNN
• Some optimization-based models

6.7.14Standardization

Standardization transforms data so that it has approximately

Mean = 0

Standard deviation = 1

Formula

Z = (X - Mean) / Standard Deviation

Using Scikit-learn

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()

• df["Income_Standardized"] = (
• scaler.fit_transform(
• df[["Income"]]

)

)

6.7.15Min-Max vs Standardization

Neither method is universally better.

6.7.16Robust Scaling

When data contains significant outliers, robust scaling can be useful.

It uses

Median

Interquartile range

from sklearn.preprocessing import RobustScaler
scaler = RobustScaler()

• df["Income_Robust"] = (
• scaler.fit_transform(
• df[["Income"]]

)

)

This is less affected by extreme observations than mean/std-based scaling.

6.7.17Log Transformation

Consider highly skewed revenue

• 100
• 200
• 300
• 500
• 10000
• 50000
• A log transformation compresses large values.

import numpy as np

df["Log_Revenue"] = np.log1p(

df["Revenue"]

)

log1p(x) means

log(1 + x)

It is useful when zero values are possible.

6.7.18Why Log Transformation Helps

Without transformation

• 100
• 200
• 300
• 500
• 10000
• 50000

After log transformation, the difference between large values is compressed.

This can

• Reduce skewness
• Reduce influence of extreme values
• Improve visualization
• Help some statistical models
• Make relationships more linear

6.7.19Power Transformation

Power transformations can help make skewed distributions more suitable for analysis.

Scikit-learn provides

from sklearn.preprocessing import PowerTransformer
transformer = PowerTransformer()
X_transformed = (
    transformer.fit_transform(X)
)

Common methods include

Yeo-Johnson

Box-Cox

Yeo-Johnson can handle zero and negative values, while Box-Cox has stricter input requirements.

6.7.20Quantile Transformation

Quantile transformation maps values according to their rank distribution.

from sklearn.preprocessing import QuantileTransformer
transformer = QuantileTransformer(
    output_distribution="normal"
)

X_transformed = (
    transformer.fit_transform(X)
)

This can be useful for heavily skewed data, but it changes the original distribution substantially.

6.7.21Categorical Transformation

Categorical values need to be represented appropriately.

Example

• Gender
• Male
• Female
• Male
• Female

One-hot encoding

df = pd.get_dummies(
    df,
    columns=["Gender"]
)

Result

Gender_Female

Gender_Male

6.7.22Mapping Categories

Suppose

• Low
• Medium
• High

You can map them

mapping = {
    "Low": 1,
    "Medium": 2,
    "High": 3
}

df["Priority_Code"] = (

df["Priority"].map(mapping)

)

This is appropriate when the categories have a genuine ordinal relationship.

6.7.23Ordinal vs Nominal Categories

Ordinal

Categories have an order

Low < Medium < High

Numeric mapping can make sense

Low → 1

Medium → 2

High → 3

Nominal

Categories have no natural order

• Hyderabad
• Bengaluru
• Chennai

Using

Hyderabad = 1
Bengaluru = 2
Chennai = 3

would incorrectly imply an ordering.

One-hot encoding is usually more appropriate.

6.7.24String Transformation

Convert text to lowercase

df["City"] = (

df["City"]

.str.lower()

)

Remove spaces

df["City"] = (

df["City"]

.str.strip()

)

Replace values

df["City"] = df["City"].replace({

"bangalore": "bengaluru"

})

6.7.25Conditional Transformation

Suppose

df["Sales"]

Create a sales category

• df["Sales_Category"] = np.where(
• df["Sales"] >= 100000,
• "High",
• "Normal"

)

For multiple conditions, use np.select()

conditions = [
    df["Sales"] < 10000,
    df["Sales"].between(
        10000, 50000
    ),
    df["Sales"] > 50000
]

choices = [
    "Low",
    "Medium",
    "High"
]

• df["Sales_Category"] = np.select(
• conditions,
• choices,

default="Unknown"

)

6.7.26Aggregation Transformation

Suppose transaction-level data

Aggregate by customer

customer_sales = (
    df.groupby("Customer")["Revenue"]
    .sum()
    .reset_index()
)

Result

Aggregation converts transaction-level data into customer-level data.

6.7.27Multiple Aggregations

summary = (
    df.groupby("Customer")
    .agg(
        Total_Revenue=("Revenue", "sum"),
        Average_Revenue=("Revenue", "mean"),
        Max_Revenue=("Revenue", "max"),
        Order_Count=("Order", "nunique")
    )
    .reset_index()
)

This is extremely common in data analysis.

6.7.28Transformation Using apply()

You can apply a custom function.

def categorize_sales(value):
if value < 10000:
return "Low"
elif value < 50000:
return "Medium"
else:
return "High"

• df["Sales_Category"] = (
• df["Sales"].apply(
• categorize_sales

)

)

For simple conditions, vectorized operations are generally preferable because they are usually faster.

6.7.29Lambda Transformation

• df["Sales_Thousands"] = (
• df["Sales"]
• .apply(lambda x: x / 1000)

)

However, prefer vectorized Pandas operations where possible

df["Sales_Thousands"] = (

df["Sales"] / 1000

)

This is usually faster and more idiomatic.

6.7.30Reshaping Data

Data can exist in

Wide format

Month Product_A Product_B Product_C

Jan 100 200 300

Feb 120 220 320

Long format

Month Product Sales

Jan Product_A 100

Jan Product_B 200

Jan Product_C 300

Feb Product_A 120

...

Pandas provides

• melt()
• pivot()
• pivot_table()
• stack()
• unstack()

6.7.31Melting Data

Example

df_long = df.melt(
    id_vars=["Month"],
    var_name="Product",
    value_name="Sales"
)

melt() converts wide data into long format.

This is particularly useful for

• visualization
• statistical analysis
• BI workflows

6.7.32Pivoting Data

Suppose long data

Month Product Sales

Jan A 100

Jan B 200

Feb A 120

Feb B 220

Pivot

pivoted = df.pivot(
    index="Month",
    columns="Product",
    values="Sales"
)

Result

Product A B

Month

Jan 100 200

Feb 120 220

6.7.33Pivot Tables

For duplicate combinations, use pivot_table()

pivot = pd.pivot_table(
    df,
    index="Region",
    columns="Product",
    values="Sales",
    aggfunc="sum"
)

This is extremely useful for business analysis.

6.7.34Transposing Data

df.T

This swaps rows and columns.

Useful for quick inspection, but usually not a primary analytical transformation.

6.7.35Merging DataFrames

Suppose

customers

contains

• Customer_ID
• Customer_Name
• City

and

orders

contains

• Order_ID
• Customer_ID
• Revenue

Merge

result = pd.merge(
    orders,
    customers,
    on="Customer_ID",
    how="left"
)

6.7.36Types of Joins

Common joins

• inner
• left
• right
• outer
• cross
• Inner Join
• Returns matching records.

pd.merge(
    orders,
    customers,
    on="Customer_ID",
    how="inner"
)

Left Join

Keeps all orders

pd.merge(
    orders,
    customers,
    on="Customer_ID",
    how="left"
)

This is often useful when enriching transaction data with master data.

6.7.37Concatenating Data

Suppose you have January and February datasets

jan = pd.read_csv("jan.csv")
feb = pd.read_csv("feb.csv")

Combine rows

df = pd.concat(
    [jan, feb],
    ignore_index=True
)

This is useful when datasets have the same structure.

6.7.38Combining Columns

You can also concatenate along columns

result = pd.concat(
    [df1, df2],
    axis=1
)

Use this carefully because rows must be aligned correctly.

6.7.39Data Transformation for Power BI

A typical analytics workflow might be

For example, transaction data

• Customer_ID
• Order_Date
• Quantity
• Unit_Price

can be transformed into

• Customer_ID
• Year
• Month
• Revenue
• Order_Count
• Average_Order_Value

The resulting dataset is much easier to use for dashboard development.

6.7.40Transformation Order Matters

A common pipeline is

But the exact order depends on the use case.

For example, scaling should generally happen after splitting train/test data in a predictive modeling workflow.

6.7.41Avoiding Data Leakage

Suppose you want to standardize

Income

Incorrect approach

This allows test-set information to influence the transformation.

Correct

In Scikit-learn

scaler.fit(X_train)

X_train_scaled = (
    scaler.transform(X_train)
)

X_test_scaled = (
    scaler.transform(X_test)
)

6.7.42Pipeline-Based Transformation

A better approach is

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.impute import SimpleImputer
pipeline = Pipeline([
    (
        "imputer",
        SimpleImputer(strategy="median")
    ),
    (
        "scaler",
        StandardScaler()
    )
])

Then

X_train_transformed = (
    pipeline.fit_transform(X_train)
)

X_test_transformed = (
    pipeline.transform(X_test)
)

This helps keep preprocessing consistent.

6.7.43Transformation of Skewed Data

Suppose revenue is

• 100
• 200
• 300
• 400
• 10000
• 50000

Check skewness

df["Revenue"].skew()

If strongly positive

df["Log_Revenue"] = np.log1p(

df["Revenue"]

)

Check again

df["Log_Revenue"].skew()

This allows you to compare the distribution before and after transformation.

6.7.44Transformation Validation

After transformation, always verify

print(df.info())
print(df.describe())

For numerical transformation

print(
    df["Revenue"].describe()
)

print(
    df["Log_Revenue"].describe()
)

For categorical transformation

print(
    df["Category"].value_counts()
)

For joins

print(
    len(before),
    len(after)
)

Unexpected row-count changes can reveal join problems.

6.7.45Common Data Transformation Mistakes

• Mistake 1 — Transforming without understanding the data
• Always inspect the data first.
• Mistake 2 — Treating missing values as zero
• Missing ≠ Zero
• Mistake 3 — Scaling before train/test split
• This can cause leakage.
• Mistake 4 — Incorrect join keys
• A many-to-many join can unexpectedly multiply records.

Always validate

• Before rows
• After rows
• Expected relationship
• Mistake 5 — Applying log to negative values

This is invalid for the ordinary logarithm.

For non-negative values

np.log1p(x)

can handle zero, but not values below -1.

Mistake 6 — Encoding nominal variables as arbitrary numbers

Don't assume

Hyderabad = 1
Chennai = 2
Bengaluru = 3

has meaningful order.

Use one-hot or another appropriate encoding.

6.7.46Complete Example

Consider an e-commerce dataset

df = pd.DataFrame({
    "Order_ID": [1, 2, 3, 4],
    "Order_Date": [
        "2026-01-10",
        "2026-02-15",
        "2026-02-20",
        "2026-03-10"
    ],
    "Quantity": [2, 5, 1, 10],
    "Unit_Price": [
        "₹500",
        "₹1,000",
        "₹750",
        "₹200"
    ],
    "Discount": [
        "10%",
        "5%",
        "0%",
        "20%"
    ]
})

• Step 1 — Convert date
• df["Order_Date"] = pd.to_datetime(
• df["Order_Date"]

)

• Step 2 — Clean price
• df["Unit_Price"] = (
• df["Unit_Price"]
• .str.replace("₹", "", regex=False)
• .str.replace(",", "", regex=False)
• .astype(float)

)

• Step 3 — Convert discount
• df["Discount"] = (
• df["Discount"]
• .str.replace("%", "", regex=False)
• .astype(float)
• / 100

)

• Step 4 — Create gross revenue
• df["Gross_Revenue"] = (
• df["Quantity"] *
• df["Unit_Price"]

)

• Step 5 — Calculate discount amount
• df["Discount_Amount"] = (
• df["Gross_Revenue"] *
• df["Discount"]

)

• Step 6 — Calculate net revenue
• df["Net_Revenue"] = (
• df["Gross_Revenue"] -
• df["Discount_Amount"]

)

Step 7 — Extract month

df["Month"] = (

df["Order_Date"].dt.month

)

Step 8 — Validate

print(df.describe())
print(df.info())

This converts messy business data into analysis-ready data.

6.7.47Real-World Transformation Architecture

A production analytics environment may look like

SOURCE SYSTEMS

│

┌───────────┼───────────┐

▼ ▼ ▼

SQL Server Oracle CSV

│ │ │

└───────────┼───────────┘

▼

RAW DATA

│

▼

DATA CLEANING

│

▼

DATA TRANSFORMATION

┌───────────┼───────────┐

▼ ▼ ▼

Types Scaling Business

Conversion Calculations

│ │ │

└───────────┼───────────┘

▼

AGGREGATION

│

▼

DATA MODEL

│

┌────────┴────────┐

▼ ▼

Power BI ML / Python

6.7.48Mini Project — Sales Data Transformation

Create a dataset containing

• Order_ID
• Customer_ID
• Order_Date
• Region
• Product
• Quantity
• Unit_Price
• Discount
• Perform the following transformations.
• Step 1 — Clean types

Convert

Order_Date → datetime

Quantity → numeric

Unit_Price → numeric

Discount → numeric

• Step 2 — Create financial measures
• Gross_Revenue
• Discount_Amount
• Net_Revenue
• Step 3 — Create date features
• Year
• Month
• Quarter
• DayOfWeek
• IsWeekend
• Step 4 — Create customer-level aggregation
• Total_Revenue
• Order_Count
• Average_Order_Value
• Step 5 — Create regional summary
• Region
• Total_Revenue
• Average_Revenue
• Order_Count
• Step 6 — Create a pivot table

Rows

Region

Columns

Month

Values

Net_Revenue

Step 7 — Scale numerical variables

Apply

• Min-Max Scaling
• Standardization
• Robust Scaling
• Compare the results.
• Step 8 — Apply log transformation

Transform

Net_Revenue

if it is heavily skewed.

Step 9 — Validate

Check

• Row count
• Null count
• Data types
• Minimum values
• Maximum values
• Duplicate IDs
• Unexpected join multiplication

6.7.49Interview Questions

• Basic
• What is data transformation?
• Why is data transformation important?
• What is the difference between data cleaning and transformation?
• How do you convert strings to numeric values in Pandas?
• How do you convert strings to datetime?
• What is Min-Max scaling?
• What is standardization?
• What is log transformation?
• What is one-hot encoding?
• What is data reshaping?

Advanced

• What is the difference between normalization and standardization?
• When would you use RobustScaler?
• When would you use a log transformation?
• What is the difference between pivot() and pivot_table()?
• What is the difference between merge() and concat()?
• How can a join unexpectedly increase row count?
• How do you transform categorical variables?
• How do you handle skewed numerical variables?
• How do you prevent transformation-related data leakage?
• Why should scalers be fitted only on training data?
• What is a PowerTransformer?
• What is QuantileTransformer?
• How would you build a reusable transformation pipeline?
• How would you validate transformations in production?
• How would you transform a raw transactional dataset into a Power BI-ready dataset?

6.7.50Key Takeaways

The complete transformation process is

RAW DATA

│

▼

DATA CLEANING

│

▼

TYPE CONVERSION

│

▼

VALUE TRANSFORMATION

│

┌────────────────┼────────────────┐

▼ ▼ ▼

Numerical Categorical Date/Time

Scaling Encoding Extraction

│ │ │

└────────────────┼────────────────┘

▼

AGGREGATION

│

▼

RESHAPING DATA

│

▼

MERGE / JOIN DATA

│

▼

VALIDATION

│

▼

ANALYSIS / BI / ML

• Most important concepts

Focus especially on

• Data-type conversion
• Min-Max scaling
• Standardization
• Robust scaling
• Log transformation
• Categorical encoding
• Date transformation
• Aggregation and grouping
• Pivot / melt / reshape
• Merge / join / concatenate
• Transformation pipelines
• Data-leakage prevention

Data transformation converts raw data into a representation that is technically correct, analytically useful, and appropriate for the intended business or machine-learning task.

A useful way to remember the distinction is

Data Cleaning → Fix the data. Feature Engineering → Create useful information. Data Transformation → Change the representation or structure of the data.

Matplotlib is one of the most widely used Python libraries for creating static data visualizations.

It allows analysts and developers to create

• Line charts
• Bar charts
• Histograms
• Scatter plots
• Pie charts
• Box plots
• Area charts
• Heatmaps
• Subplots
• Customized dashboards and reports

Matplotlib is particularly important because libraries such as Seaborn and many other Python visualization tools build on concepts from Matplotlib.

6.8.1Learning Objectives

By the end of this topic, you should be able to

• Understand Matplotlib architecture.
• Create basic charts.
• Customize titles, labels, and legends.
• Control axes and ticks.
• Create multiple plots.
• Create subplots.
• Visualize Pandas DataFrames.
• Create statistical charts.
• Annotate charts.
• Save charts to files.
• Customize chart appearance.
• Build business-oriented visualizations.

6.8.2Installing Matplotlib

Install using

pip install matplotlib

Import

import matplotlib.pyplot as plt

The common alias is

plt

6.8.3Basic Matplotlib Structure

A typical chart follows

Basic example

import matplotlib.pyplot as plt
x = [1, 2, 3, 4, 5]
y = [10, 20, 15, 30, 25]
plt.plot(x, y)
plt.show()

6.8.4Figure and Axes

Matplotlib has two important concepts

• Figure
• The entire drawing area.
• Axes

The actual plotting area inside the figure.

Conceptually

Figure

┌─────────────────────────────┐

│ │

│ Axes │

│ ┌───────────────┐ │

│ │ │ │

│ │ Chart │ │

│ │ │ │

│ └───────────────┘ │

│ │

└─────────────────────────────┘

The object-oriented approach is

fig, ax = plt.subplots()

ax.plot(x, y)

plt.show()

For larger applications, this approach is usually preferable.

6.8.5Line Chart

A line chart is useful for showing trends over an ordered variable such as time.

import matplotlib.pyplot as plt
months = [
    "Jan", "Feb", "Mar",
    "Apr", "May", "Jun"
]

sales = [
    10000, 12000, 15000,
    13000, 18000, 20000
]

plt.plot(
    months,
    sales
)

plt.title("Monthly Sales")
plt.xlabel("Month")
plt.ylabel("Sales")
plt.show()

6.8.6Adding Markers

plt.plot(
    months,
    sales,
    marker="o"
)

Markers make individual observations easier to see.

Common markers

• o → Circle
• s → Square
• ^ → Triangle
• D → Diamond
• x → X
• * → Star

6.8.7Line Styles

Matplotlib supports several line styles

plt.plot(
    months,
    sales,
    linestyle="--"
)

Common options

- Solid

-- Dashed

-. Dash-dot

: Dotted

6.8.8Multiple Lines

Suppose you want to compare two products

product_a = [100, 120, 140, 160, 180]
product_b = [90, 130, 125, 170, 190]
plt.plot(
    months[:5],
    product_a,
    label="Product A"
)

plt.plot(
    months[:5],
    product_b,
    label="Product B"
)

plt.title("Product Sales")
plt.xlabel("Month")
plt.ylabel("Sales")
plt.legend()
plt.show()

The legend identifies each series.

6.8.9Grid

Add a grid

plt.grid()

Or

plt.grid(
    axis="y"
)

A grid can help users compare values more easily.

Avoid excessive gridlines in polished business charts.

6.8.10Figure Size

Set the chart size

plt.figure(
    figsize=(10, 5)
)

For example

plt.figure(
    figsize=(12, 6)
)

plt.plot(months, sales)
plt.show()

6.8.11Bar Chart

Bar charts are useful for comparing categories.

products = [
    "Laptop",
    "Phone",
    "Tablet",
    "Monitor"
]

sales = [
    50000,
    80000,
    30000,
    45000
]

plt.bar(
    products,
    sales
)

plt.title("Product Sales")
plt.xlabel("Product")
plt.ylabel("Sales")
plt.show()

6.8.12Horizontal Bar Chart

Use barh()

plt.barh(
    products,
    sales
)

plt.title("Product Sales")
plt.show()

Horizontal bars are often better when category names are long.

6.8.13Grouped Bar Chart

Suppose

months = ["Jan", "Feb", "Mar"]
product_a = [100, 120, 140]
product_b = [90, 130, 150]

Use positions

import numpy as np
x = np.arange(len(months))
width = 0.35
plt.bar(
    x - width/2,
    product_a,
    width,
    label="Product A"
)

plt.bar(
    x + width/2,
    product_b,
    width,
    label="Product B"
)

plt.xticks(
    x,
    months
)

plt.legend()
plt.show()

6.8.14Stacked Bar Chart

plt.bar(
    months,
    product_a,
    label="Product A"
)

plt.bar(
    months,
    product_b,
    bottom=product_a,
    label="Product B"
)

plt.legend()
plt.show()

This is useful for showing composition.

6.8.15Histogram

A histogram shows the distribution of a numerical variable.

ages = [
    21, 22, 25, 28,
    30, 32, 35, 36,
    40, 42, 45, 50
]

plt.hist(
    ages,
    bins=5
)

plt.title("Age Distribution")
plt.xlabel("Age")
plt.ylabel("Frequency")
plt.show()

6.8.16Understanding Bins

Suppose

plt.hist(
    ages,
    bins=10
)

More bins

Show more detail.

May make the chart noisy.

Fewer bins

Simplify the distribution.

May hide useful patterns.

Choosing bins is partly a visualization judgment.

6.8.17Density-Style Histogram

You can normalize the histogram

plt.hist(
    ages,
    bins=10,
    density=True
)

This represents density rather than raw count.

6.8.18Scatter Plot

Scatter plots show relationships between two numerical variables.

Example

income = [
    30000, 40000, 50000,
    60000, 70000, 80000
]

spending = [
    2000, 2500, 3000,
    4000, 5000, 6500
]

plt.scatter(
    income,
    spending
)

plt.title(
    "Income vs Spending"
)

plt.xlabel("Income")
plt.ylabel("Spending")
plt.show()

Scatter plots help identify

• Correlation
• Clusters
• Outliers
• Nonlinear patterns

6.8.19Scatter Plot with Categories

Suppose customers belong to different segments.

plt.scatter(
    income,
    spending,
    label="Customers"
)

plt.legend()
plt.show()

For more advanced categorical visualization, Seaborn is often more convenient.

6.8.20Pie Chart

Pie charts show proportions.

categories = [
    "Electronics",
    "Clothing",
    "Food",
    "Books"
]

sales = [
    40,
    25,
    20,
    15
]

plt.pie(
    sales,
    labels=categories,
    autopct="%1.1f%%"
)

plt.title("Sales Distribution")
plt.show()

Pie charts work best with a small number of categories.

6.8.21Exploded Pie Chart

explode = [
    0.1,
    0,
    0,
    0
]

plt.pie(
    sales,
    labels=categories,
    explode=explode,
    autopct="%1.1f%%"
)

plt.show()

Use this sparingly.

6.8.22Box Plot

Box plots are useful for understanding

• Median
• Quartiles
• IQR
• Potential outliers

sales = [
    100, 120, 130,
    140, 150, 160,
    500
]

plt.boxplot(
    sales
)

plt.title(
    "Sales Distribution"
)

plt.ylabel("Sales")
plt.show()

6.8.23Understanding a Box Plot

Conceptually

Maximum

│

│

┌─────────────┐

│ │

│ Q3 │

│─────────────│

│ Median │

│─────────────│

│ Q1 │

└─────────────┘

│

│

Minimum

Potential outliers appear separately from the whiskers.

6.8.24Multiple Box Plots

Compare several categories

sales_data = [
    [100, 120, 130, 140],
    [200, 220, 250, 270],

]

plt.boxplot(
    sales_data,
    labels=[
        "Region A",
        "Region B",
        "Region C"
    ]
)

plt.title(
    "Regional Sales"
)

plt.show()

6.8.25Area Chart

plt.fill_between(
    months,
    sales
)

plt.title(
    "Sales Trend"
)

plt.show()

Area charts emphasize magnitude over time.

6.8.26Step Chart

plt.step(
    months,
    sales
)

plt.title(
    "Sales Progression"
)

plt.show()

Useful when values change in discrete steps.

6.8.27Stem Plot

plt.stem(
    [1, 2, 3, 4],

)

plt.show()

Useful for discrete observations.

6.8.28Titles and Labels

A professional chart should usually contain

plt.title(
    "Monthly Revenue"
)

plt.xlabel(
    "Month"
)

plt.ylabel(
    "Revenue (₹)"
)

Labels make charts understandable without requiring additional explanation.

6.8.29Legend

Multiple series require a legend

plt.plot(
    months,
    product_a,
    label="Product A"
)

plt.plot(
    months,
    product_b,
    label="Product B"
)

plt.legend()

You can control location

plt.legend(
    loc="upper left"
)

6.8.30Axis Limits

Set y-axis limits

plt.ylim(
    0,
    100000
)

Set x-axis limits

plt.xlim(
    0,
    12
)

Be careful not to manipulate axis limits in a way that exaggerates or hides important differences.

6.8.31Tick Rotation

Long category names may overlap.

plt.xticks(
    rotation=45
)

For 90 degrees

plt.xticks(
    rotation=90
)

6.8.32Custom Tick Labels

plt.xticks(
    [0, 1, 2],

)

This is useful when x-values are numerical positions but the desired labels are text.

6.8.33Formatting Large Numbers

For business charts, values like

1000000

may be harder to read.

You can use a formatter

from matplotlib.ticker import FuncFormatter
def millions(x, pos):
    return f"{x/1_000_000:.1f}M"
    plt.gca().yaxis.set_major_formatter(
        FuncFormatter(millions)
    )

The axis can then display values such as

• 1.0M
• 2.0M
• 3.0M

6.8.34Adding Annotations

You can highlight an important point

plt.plot(
    months,
    sales,
    marker="o"
)

plt.annotate(
    "Highest Sales",
    xy=("Jun", 20000),
    xytext=("Apr", 22000),
    arrowprops=dict(
        arrowstyle="->"
    )
)

plt.show()

Annotations are useful for highlighting

• Peaks
• Drops
• Business events
• Outliers
• Important milestones

6.8.35Reference Lines

Add a target

plt.axhline(
    y=15000,
    linestyle="--"
)

For example

Target = ₹15,000

This allows users to compare actual performance against a target.

6.8.36Vertical Reference Line

plt.axvline(
    x=3,
    linestyle="--"
)

Useful for marking

• Launch date
• Policy change
• Incident
• Promotion
• Business event

6.8.37Multiple Subplots

Create multiple charts in one figure

fig, axes = plt.subplots(

2,

2,

figsize=(10, 8)

)

Then

• axes[0, 0].plot(
• months,
• sales

)

• axes[0, 1].bar(
• products,
• product_sales

)

axes[1, 0].hist(

ages

)

• axes[1, 1].scatter(
• income,
• spending

)

plt.tight_layout()
plt.show()

This creates a 2 × 2 layout.

6.8.38Understanding subplots()

• fig, ax = plt.subplots()
• One plot.
• fig, axes = plt.subplots(

2,

2

)

Four plots.

┌────────────┬────────────┐

│ Plot 1 │ Plot 2 │

├────────────┼────────────┤

│ Plot 3 │ Plot 4 │

└────────────┴────────────┘

6.8.39Object-Oriented Matplotlib

For professional code, prefer

• fig, ax = plt.subplots()
• ax.plot(
• months,
• sales

)

ax.set_title(

"Monthly Sales"

)

ax.set_xlabel(

"Month"

)

ax.set_ylabel(

"Sales"

)

plt.show()

Instead of relying entirely on global plt state.

This becomes especially useful when working with multiple axes.

6.8.40Pandas + Matplotlib

Pandas integrates directly with Matplotlib.

Suppose

df = pd.DataFrame({
    "Month": [
        "Jan", "Feb", "Mar"
    ],
    "Sales": [
        10000, 12000, 15000
    ]
})

Plot

df.plot(
    x="Month",
    y="Sales",
    kind="line"
)

plt.show()

Bar

df.plot(
    x="Month",
    y="Sales",
    kind="bar"
)

plt.show()

Histogram

df["Sales"].plot(

kind="hist"

)

plt.show()

6.8.41Common Pandas Plot Types

• line
• bar
• barh
• hist
• box
• area
• scatter
• pie

Example

df.plot(
    kind="box"
)

6.8.42Plotting Multiple Columns

df.plot(
    x="Month",
    y=[
        "Product_A",
        "Product_B"
    ]
)

plt.show()

This creates multiple series in one chart.

6.8.43Saving a Chart

Use

plt.savefig(
    "sales_chart.png"
)

High-resolution output

plt.savefig(
    "sales_chart.png",
    dpi=300,
    bbox_inches="tight"
)

Common formats

• PNG
• JPG
• SVG
• PDF

For example

plt.savefig(
    "sales_chart.pdf"
)

6.8.44Saving Before show()

Prefer

plt.savefig(
    "chart.png"
)

plt.show()

rather than depending on state after show().

For reusable plotting functions, it is often cleaner to explicitly save from the Figure object:

• fig, ax = plt.subplots()
• ax.plot(months, sales)
• fig.savefig(
• "chart.png",

dpi=300,
bbox_inches="tight"

)

6.8.45Handling Missing Values

Matplotlib generally skips missing numerical observations in line plots.

Example

sales = [
    100,
    120,
    np.nan,
    160
]

plt.plot(
    sales,
    marker="o"
)

plt.show()

The missing point creates a break rather than automatically inventing a value.

This is often preferable because the visualization doesn't silently fabricate data.

6.8.46Plotting Dates

dates = pd.date_range(
    "2026-01-01",
    periods=10
)

sales = [
    100, 120, 130,
    110, 150, 160,
    170, 165, 180, 200
]

• fig, ax = plt.subplots()
• ax.plot(
• dates,
• sales

)

ax.set_title(

"Daily Sales"

)

fig.autofmt_xdate()

plt.show()

autofmt_xdate() helps format date labels.

6.8.47Business KPI Visualization

Suppose a company tracks

• Revenue
• Profit
• Orders
• Customers
• Conversion Rate

A Matplotlib dashboard might include

┌─────────────────────────────────────┐

│ Revenue Trend │

│ Line Chart │

├───────────────────┬─────────────────┤

│ Regional Revenue │ Order Volume │

│ Bar Chart │ Line Chart │

├───────────────────┼─────────────────┤

│ Profit Margin │ Customer Growth │

│ Line Chart │ Area Chart │

└───────────────────┴─────────────────┘

This combines multiple visualizations into one analytical view.

6.8.48Choosing the Right Chart

6.8.49Matplotlib Styling Principles

A good business visualization should have

• Clear title
• ax.set_title(
• "Monthly Revenue"

)

• Meaningful labels
• ax.set_xlabel("Month")
• ax.set_ylabel("Revenue (₹)")
• Appropriate scale
• Avoid misleading axes.
• Limited visual complexity
• Don't add unnecessary elements.
• Readable text
• Make sure labels are large enough.
• Appropriate chart type

Use a chart that matches the analytical question.

6.8.50Color and Styling

Matplotlib allows extensive customization

plt.plot(
    months,
    sales,
    color="blue",
    linewidth=2,
    linestyle="--",
    marker="o"
)

You can also define styles, fonts, markers, and transparency.

For example

plt.plot(
    months,
    sales,
    alpha=0.7
)

However, customization should support readability rather than become decoration.

6.8.51Transparency

Transparency is controlled using alpha

plt.scatter(
    income,
    spending,
    alpha=0.5
)

This is useful when many points overlap.

6.8.52Chart Annotations for Business Events

• Suppose revenue increased after a marketing campaign.
• fig, ax = plt.subplots()
• ax.plot(
• months,
• sales,

marker="o"

)

ax.annotate(

"Marketing Campaign",

xy=("Apr", 18000),
xytext=("Feb", 21000),
arrowprops=dict(
    arrowstyle="->"
)

)

plt.show()

This turns a chart into a simple business story.

6.8.53Advanced Example — Sales Dashboard

import pandas as pd
import matplotlib.pyplot as plt
df = pd.DataFrame({
    "Month": [
        "Jan", "Feb", "Mar",
        "Apr", "May", "Jun"
    ],
    "Revenue": [
        100000,
        120000,
        115000,
        150000,
        165000,
        180000
    ],
    "Orders": [
        1000,
        1100,
        1050,
        1300,
        1400,
        1500
    ],
    "Profit": [
        20000,
        24000,
        23000,
        30000,
        35000,
        40000
    ]
})

Create dashboard

fig, axes = plt.subplots(

2,

2,

figsize=(12, 8)

)

• # Revenue
• axes[0, 0].plot(
• df["Month"],
• df["Revenue"],

marker="o"

)

axes[0, 0].set_title(

"Revenue Trend"

)

• # Orders
• axes[0, 1].bar(
• df["Month"],
• df["Orders"]

)

axes[0, 1].set_title(

"Order Volume"

)

• # Profit
• axes[1, 0].plot(
• df["Month"],
• df["Profit"],

marker="o"

)

axes[1, 0].set_title(

"Profit Trend"

)

• # Revenue vs Orders
• axes[1, 1].scatter(
• df["Orders"],
• df["Revenue"]

)

axes[1, 1].set_title(

"Revenue vs Orders"

)

plt.tight_layout()
plt.show()

This demonstrates how Matplotlib can be used to build a small analytical dashboard.

6.8.54Matplotlib with SQL Data

In real-world data analysis, data often comes from databases.

For example

import pandas as pd
import matplotlib.pyplot as plt
import sqlalchemy
engine = sqlalchemy.create_engine(
    "your_connection_string"
)

query = """

• SELECT
• OrderDate,
• SUM(Revenue) AS Revenue
• FROM Sales
• GROUP BY OrderDate
• ORDER BY OrderDate

"""

df = pd.read_sql(
    query,
    engine
)

Then

df["OrderDate"] = pd.to_datetime(

df["OrderDate"]

)

• fig, ax = plt.subplots()
• ax.plot(
• df["OrderDate"],
• df["Revenue"]

)

ax.set_title(

"Daily Revenue"

)

plt.show()

This is a common SQL → Pandas → Matplotlib workflow.

6.8.55Matplotlib with Power BI-Oriented Analysis

Even if the final dashboard is built in Power BI, Matplotlib is useful for

• Exploratory data analysis
• Validating KPIs
• Checking distributions
• Finding outliers
• Comparing source data with dashboard data
• Creating custom analytical images
• Investigating trends before building the semantic model

Example workflow

6.8.56Common Matplotlib Mistakes

• Mistake 1 — Missing labels
• A chart without labels is difficult to interpret.
• Mistake 2 — Wrong chart type
• Don't use a pie chart for 20 categories.
• Mistake 3 — Too many colors
• Visual complexity can reduce readability.
• Mistake 4 — Overlapping labels

Use

plt.xticks(
    rotation=45
)

when necessary.

• Mistake 5 — Misleading axis limits
• Always ensure the scale represents the data honestly.
• Mistake 6 — Forgetting tight_layout()

For multiple charts

plt.tight_layout()

• helps prevent overlapping elements.
• Mistake 7 — Mixing plt state and object-oriented code carelessly
• For simple charts, plt is convenient.

For complex visualizations, prefer

fig, ax = plt.subplots()

6.8.57Matplotlib Cheat Sheet

import matplotlib.pyplot as plt

# Line

plt.plot(x, y)

# Bar

plt.bar(x, y)

# Horizontal bar

plt.barh(x, y)

# Scatter

plt.scatter(x, y)

# Histogram

plt.hist(x)

# Box plot

plt.boxplot(x)

# Pie

plt.pie(x)

# Area

plt.fill_between(x, y)

# Title

plt.title("Title")

# X label

plt.xlabel("X")

# Y label

plt.ylabel("Y")

# Legend

plt.legend()

# Grid

plt.grid()

# Axis limits

plt.xlim(0, 10)
plt.ylim(0, 100)

# Rotate labels

plt.xticks(rotation=45)

# Save

plt.savefig("chart.png")

# Display

plt.show()

6.8.58Mini Project — Sales Visualization Dashboard

Create a dataset

• Order_Date
• Region
• Product
• Quantity
• Revenue
• Profit
• Create the following charts.
• Chart 1 — Monthly Revenue

Use

Line chart

Question

How is revenue changing over time?

Chart 2 — Regional Revenue

Use

Bar chart

Question

Which region generates the most revenue?

Chart 3 — Revenue Distribution

Use

Histogram

Question

How are transaction values distributed?

Chart 4 — Revenue vs Profit

Use

Scatter plot

Question

Is higher revenue associated with higher profit?

Chart 5 — Profit by Region

Use

Box plot

Question

Which regions have greater variation and potential outliers in profit?

Chart 6 — KPI Dashboard

Create a 2 × 2 dashboard containing

• Revenue Trend
• Order Volume
• Profit Trend
• Revenue vs Profit

Save it as

sales_dashboard.png

6.8.59Interview Questions

• Basic
• What is Matplotlib?
• Why is Matplotlib used in data analysis?
• What is pyplot?
• What is the difference between Figure and Axes?
• How do you create a line chart?
• How do you create a bar chart?
• How do you create a histogram?
• How do you create a scatter plot?
• How do you create a box plot?
• How do you add titles and labels?

Intermediate

• How do you create multiple charts in one figure?
• What is plt.subplots()?
• What is the difference between plt.plot() and ax.plot()?
• How do you rotate x-axis labels?
• How do you add a legend?
• How do you add a reference line?
• How do you annotate a chart?
• How do you save a chart?
• How do you visualize missing values?
• How do you visualize outliers?

Advanced

• What is the object-oriented approach in Matplotlib?
• Why is it preferred for complex visualizations?
• How do you create a dashboard using Matplotlib?
• How do you visualize time-series data?
• How do you format large numbers on an axis?
• How do you handle overlapping labels?
• How would you visualize millions of data points?
• How would you create reusable plotting functions?
• How would you connect SQL data to Matplotlib?
• How would you use Matplotlib as part of an EDA workflow?

6.8.60Key Takeaways

The Matplotlib workflow is

DATA

│

▼

Prepare Data

│

▼

Create Figure

│

▼

Create Axes

│

▼

Plot Data

│

▼

┌───────────┼───────────┐

▼ ▼ ▼

Title Labels Legend

│ │ │

└───────────┼───────────┘

▼

Customize Axes

│

▼

Add Context

/ Annotations

│

▼

Save / Display

• Most important charts to master

The key principle is

A good visualization is not simply a chart that looks attractive; it is a chart that makes the underlying data and business insight easy to understand.

For a Data Analyst, the essential Matplotlib progression is

Line → Bar → Histogram → Scatter → Box Plot → Subplots → Business Dashboard → Advanced customization.

Seaborn is a Python data-visualization library built on top of Matplotlib. It is designed specifically for statistical data visualization and works especially well with Pandas DataFrames.

Seaborn makes it easier to create informative visualizations with less code than raw Matplotlib.

It is particularly useful for

• Exploratory Data Analysis (EDA)
• Statistical distributions
• Correlation analysis
• Categorical comparisons
• Regression analysis
• Multivariate analysis
• Heatmaps
• Business analytics

6.9.1Learning Objectives

By the end of this topic, you should be able to

• Understand Seaborn and its relationship with Matplotlib.
• Create statistical visualizations.
• Create distribution plots.
• Create categorical plots.
• Create relationship plots.
• Create regression plots.
• Create heatmaps.
• Visualize correlations.
• Work with Pandas DataFrames.
• Customize Seaborn charts.
• Build EDA visualizations.
• Create multi-chart analytical views.

6.9.2Installing Seaborn

Install

pip install seaborn

Import

import seaborn as sns

It is commonly imported as

sns

You will normally also import Matplotlib

import matplotlib.pyplot as plt

6.9.3Seaborn vs Matplotlib

A useful way to think about them

You can use both together.

6.9.4Basic Seaborn Example

import seaborn as sns
import matplotlib.pyplot as plt
tips = sns.load_dataset(
    "tips"
)

sns.scatterplot(

data=tips,
x="total_bill",
y="tip"

)

plt.show()

Seaborn automatically works with the DataFrame columns.

6.9.5Seaborn Dataset

Seaborn provides example datasets.

For example

tips = sns.load_dataset(
    "tips"
)

print(tips.head())

Other commonly used example datasets include

• tips
• iris
• penguins
• titanic
• flights
• car_crashes
• diamonds

These datasets are useful for learning visualization techniques.

6.9.6Understanding the data, x, and y Pattern

One of Seaborn's most useful features is

sns.scatterplot(

data=df,
x="Income",
y="Spending"

)

Instead of manually extracting

• df["Income"]
• df["Spending"]
• Seaborn understands the DataFrame structure directly.

6.9.7Scatter Plot

A scatter plot shows the relationship between two numerical variables.

sns.scatterplot(

data=tips,
x="total_bill",
y="tip"

)

plt.show()

Useful for identifying

• Correlation
• Clusters
• Outliers
• Nonlinear relationships

6.9.8Adding a Categorical Variable

Suppose we want to compare lunch and dinner.

sns.scatterplot(

data=tips,
x="total_bill",
y="tip",
hue="time"

)

plt.show()

hue separates observations by category.

6.9.9Using Size

You can encode another variable through point size

sns.scatterplot(

data=tips,
x="total_bill",
y="tip",
hue="time",
size="size"

)

plt.show()

This allows multiple dimensions to be represented in one chart.

6.9.10Using Style

Another variable can control marker style

sns.scatterplot(

data=tips,
x="total_bill",
y="tip",
hue="time",
style="sex"

)

plt.show()

Be careful not to encode too many variables at once because the chart can become difficult to read.

6.9.11Line Plot

Line charts are useful for trends.

sns.lineplot(

data=tips,
x="size",
y="tip"

)

plt.show()

For time-series data

sns.lineplot(

data=df,
x="Date",
y="Sales"

)

plt.show()

6.9.12Aggregation in Line Plots

Suppose multiple observations exist for each date.

Seaborn can aggregate values automatically in many cases

sns.lineplot(

data=df,
x="Month",
y="Sales"

)

Depending on the estimator and version/configuration, Seaborn can display an estimated central tendency and uncertainty.

For exact business KPI charts, however, it is often better to explicitly aggregate the data first:

monthly = (
df.groupby("Month", as_index=False)

.sum()

)

Then plot

sns.lineplot(

data=monthly,
x="Month",
y="Sales"

)

This makes the business metric definition explicit.

6.9.13Bar Plot

Bar plots compare categories.

sns.barplot(

data=tips,
x="day",
y="total_bill"

)

plt.show()

Seaborn can calculate an aggregate statistic for each category.

For business reporting, if you already have a summarized dataset, plot that directly to avoid confusion about what is being aggregated.

6.9.14Bar Plot with Hue

sns.barplot(

data=tips,
x="day",
y="total_bill",
hue="sex"

)

plt.show()

This compares groups within each category.

6.9.15Count Plot

A count plot shows how many observations belong to each category.

sns.countplot(

data=tips,
x="day"

)

plt.show()

Useful for questions such as

How many transactions occurred on each day?

6.9.16Count Plot with Hue

sns.countplot(

data=tips,
x="day",
hue="sex"

)

plt.show()

This allows you to compare category counts.

6.9.17Box Plot

Seaborn makes box plots very convenient.

sns.boxplot(

data=tips,
x="day",
y="total_bill"

)

plt.show()

Box plots show

• Median
• Q1
• Q3
• IQR
• Potential outliers

This connects directly to the Outlier Detection topic.

6.9.18Box Plot with Hue

sns.boxplot(

data=tips,
x="day",
y="total_bill",
hue="sex"

)

plt.show()

This allows distribution comparisons across multiple groups.

6.9.19Violin Plot

A violin plot combines ideas from a box plot and a density plot.

sns.violinplot(

data=tips,
x="day",
y="total_bill"

)

plt.show()

It shows

• Distribution shape
• Density
• Median
• Spread
• Useful when comparing distributions across groups.

6.9.20Strip Plot

A strip plot displays individual observations.

sns.stripplot(

data=tips,
x="day",
y="total_bill"

)

plt.show()

This is useful when you want to see the actual data points rather than only summary statistics.

6.9.21Swarm Plot

A swarm plot arranges individual observations to reduce overlap.

sns.swarmplot(

data=tips,
x="day",
y="total_bill"

)

plt.show()

It can become crowded for very large datasets.

6.9.22Combining Box and Strip Plots

A useful EDA technique

sns.boxplot(

data=tips,
x="day",
y="total_bill"

)

sns.stripplot(

data=tips,
x="day",
y="total_bill",
alpha=0.5

)

plt.show()

The box plot shows the summary while the strip plot shows individual observations.

6.9.23Histogram

Seaborn provides histplot().

sns.histplot(

data=tips,
x="total_bill"

)

plt.show()

Control bins

sns.histplot(

data=tips,
x="total_bill",
bins=20

)

6.9.24Histogram by Category

sns.histplot(

data=tips,
x="total_bill",
hue="sex",
bins=20

)

plt.show()

This allows distributions to be compared.

6.9.25KDE Plot

• KDE stands for Kernel Density Estimate.
• It provides a smooth representation of a distribution.
• sns.kdeplot(

data=tips,
x="total_bill"

)

plt.show()

KDE can be useful for comparing distribution shapes.

6.9.26KDE by Category

sns.kdeplot(

data=tips,
x="total_bill",
hue="sex"

)

plt.show()

This shows separate distributions for the categories.

6.9.27Histogram + KDE

sns.histplot(

data=tips,
x="total_bill",
kde=True

)

plt.show()

This provides both

Histogram

Smooth density estimate

6.9.28ECDF Plot

ECDF means Empirical Cumulative Distribution Function.

sns.ecdfplot(

data=tips,
x="total_bill"

)

plt.show()

It shows the proportion of observations less than or equal to each value.

6.9.29Regression Plot

Seaborn can easily display a regression relationship.

sns.regplot(

data=tips,
x="total_bill",
y="tip"

)

plt.show()

The chart contains

• Scatter points
• Regression line
• Confidence interval

This is useful for exploratory relationship analysis.

6.9.30Regression by Category

For categorical comparisons

sns.lmplot(

data=tips,
x="total_bill",
y="tip",
hue="sex"

)

plt.show()

This creates separate regression relationships by category.

6.9.31Heatmap

Heatmaps are one of Seaborn's most useful features.

Suppose

corr = df.corr(
    numeric_only=True
)

Create

sns.heatmap(

corr,

annot=True

)

plt.show()

This visualizes correlations between numerical variables.

6.9.32Understanding Correlation Heatmaps

Example

Revenue Profit Orders

Revenue 1.00 0.85 0.72

Profit 0.85 1.00 0.65

Orders 0.72 0.65 1.00

Interpretation

• +1 → Strong positive relationship
• 0 → Little linear relationship
• -1 → Strong negative relationship

Remember

Correlation does not prove causation.

6.9.33Heatmap with Formatting

sns.heatmap(

corr,

annot=True,
fmt=".2f"

)

plt.show()
fmt=".2f" displays values with two decimal places.

6.9.34Masking the Upper Triangle

Correlation matrices are symmetrical, so you can display only half.

import numpy as np
mask = np.triu(
    np.ones_like(
        corr,
        dtype=bool
    )
)

sns.heatmap(

corr,

mask=mask,
annot=True,
fmt=".2f"

)

plt.show()

This produces a cleaner correlation matrix.

6.9.35Pair Plot

• A pair plot displays relationships among several numerical variables.
• sns.pairplot(
• tips

)

plt.show()

It creates

• Scatter plots between variables
• Distribution plots along the diagonal
• Very useful during EDA.

6.9.36Pair Plot with Hue

sns.pairplot(

tips,

hue="sex"

)

plt.show()

This helps identify differences between groups.

6.9.37Joint Plot

A joint plot combines

• Relationship plot
• Marginal distributions
• sns.jointplot(

data=tips,
x="total_bill",
y="tip"

)

plt.show()

Regression

sns.jointplot(

data=tips,
x="total_bill",
y="tip",
kind="reg"

)

6.9.38FacetGrid

FacetGrid allows the same chart to be repeated across categories.

g = sns.FacetGrid(
    tips,
    col="time"
)

g.map_dataframe(

sns.scatterplot,

x="total_bill",
y="tip"

)

plt.show()

This is useful for comparing patterns across groups.

6.9.39FacetGrid by Rows and Columns

g = sns.FacetGrid(
    tips,
    row="sex",
    col="time"
)

g.map_dataframe(

sns.scatterplot,

x="total_bill",
y="tip"

)

plt.show()

This can create a grid such as

Lunch Dinner

Male [Chart] [Chart]

Female [Chart] [Chart]

6.9.40Catplot

catplot() provides a figure-level interface for categorical plots.

sns.catplot(

data=tips,
x="day",
y="total_bill",
kind="box"

)

plt.show()

Change the chart type

sns.catplot(

data=tips,
x="day",
y="total_bill",
kind="violin"

)

or

sns.catplot(

data=tips,
x="day",
kind="count"

)

6.9.41Relplot

relplot() provides a figure-level interface for relational charts.

sns.relplot(

data=tips,
x="total_bill",
y="tip",
hue="sex"

)

plt.show()

You can create multiple panels

sns.relplot(

data=tips,
x="total_bill",
y="tip",
col="time"

)

6.9.42Seaborn Themes

Seaborn provides high-level styling options.

For example

sns.set_theme(

style="whitegrid"

)

Common styles include

• white
• dark
• whitegrid
• darkgrid
• ticks

You can also set a context

sns.set_context(

"talk"

)

Common contexts include

• paper
• notebook
• talk
• poster

6.9.43Color Palettes

Seaborn provides many built-in palettes.

Example

sns.color_palette(

"deep"

)

For categorical plots

sns.barplot(

data=tips,
x="day",
y="total_bill",
palette="deep"

)

Use color intentionally. The goal should be clear communication rather than decoration.

6.9.44Sequential Palettes

Sequential palettes are useful when values represent magnitude.

Examples include

• Blues
• Greens
• Oranges
• Purples

Useful for

• Heatmaps
• Intensity
• Geographic values
• Ranking

6.9.45Diverging Palettes

Diverging palettes are useful when values have a meaningful midpoint, such as zero.

Example

Negative ← 0 → Positive

They are particularly useful for

• Correlation matrices
• Variance from target
• Positive/negative change

6.9.46Categorical Palettes

Categorical palettes distinguish discrete groups.

Examples

• Region
• Product
• Customer Segment
• Department

Each category receives a distinct visual representation.

6.9.47Seaborn with Pandas

Suppose

df = pd.DataFrame({
    "Region": [
        "South",
        "South",
        "North",
        "North"
    ],
    "Revenue": [
        10000,
        12000,
        15000,
        14000
    ]
})

Plot

sns.barplot(

data=df,
x="Region",
y="Revenue"

)

plt.show()

Seaborn understands the DataFrame structure directly.

6.9.48Statistical Estimation

Seaborn's categorical plots can calculate summary statistics.

For example

sns.barplot(

data=tips,
x="day",
y="total_bill"

)

The bar represents an aggregate estimate, and Seaborn can display uncertainty around the estimate.

For business dashboards, however, it is important to know exactly what the aggregation represents.

For example, if you want total revenue, calculate total revenue first

daily_sales = (
    df.groupby(
        "day",
        as_index=False
    )["Revenue"]
    .sum()
)

Then

sns.barplot(

data=daily_sales,
x="day",
y="Revenue"

)

6.9.49Seaborn for Outlier Detection

Seaborn is excellent for visualizing outliers.

sns.boxplot(

data=df,
x="Region",
y="Revenue"

)

plt.show()

Potential outliers appear beyond the whiskers.

You can combine this with the IQR calculations from the previous topic.

6.9.50Seaborn for Missing-Value Analysis

A simple visualization

missing = (
    df.isnull()
    .sum()
    .sort_values(
        ascending=False
    )
)

sns.barplot(

x=missing.values,
y=missing.index