Module 1

Python Programming

Core Python for AI/ML engineering — syntax, data structures, OOP, and the NumPy/Pandas foundations every later module builds on.

25 lessonsAI & MLHarinIT Academy
Module 1 · Lesson 1.1

Python Basics

1.1.1What is Python?

  • Python is a high-level, general-purpose programming language known for its simple syntax, readability, and extensive ecosystem of libraries.
  • Python is widely used in:
  • Software development
  • Data Engineering
  • Data Science
  • Machine Learning
  • Artificial Intelligence
  • Automation
  • Web development
  • API development
  • DevOps and MLOps
  • Generative AI and LLM applications

For an AI/ML engineer, Python is one of the most important programming languages to learn.

1.1.2Why is Python Popular in AI/ML?

Python provides powerful libraries and frameworks for almost every stage of an AI/ML project.

AreaPopular Python Libraries
Numerical ComputingNumPy
Data AnalysisPandas
VisualizationMatplotlib, Seaborn, Plotly
Machine LearningScikit-learn
Deep LearningTensorFlow, PyTorch
NLPNLTK, spaCy, Transformers
Generative AILangChain, LlamaIndex
APIsFastAPI, Flask
Data EngineeringPySpark
Cloud/ML OpsAzure SDK, MLflow

A typical ML project might look like:

Python
Pandas / NumPy
Data Cleaning
Visualization
Scikit-learn
Machine Learning Model
FastAPI
Docker
Azure

1.1.3Features of Python

1. Easy to Learn

Python syntax is relatively simple.

For example:

print("Hello, World!")

Compare this with languages that require more boilerplate code just to print a message.

2. Readable Syntax

Python code is designed to be easy to understand.

name = "Sreehari"
if name == "Sreehari":
    print("Welcome!")

The structure is clear even for someone who is new to programming.

3. Dynamically Typed

You don't normally need to specify a variable's type explicitly.

name = "Sreehari"
age = 37
salary = 25000.50

Python determines the type automatically.

name = "Sreehari"

Here, Python understands that name contains a string.

4. Interpreted

Python programs are generally executed through the Python interpreter.

For example:

print("Hello")
print("Welcome")

Python executes the statements as the program runs.

In practice, modern Python implementations first compile source code to bytecode and then execute it using the Python virtual machine, so "interpreted language" is a useful simplification rather than the whole implementation story.

5. Cross-Platform

  • Python programs can run on:
  • Windows
  • Linux
  • macOS

For example, the same basic Python program can run on Windows and Linux.

6. Large Ecosystem

Python has thousands of third-party packages.

Packages can be installed using pip.

pip install pandas

Then you can use the package:

import pandas as pd

1.1.4Installing Python

You can install Python from the official Python website.

Python.org

After installation, verify it from the command line:

python --version

Depending on your operating system, you may also use:

python3 --version

Example:

Python 3.12.5

The exact version will depend on what you have installed.

1.1.5Running Python

  • There are several ways to run Python.
  • Method 1 — Python Terminal
  • Open Command Prompt or PowerShell:
  • python
  • You may see:

>>>

This is the Python interactive interpreter, commonly called the REPL.

Try:

>>> 10 + 20

30

And:

>>> print("Hello Python")
  • Hello Python
  • Exit using:
  • exit()

1.1.6Python Script

  • Instead of typing commands one by one, you can create a Python file.
  • Create:
  • hello.py
  • Add:
print("Hello, Python!")

Run it:

python hello.py

Output:

Hello, Python!

Python source files normally use the .py extension.

1.1.7Your First Python Program

Let's create a slightly more useful program.

name = "Sreehari"
print("Hello", name)

Output:

Hello Sreehari

Another example:

name = "Sreehari"
age = 37
print("Name:", name)
print("Age:", age)

Output:

Name: Sreehari

Age: 37

1.1.8Comments in Python

  • Comments are notes written inside the code for humans.
  • Python ignores comments during normal execution.
  • Single-line comment
  • # This is a comment
print("Hello")

You can also place a comment after code:

name = "Sreehari"  # Store the user's name
  • Why use comments?
  • Comments can explain:
  • Why something is being done
  • What a complex section does
  • Important business logic
  • Temporary assumptions

However, don't comment every obvious line.

Bad:

# Store 10 in x

x = 10

Better:

# Maximum number of retry attempts allowed by the API

max_retries = 10

1.1.9The print() Function

print() displays information on the screen.
print("Hello")

You can print numbers:

print(100)

Multiple values:

name = "Sreehari"
age = 37
print(name, age)

Output:

Sreehari 37

You can also use multiple arguments:

print("Name:", name, "Age:", age)

1.1.10Strings

A string represents text.

name = "Sreehari"

You can use single quotes:

name = 'Sreehari'

or double quotes:

name = "Sreehari"

Both are valid.

Example:

message = "Welcome to Python"
print(message)

Output:

Welcome to Python

1.1.11Basic Arithmetic

Python can perform mathematical operations.

a = 10
b = 5
print(a + b)
print(a - b)
print(a * b)
print(a / b)

Output:

  • 15
  • 5
  • 50
  • 2.0
  • Other operators include:
print(10 // 3)
print(10 % 3)
print(10 ** 3)

Output:

  • 3
  • 1
  • 1000

These operators will be covered in greater detail in 1.3 Operators.

1.1.12Getting User Input

Python provides the input() function to accept user input.

name = input("Enter your name: ")
print("Hello", name)

If the user enters:

Sreehari

Output:

Hello Sreehari

Important

input() returns text.

For example:

age = input("Enter your age: ")
  • Even if the user enters:
  • 37
  • age is initially a string.
  • To convert it to an integer:
age = int(input("Enter your age: "))
print(age + 1)

Output:

38

Type conversion will be covered more thoroughly in 1.2 Variables & Data Types.

1.1.13Python Indentation

Indentation is extremely important in Python.

Consider:

age = 25
if age >= 18:
    print("Adult")

The indentation tells Python that the print() statement belongs to the if block.

Incorrect indentation can cause an error:

age = 25
if age >= 18:
    print("Adult")

Python will raise an indentation-related error.

A common convention is 4 spaces per indentation level.

1.1.14Python is Case-Sensitive

Python treats uppercase and lowercase letters differently.

These are different variables:

name = "Sreehari"
Name = "Hari"
NAME = "Sree"

Therefore:

print(name)
print(Name)
print(NAME)

produces three different values.

1.1.15Python Naming Conventions

Use meaningful names.

Good:

customer_name = "Ravi"
total_sales = 50000
employee_count = 25

Avoid unclear names:

x = "Ravi"
a = 50000
n = 25
  • unless the short name has a clear purpose, such as a mathematical formula or loop.
  • Python convention: snake_case
  • customer_name
  • total_amount
  • employee_count
  • Classes normally use PascalCase:
  • Customer
  • SalesReport
  • DataProcessor

You will learn this more deeply in the OOP section.

1.1.16Python Keywords

Python reserves certain words for specific purposes.

Examples:

if
else
elif
for
while
def
class
return

import

from

try
except
  • True
  • False
  • None

You cannot normally use these as variable names.

Incorrect:

class = "Python"

Correct:

class_name = "Python"

You can inspect Python's keywords:

import keyword
print(keyword.kwlist)

1.1.17Python Expressions

  • An expression produces a value.
  • 10 + 20
  • Result:
  • 30
  • Another example:
salary = 50000
bonus = 10000
total = salary + bonus
  • Here:
  • salary + bonus
  • is an expression.

1.1.18Python Statements

A statement is an instruction executed by Python.

Example:

name = "Sreehari"

Assignment is a statement.

Another:

print(name)

Function call statement.

Later you will learn statements involving:

if
for
while
def
class
try

1.1.19A Simple Real-World Example

Suppose we want to calculate the total sales of a business.

product_price = 1500
quantity = 5
total_sales = product_price * quantity
print("Product Price:", product_price)
print("Quantity:", quantity)
print("Total Sales:", total_sales)

Output:

  • Product Price: 1500
  • Quantity: 5
  • Total Sales: 7500

This simple example introduces concepts that will become important in Data Science and Machine Learning.

1.1.20Python in Data Engineering

Since Python is heavily used in data engineering, you will eventually encounter code such as:

import pandas as pd
df = pd.read_csv("sales.csv")
print(df.head())

The flow is:

CSV File
Python
Pandas
DataFrame
Data Cleaning
Analysis
Machine Learning

Don't worry about understanding Pandas yet. That comes later.

1.1.21Python in Machine Learning

A simple ML program might eventually look like:

from sklearn.linear_model import LinearRegression
model = LinearRegression()

model.fit(X_train, y_train)

prediction = model.predict(X_test)

This is where your Python fundamentals become useful.

Before learning ML, you should be comfortable with:

  • Variables
  • Data types
  • Conditions
  • Loops
  • Functions
  • Lists
  • Dictionaries
  • Modules
  • Exceptions
  • Classes
  • File handling

1.1.22Python Execution Flow

Consider:

name = "Sreehari"
print("Welcome", name)
age = 37
print("Age:", age)

Python executes the instructions in sequence:

Start
Create name
Print name
Create age
Print age
End

Later, control-flow statements such as if, for, and while allow the execution path to change.

1.1.23Common Beginner Mistakes

Mistake 1 — Missing quotes

Incorrect:

name = Sreehari

Correct:

name = "Sreehari"
  • Mistake 2 — Incorrect capitalization
  • Incorrect:
  • Print("Hello")
  • Correct:
print("Hello")
  • Python is case-sensitive.
  • Mistake 3 — Incorrect indentation
  • Incorrect:
if age > 18:
    print("Adult")

Correct:

if age > 18:
    print("Adult")

Mistake 4 — Treating input as a number

Incorrect:

age = input("Enter age: ")
print(age + 1)

This can produce a type error because input() returns a string.

Correct:

age = int(input("Enter age: "))
print(age + 1)

1.1.24Practice Exercises

Exercise 1

  • Write a program that prints:
  • Name: Sreehari
  • Age: 37
  • Profession: Data Engineer

Exercise 2

Create two variables:

a = 100
b = 25
  • Print:
  • Addition
  • Subtraction
  • Multiplication
  • Division

Exercise 3

Ask the user for their name and print:

Welcome, <name>

Exercise 4

  • Ask the user for:
  • Product price
  • Quantity
  • Calculate the total amount.

Example:

  • Product price: 500
  • Quantity: 4
  • Total: 2000
  • Exercise 5 — Data Engineering Practice
  • Create:
table_name = "CUSTOMER"
record_count = 150000
status = "SUCCESS"
  • Print:
  • Table: CUSTOMER
  • Records: 150000
  • Status: SUCCESS

1.1.25Interview Questions

Beginner

1. What is Python?

Python is a high-level, general-purpose programming language known for readable syntax and a large ecosystem of libraries.

2. Why is Python popular in AI and Machine Learning?

Because of its simple syntax and extensive ecosystem, including NumPy, Pandas, Scikit-learn, TensorFlow, PyTorch, and many NLP/GenAI libraries.

3. Is Python case-sensitive?

  • Yes.
  • name
  • Name
  • NAME
  • are different identifiers.

4. What is a Python interpreter?

It is the runtime system that executes Python programs. Modern CPython typically compiles source code to bytecode before execution.

5. What is indentation in Python?

Indentation defines the structure of blocks of code.

6. What is a .py file?

A .py file normally contains Python source code.

7. What does print() do?

It writes values to standard output.

8. What does input() do?

It reads user input and returns it as a string.

9. Is Python statically or dynamically typed?

Python is dynamically typed.

10. What is PEP 8?

PEP 8 is Python's widely used style guide for writing readable and consistent Python code.

1.1.26Key Takeaways

By the end of 1.1 Python Basics, you should understand:

  • Python
  • ├── Syntax
  • ├── Variables

├── print()

├── input()

  • ├── Comments
  • ├── Strings
  • ├── Basic arithmetic
  • ├── Indentation
  • ├── Case sensitivity
  • ├── Keywords
  • ├── Expressions
  • ├── Statements
  • └── Basic program execution
  • Your first milestone

You don't need to memorize Python syntax.

You should be able to write small programs without copying them, understand basic errors, and explain what each line of your program does.

Next lesson → 1.2 Variables & Data Types

Module 1 · Lesson 1.2

Variables & Data Types

1.2 Variables & Data Types

Variables and data types are among the most fundamental concepts in Python. Almost every Python program you write will use variables to store and process data.

For AI/ML and Data Engineering, understanding data types is especially important because your data will come from sources such as databases, CSV files, JSON APIs, Excel files, and cloud storage.

1.2.1What is a Variable?

A variable is a name that refers to a value stored by your program.

name = "Sreehari"
age = 37
salary = 25000

Here:

VariableValue
name"Sreehari"
age37
salary25000

Think of a variable as a label attached to a value.

name ───────► "Sreehari"

age ───────► 37

salary ─────► 25000

1.2.2Creating Variables

Python doesn't require you to declare the variable type separately.

name = "Sreehari"
age = 37
height = 5.10

Python determines the type automatically.

name = "Sreehari"    # str
age = 37             # int
height = 5.10        # float

This is called dynamic typing.

1.2.3Variable Assignment

The = operator assigns a value to a variable.
x = 10

Read this as:

Assign the value 10 to x.

You can change the value later:

x = 10
print(x)
x = 20
print(x)

Output:

10

20

The variable now refers to the new value.

1.2.4Multiple Variable Assignment

  • Python allows multiple variables to be assigned in one statement.
  • name, age, city = "Sreehari", 37, "Hyderabad"
  • Equivalent to:
name = "Sreehari"
age = 37
city = "Hyderabad"

You can also assign the same value to multiple variables:

a = b = c = 100
  • Now:
  • a → 100
  • b → 100
  • c → 100

1.2.5Swapping Variables

Python makes swapping values very easy.

a = 10
b = 20

a, b = b, a

print(a)
print(b)

Output:

20

  • 10
  • In many other languages, you might need a temporary variable.
  • Python doesn't require one here.

1.2.6Variable Naming Rules

  • Python variable names must follow certain rules.
  • Rule 1 — Must start with a letter or underscore
  • Valid:
name = "Sreehari"
_age = 37

Invalid:

1name = "Sreehari"
  • Rule 2 — Cannot contain spaces
  • Invalid:
  • customer name = "Ravi"
  • Use:
customer_name = "Ravi"

Rule 3 — Can contain letters, numbers and underscores

Valid:

customer1 = "Ravi"
customer_1 = "Ravi"
total_sales_2026 = 50000

Rule 4 — Cannot use Python keywords

Invalid:

class = "Customer"

Valid:

class_name = "Customer"

1.2.7Naming Convention

Python commonly uses snake_case for variables.

Good:

customer_name = "Ravi"
total_sales = 50000
employee_count = 150
average_salary = 65000

Avoid:

customername = "Ravi"
TotalSales = 50000
employeeCount = 150

The latter may work, but snake_case is the standard Python convention for variables and functions.

1.2.8What is a Data Type?

A data type defines what kind of data a value represents and what operations can be performed on it.

For example:

age = 37

37 is an integer.

name = "Sreehari"

"Sreehari" is a string.

Python has several built-in data types.

1.2.9Major Python Data Types

The most important built-in types are:

CategoryData TypeExample
Numericint100
Numericfloat10.5
Numericcomplex3 + 4j
BooleanboolTrue
Textstr"Hello"
Sequencelist[10, 20, 30]
Sequencetuple(10, 20, 30)
Setset{10, 20, 30}
Mappingdict{"name": "Sreehari"}
SpecialNoneTypeNone

We'll look at each one.

1.2.10Integer — int

Integers are whole numbers without decimal values.

age = 37
count = 100
temperature = -5

You can perform mathematical operations:

a = 10
b = 3
print(a + b)
print(a - b)
print(a * b)

Output:

  • 13
  • 7
  • 30
  • Check the type:
age = 37
print(type(age))

Output:

<class 'int'>

1.2.11Floating-Point — float

Floating-point numbers contain decimal values.

price = 1500.50
temperature = 36.7
percentage = 85.5

Check:

price = 1500.50
print(type(price))

Output:

<class 'float'>

Example

price = 500.50
quantity = 3
total = price * quantity
print(total)

Output:

1501.5

1.2.12Complex Numbers — complex

Python supports complex numbers.

z = 3 + 4j
  • Here:
  • Real part = 3
  • Imaginary part = 4

You can access them:

z = 3 + 4j
print(z.real)
print(z.imag)

Output:

3.0

4.0

Complex numbers are useful in specialized mathematical and engineering applications, though they are not commonly used in everyday Data Science workflows.

1.2.13Boolean — bool

  • Boolean values represent logical states.
  • There are two Boolean values:
  • True
  • False

Example:

is_active = True
is_deleted = False

Check:

print(type(is_active))

Output:

<class 'bool'>

Booleans are heavily used in conditions.

age = 25
is_adult = age >= 18
print(is_adult)

Output:

True

1.2.14String — str

A string represents text.

name = "Sreehari"
city = "Hyderabad"
message = "Welcome to Python"

Strings can use single or double quotes:

name = 'Sreehari'

or:

name = "Sreehari"

Both are valid.

String Concatenation

You can combine strings using +.

first_name = "Sreehari"
last_name = "Mekala"
full_name = first_name + " " + last_name
print(full_name)

Output:

  • Sreehari Mekala
  • String Length
  • Use len():
name = "Sreehari"
print(len(name))

Output:

  • 8
  • String Indexing
  • Python indexes characters starting from 0.
name = "Python"

Positions:

P y t h o n

0 1 2 3 4 5

Therefore:

print(name[0])
print(name[1])
print(name[5])

Output:

  • P
  • y
  • n
  • Negative indexing starts from the end:
print(name[-1])

Output:

n

String operations will be covered in greater depth later.

1.2.15List — list

A list stores multiple values.

numbers = [10, 20, 30, 40, 50]

A list can contain different types:

data = [10, "Python", 25.5, True]
  • Lists are:
  • Ordered
  • Mutable
  • Allow duplicate values

Example:

numbers = [10, 20, 30]

numbers.append(40)

print(numbers)

Output:

[10, 20, 30, 40]

List operations will be covered in more detail later.

1.2.16Tuple — tuple

A tuple is similar to a list but is immutable.

coordinates = (10, 20)

You can access values:

print(coordinates[0])

Output:

10

But you cannot modify an existing tuple element:

coordinates[0] = 50

This raises an error.

Tuples are useful when you want a collection of values that should not be changed.

1.2.17Set — set

A set stores unique values.

numbers = {10, 20, 30, 40}

Duplicate values are removed:

numbers = {10, 20, 20, 30, 30}
print(numbers)

The result contains each value only once.

Sets are useful for operations such as:

  • Removing duplicates
  • Union
  • Intersection
  • Difference

1.2.18Dictionary — dict

A dictionary stores data as key-value pairs.

employee = {
    "name": "Sreehari",
    "age": 37,
    "role": "Data Engineer"
}

Access a value:

print(employee["name"])

Output:

Sreehari

Another example:

customer = {
    "customer_id": 101,
    "name": "Ravi",
    "city": "Hyderabad"
}

Dictionaries are extremely important in Python because JSON data, API responses, configuration objects, and many application structures map naturally to dictionaries.

1.2.19None — NoneType

None represents the absence of a value.

result = None

Check:

print(type(result))

Output:

<class 'NoneType'>

Example:

customer_address = None

This can mean that the address is currently unavailable.

In data engineering, you will frequently encounter missing or null values. Python's None is one way such absence can be represented.

1.2.20Checking Data Types with type()

The type() function tells you the type of an object.

x = 100
print(type(x))

Output:

<class 'int'>

Examples:

print(type(100))
print(type(10.5))
print(type("Python"))
print(type(True))
print(type([1, 2, 3]))
print(type((1, 2, 3)))
print(type({"a": 1}))

1.2.21Type Conversion

Sometimes you need to convert one data type into another.

This is called type casting or type conversion.

Common functions include:

int()
float()
str()
bool()
list()
tuple()
set()
dict()

String → Integer

age = "37"
age = int(age)
print(age)
print(type(age))

Output:

  • 37
  • <class 'int'>
  • Integer → String
age = 37
age_text = str(age)
print(type(age_text))

Output:

<class 'str'>

Integer → Float

x = 10
y = float(x)
print(y)

Output:

10.0

Float → Integer

x = 10.9
y = int(x)
print(y)

Output:

10

Notice that int() truncates the fractional portion here; it does not round 10.9 to 11.

1.2.22User Input and Type Conversion

Remember from the previous lesson:

age = input("Enter your age: ")
input() returns a string.

Therefore:

age = int(input("Enter your age: "))

is commonly used when you need an integer.

Example:

price = float(input("Enter product price: "))
quantity = int(input("Enter quantity: "))
total = price * quantity
print("Total:", total)

1.2.23Mutable vs Immutable Data Types

This is an important Python concept.

  • Mutable
  • A mutable object can be changed after it is created.
  • Examples:
  • list
  • dict
  • set

Example:

numbers = [10, 20, 30]

numbers[0] = 100

print(numbers)

Output:

[100, 20, 30]

  • Immutable
  • An immutable object cannot be changed after creation.
  • Examples:
  • int
  • float
  • bool
  • str
  • tuple
  • For example:
name = "Python"

You cannot directly change one character inside the string.

1.2.24Important Difference: Variable vs Object

Consider:

x = 10

It is more accurate to think of x as a name referring to an object rather than a box permanently containing the value.

x ───────► 10

Then:

x = 20

Now:

x ───────► 20

This mental model becomes important when you learn:

  • Mutable objects
  • References
  • Functions
  • Classes
  • Object-oriented programming

1.2.25id() Function

Python provides id() to identify an object during its lifetime.

x = 100
print(id(x))

You will get a number representing the object's identity.

For example:

140735...

The exact value is implementation-dependent and can change between executions.

1.2.26is vs ==

These are different concepts.

==

Checks whether two objects have equal values.

a = [1, 2, 3]
b = [1, 2, 3]
print(a == b)

Output:

True

is

Checks whether two names refer to the same object.

print(a is b)

Usually:

False

because a and b are separate list objects with the same contents.

For checking None, the preferred style is:

if value is None:
    print("No value")

rather than:

if value == None:
    print("No value")

1.2.27Variables in Data Engineering

Consider a pipeline monitoring program:

pipeline_name = "Customer_Load"
run_id = 10245
record_count = 150000
success = True
duration_minutes = 12.5

These represent different data types:

VariableValueType
pipeline_name"Customer_Load"str
run_id10245int
record_count150000int
successTruebool
duration_minutes12.5float

This is a realistic example of how Python variables are used in data engineering.

1.2.28Variables in Machine Learning

A machine learning program may contain:

learning_rate = 0.01
epochs = 100
model_name = "LinearRegression"
is_training = True

Different values require different data types.

You might also have:

features = [10.5, 20.2, 15.7]

and:

customer = {
    "age": 35,
    "income": 75000,
    "city": "Hyderabad"
}

These basic Python structures become the foundation for working with NumPy arrays, Pandas DataFrames, JSON, and ML datasets.

1.2.29Data Type Summary

Python Data Types
├── Numeric

│ ├── int

│ ├── float

│ └── complex

├── Boolean

│ └── bool

├── Text

│ └── str

├── Collections

│ ├── list

│ ├── tuple

│ ├── set

│ └── dict

└── Special

└── NoneType

1.2.30Practical Example

Let's combine several concepts.

employee_name = "Sreehari"
employee_id = 1001
salary = 85000.50
is_active = True
skills = ["Python", "SQL", "Azure"]
employee = {
    "name": employee_name,
    "id": employee_id,
    "salary": salary,
    "active": is_active,
    "skills": skills
}
print(employee)
  • This one example contains:
  • str
  • int
  • float
  • bool
  • list
  • dict

These are exactly the kinds of structures you'll encounter when working with real-world data.

1.2.31Practice Exercises

  • Exercise 1 — Employee
  • Create variables for:
  • Employee Name
  • Employee ID
  • Salary
  • Department
  • Is Active
  • Print their values and data types.
  • Exercise 2 — Sales
  • Create:
product_name = "Laptop"
price = 75000
quantity = 3
  • Calculate:
  • Total Sales
  • Exercise 3 — User Input
  • Ask the user for:
  • Name
  • Age
  • Salary
  • Convert age to int and salary to float.
  • Print their values and types.
  • Exercise 4 — Customer Dictionary
  • Create a dictionary containing:
  • Customer ID
  • Customer Name
  • City
  • Age
  • Is Active
  • Print each value.
  • Exercise 5 — Data Engineering
  • Create the following variables:
pipeline_name = "FACT_SALES_LOAD"
source_records = 250000
target_records = 249850
execution_time = 18.7
pipeline_success = True
  • Calculate the difference between source and target records.
  • Expected:
  • Record Difference: 150

1.2.32Interview Questions

1. What is a variable in Python?

A variable is a name that refers to an object/value.

2. Is Python statically typed?

No. Python is dynamically typed.

3. What is the difference between int and float?

int represents whole numbers, while float represents floating-point numbers.

10 # int

10.5 # float

4. What is a Boolean?

  • A Boolean represents a logical value:
  • True
  • False

5. What is the difference between a list and tuple?

A list is mutable, while a tuple is immutable.

6. What is a dictionary?

A dictionary stores key-value pairs.

{"name": "Sreehari", "age": 37}

7. What is type casting?

Type casting means converting a value from one data type to another.

age = int("37")

8. What does type() do?

It returns the type of an object.

9. What is None?

None represents the absence of a value and has type NoneType.

10. What is the difference between == and is?

== compares values for equality, while is checks object identity.

11. What are mutable objects?

Objects whose contents can be changed after creation, such as lists, dictionaries, and sets.

12. What are immutable objects?

Objects that cannot be modified after creation, such as strings, integers, floats, booleans, and tuples.

1.2.33What You Should Be Able to Do Now

Before moving to 1.3 Operators, you should be comfortable writing code like:

name = "Sreehari"
age = 37
salary = 85000.50
is_active = True
skills = ["Python", "SQL", "Azure"]
employee = {
    "name": name,
    "age": age,
    "salary": salary,
    "active": is_active,
    "skills": skills
}
print(employee)
print(type(age))
print(type(salary))

The key thing to remember is:

Variables hold references to objects, and every object has a data type that determines what operations can be performed on it.

Next: 1.3 Operators — arithmetic, comparison, logical, assignment, bitwise, membership, identity, and operator precedence.

Module 1 · Lesson 1.3

Operators

Operators are symbols or keywords used to perform operations on values and variables.

For example:

a = 10
b = 5
result = a + b
print(result)

Here, + is an operator.

Output:

15

Operators are essential in Python because they are used for:

  • Mathematical calculations
  • Comparing values
  • Making decisions
  • Combining conditions
  • Assigning values
  • Checking membership
  • Checking object identity
  • Working with bits

1.3.1Types of Python Operators

Python operators can be grouped into the following categories:

CategoryOperators
Arithmetic+, -, *, /, //, %, **
Assignment=, +=, -=, *=, /=, etc.
Comparison==, !=, >, <, >=, <=
Logicaland, or, not
Bitwise&, `
Membershipin, not in
Identityis, is not

Let's understand each one.

1.3.2Arithmetic Operators

Arithmetic operators are used for mathematical calculations.

Addition +

a = 10
b = 5
print(a + b)

Output:

15

It can also concatenate strings:

first_name = "Sreehari"
last_name = "Mekala"
print(first_name + " " + last_name)

Output:

Sreehari Mekala

Subtraction -

a = 10
b = 5
print(a - b)

Output:

5

Multiplication *

a = 10
b = 5
print(a * b)

Output:

50

It can also repeat strings:

print("Python " * 3)

Output:

Python Python Python

Division /

a = 10
b = 4
print(a / b)

Output:

  • 2.5
  • Python's / operator returns a floating-point result.
  • Even:
print(10 / 2)
  • produces:
  • 5.0
  • Floor Division //

Floor division returns the floor of the division result.

print(10 // 3)

Output:

  • 3
  • For positive numbers, this looks like integer division.
  • Be careful with negative values:
print(-10 // 3)

Output:

-4

  • because floor means rounding toward negative infinity.
  • Modulus %
  • The modulus operator returns the remainder.
print(10 % 3)

Output:

1

Because:

10 = 3 × 3 + 1

Practical example

Check whether a number is even:

number = 20
print(number % 2 == 0)

Output:

  • True
  • Exponentiation **
  • Used to calculate powers.
print(2 ** 3)

Output:

  • 8
  • Because:
  • 2 × 2 × 2 = 8
  • Another example:
square = 5 ** 2
cube = 5 ** 3
print(square)
print(cube)

Output:

25

125

1.3.3Arithmetic Operator Summary

OperatorMeaningExampleResult
+Addition10 + 313
-Subtraction10 - 37
*Multiplication10 * 330
/Division10 / 33.333...
//Floor division10 // 33
%Remainder10 % 31
**Power10 ** 31000

1.3.4Assignment Operators

Assignment operators are used to assign or update values.

Basic Assignment =

x = 10

This assigns 10 to x.

Addition Assignment +=

x = 10

x += 5

print(x)

Output:

15

This is equivalent to:

x = x + 5

Subtraction Assignment -=

x = 10

x -= 3

print(x)

Output:

7

Equivalent to:

x = x - 3

Multiplication Assignment *=

x = 10

x *= 3

print(x)

Output:

30

Division Assignment /=

x = 10

x /= 2

print(x)

Output:

5.0

Floor Division Assignment //=

x = 10

x //= 3

print(x)

Output:

3

Modulus Assignment %=

x = 10

x %= 3

print(x)

Output:

1

Exponentiation Assignment **=

x = 2

x **= 3

print(x)

Output:

8

1.3.5Assignment Operator Summary

OperatorEquivalent
=x = value
+=x = x + value
-=x = x - value
*=x = x * value
/=x = x / value
//=x = x // value
%=x = x % value
**=x = x ** value

1.3.6Comparison Operators

  • Comparison operators compare two values.
  • The result is always a Boolean:
  • True
  • False
Equal ==
a = 10
b = 10
print(a == b)

Output:

True

Not Equal !=

a = 10
b = 20
print(a != b)

Output:

True

Greater Than >

a = 20
b = 10
print(a > b)

Output:

True

Less Than <

a = 10
b = 20
print(a < b)

Output:

True

Greater Than or Equal >=

age = 18
print(age >= 18)

Output:

True

Less Than or Equal <=

age = 17
print(age <= 18)

Output:

True

1.3.7Comparison Operator Summary

OperatorMeaning
==Equal
!=Not equal
>Greater than
<Less than
>=Greater than or equal
<=Less than or equal

These operators become particularly important in conditional statements.

1.3.8Logical Operators

  • Logical operators combine or modify Boolean expressions.
  • Python has:
  • and
  • or
  • not
  • and

Returns True when both conditions are true.

age = 25
salary = 60000
print(age >= 18 and salary >= 50000)

Output:

True

If either condition is false:

age = 25
salary = 30000
print(age >= 18 and salary >= 50000)

Output:

False

Truth table

ABA and B
TrueTrueTrue
TrueFalseFalse
FalseTrueFalse
FalseFalseFalse

1.3.9or

Returns True if at least one condition is true.

age = 17
has_permission = True
print(age >= 18 or has_permission)

Output:

True

Truth table

ABA or B
TrueTrueTrue
TrueFalseTrue
FalseTrueTrue
FalseFalseFalse

1.3.10not

Reverses a Boolean value.

is_active = True
print(not is_active)

Output:

False

Another:

is_deleted = False
print(not is_deleted)

Output:

True

1.3.11Practical Logical Example

Suppose a customer is eligible for a premium service if:

Age is at least 18

AND income is at least ₹50,000

age = 30
income = 75000
eligible = age >= 18 and income >= 50000
print(eligible)

Output:

True

This type of logic is common in real-world applications.

1.3.12Bitwise Operators

Bitwise operators work at the binary representation level of integers.

Python supports:

&

|

^

~

<<

>>

These are important in some areas such as:

  • Low-level programming
  • Networking
  • Cryptography
  • Embedded systems
  • Performance-sensitive operations
  • Certain algorithmic problems
  • Bitwise AND &
a = 5
b = 3
print(a & b)

Binary:

5 = 101
3 = 011

---

  • 001
  • Result:
  • 1
  • Bitwise OR |
print(5 | 3)
  • Binary:
  • 101
  • 011

---

  • 111
  • Result:
  • 7
  • Bitwise XOR ^

XOR returns 1 when the corresponding bits are different.

print(5 ^ 3)

101

011

---

  • 110
  • Result:
  • 6
  • Bitwise NOT ~
print(~5)

Output:

-6

This can be surprising at first. Python integers use signed integer semantics, so ~x is equivalent to:

-x - 1

Therefore:

~5 = -6

Left Shift <<

print(5 << 1)

Output:

  • 10
  • Conceptually, the binary representation is shifted left by one position.
  • Right Shift >>
print(10 >> 1)

Output:

5

1.3.13Membership Operators

  • Membership operators check whether a value exists inside a collection.
  • Python provides:
  • in
  • not in
  • in
skills = ["Python", "SQL", "Azure"]
print("Python" in skills)

Output:

True

Another:

print("Java" in skills)

Output:

False

not in

print("Java" not in skills)

Output:

True

Membership operators work with strings too:

message = "Python is powerful"
print("Python" in message)

Output:

True

1.3.14Identity Operators

  • Identity operators check whether two variables refer to the same object.
  • Python provides:
  • is
  • is not

Example:

a = [1, 2, 3]
b = a
print(a is b)

Output:

  • True
  • Both names refer to the same list object.
  • But:
a = [1, 2, 3]
b = [1, 2, 3]
print(a == b)
print(a is b)

Output:

  • True
  • False
  • Why?

== → same value

is → same object

1.3.15== vs is

This is an important interview question.

a = [1, 2, 3]
b = [1, 2, 3]

Value comparison

a == b
  • Result:
  • True
  • because the contents are equal.
  • Identity comparison
  • a is b
  • Result:
  • False
  • because they are different list objects.
  • For None, use:
if value is None:
    print("No value")

1.3.16Operator Precedence

When multiple operators appear in an expression, Python follows a precedence order.

Consider:

result = 10 + 5 * 2

You might think:

(10 + 5) × 2 = 30

But multiplication has higher precedence:

10 + (5 × 2)

Therefore:

  • 20
  • Use Parentheses
  • When in doubt, use parentheses.
result = (10 + 5) * 2
  • Result:
  • 30
  • Without parentheses:
result = 10 + 5 * 2

Result:

20

1.3.17Common Precedence Order

A simplified order from higher to lower precedence is:

()

**

+x, -x, ~x

*, /, //, %

+, -

<<, >>

&

^

|

==, !=, <, <=, >, >=, is, is not, in, not in

  • not
  • and
  • or
  • You don't need to memorize the entire list immediately.
  • The most important beginner rule is:
  • Parentheses → Exponentiation → Multiplication/Division → Addition/Subtraction

1.3.18Short-Circuit Evaluation

Python's and and or operators can stop evaluating as soon as the final result is known.

Example:

x = 10
result = x > 5 or x / 0 > 1
print(result)

Output:

  • True
  • Python doesn't evaluate the second expression because the first condition is already True.
  • Similarly:
x = 0
result = x != 0 and 100 / x > 5

The second condition isn't evaluated because:

x != 0

is already False.

This behavior is called short-circuit evaluation.

1.3.19Operator Chaining

  • Python allows chained comparisons.
  • Instead of:
  • age >= 18 and age <= 60

you can write:

18 <= age <= 60

Example:

age = 35
print(18 <= age <= 60)

Output:

True

This is one of Python's convenient features.

1.3.20Operators with Strings

The + operator can concatenate strings.

first = "Hello"
second = "Python"
message = first + " " + second
print(message)

Output:

Hello Python

The * operator can repeat strings:

print("Hi " * 3)

Output:

Hi Hi Hi

Comparison operators can compare strings:

print("apple" == "apple")

Output:

True

1.3.21Operators with Lists

The + operator can combine lists:

a = [1, 2]
b = [3, 4]
print(a + b)

Output:

[1, 2, 3, 4]

The * operator can repeat lists:

print([1, 2] * 3)

Output:

[1, 2, 1, 2, 1, 2]

Membership also works:

numbers = [10, 20, 30]
print(20 in numbers)

Output:

True

1.3.22Operators in Data Engineering

Operators are used constantly in data engineering.

For example, suppose a pipeline loaded:

source_records = 100000
target_records = 99850

We can calculate the difference:

difference = source_records - target_records
print(difference)

Output:

150

We can calculate the load percentage:

load_percentage = (target_records / source_records) * 100
print(load_percentage)

Output:

99.85

We can check whether the load is complete:

is_complete = source_records == target_records
print(is_complete)

Output:

False

This is exactly the type of logic used in data validation and pipeline monitoring.

1.3.23Operators in Machine Learning

Operators are also fundamental in ML.

For example:

actual = 100
predicted = 95
error = actual - predicted
print(error)

Output:

5

Absolute error:

absolute_error = abs(actual - predicted)

Percentage error:

percentage_error = abs(actual - predicted) / actual * 100
print(percentage_error)

Output:

5.0

Machine learning algorithms contain huge numbers of mathematical operations like these.

1.3.24Practical Example — Employee Eligibility

Suppose an employee is eligible for a loan if:

  • Age ≥ 21
  • Salary ≥ ₹30,000
  • Employee is active
age = 30
salary = 60000
is_active = True
eligible = (
    age >= 21
    and salary >= 30000
    and is_active
)
print("Eligible:", eligible)

Output:

  • Eligible: True
  • This combines:
  • Comparison operators
  • Logical operators
  • Assignment
  • Parentheses

1.3.25Practical Example — Sales Validation

expected_records = 100000
actual_records = 99500
difference = expected_records - actual_records
percentage_loaded = (
    actual_records / expected_records
) * 100
is_complete = actual_records == expected_records
print("Difference:", difference)
print("Loaded %:", percentage_loaded)
print("Complete:", is_complete)

Output:

  • Difference: 500
  • Loaded %: 99.5
  • Complete: False

This is a useful example for your Data Engineering background.

1.3.26Practice Exercises

Exercise 1 — Calculator

Create:

a = 100
b = 25
  • Calculate:
  • Addition
  • Subtraction
  • Multiplication
  • Division
  • Floor division
  • Remainder
  • Power
  • Exercise 2 — Even or Odd

Ask the user for a number and determine whether it is even or odd.

Hint:

  • number % 2
  • Exercise 3 — Employee Eligibility
  • Create:
age = 35
salary = 75000
  • An employee is eligible if:
  • age >= 21
  • AND
  • salary >= 50000
  • Print the result.
  • Exercise 4 — Pipeline Validation
  • Create:
source_records = 250000
target_records = 249500
  • Calculate:
  • Record difference
  • Percentage loaded
  • Whether counts match
  • Exercise 5 — Membership
  • Create:
skills = ["Python", "SQL", "Azure", "Power BI"]
  • Check whether:
  • Python exists
  • Java exists
  • SQL exists
  • Exercise 6 — Range Validation
  • Create:
score = 85
  • Check whether the score is between 50 and 100 using a chained comparison.
  • Expected:
  • True

1.3.27Interview Questions

1. What are operators in Python?

Operators are symbols or keywords used to perform operations on values and objects.

2. What is the difference between / and //?

  • / performs true division and returns a floating-point result.
  • // performs floor division.
  • 10 / 3

# 3.333...

10 // 3

# 3

3. What does % do?

It returns the remainder of a division.

10 % 3

# 1

4. What is the difference between = and ==?

= is assignment.

x = 10

== compares values.

x == 10

5. What is the difference between == and is?

== compares equality of values.

is checks object identity.

6. What are logical operators?

  • Python provides:
  • and
  • or
  • not

7. What does and do?

It evaluates to a truthy result only when both operands satisfy the required truth condition; with Boolean operands, both must be True.

8. What does or do?

It evaluates to a truthy result when at least one operand is truthy.

9. What is short-circuit evaluation?

Python can stop evaluating a logical expression once its final result is already determined.

10. What are membership operators?

in

not in

They test membership in a collection or other supported container.

11. What are identity operators?

is

is not

They test whether two references point to the same object.

12. What is operator precedence?

  • Operator precedence determines the order in which operators are evaluated.
  • For example:
  • 10 + 5 * 2
  • is evaluated as:

10 + (5 * 2)

giving:

20

1.3.28Quick Revision

Arithmetic

+ - * / // % **

Comparison

== != > < >= <=

Logical

and or not

Assignment

= += -= *= /= //= %= **=

Bitwise

& | ^ ~ << >>

Membership

in not in

Identity

is is not

The most important distinction to remember

= → assign

== → compare values

is → compare object identity

in → check membership

Once these operators become comfortable, 1.4 Conditional Statements will be much easier because if, elif, and else rely heavily on comparison and logical operators.

Module 1 · Lesson 1.4

Conditional Statements

Conditional statements allow a Python program to make decisions.

  • In simple terms:
  • If a condition is true, execute one block of code; otherwise, execute another block.
  • For example:
age = 25
if age >= 18:
    print("Eligible to vote")

The condition age >= 18 is True, so Python executes the print() statement.

Conditional statements are extremely important in Data Engineering, Data Science, Machine Learning, and AI, because real-world programs constantly make decisions based on data.

1.4.1Why Do We Need Conditional Statements?

Without conditions, a program would execute the same instructions regardless of the data.

Consider a data pipeline:

record_count = 95000
  • You may want:
  • If record count is correct
  • → mark pipeline as successful
  • Otherwise
  • → mark pipeline as failed

Python allows us to implement this logic directly.

expected = 100000
actual = 100000
if actual == expected:
    print("Pipeline successful")
else:
    print("Pipeline failed")

Output:

Pipeline successful

1.4.2The if Statement

The simplest conditional statement is if.

Syntax

if condition:
    statement

Example:

age = 25
if age >= 18:
    print("Adult")

Output:

Adult

The colon : is important.

The statements belonging to the if block must be indented.

1.4.3Understanding the Flow

Consider:

age = 25
if age >= 18:
    print("Adult")

Execution:

age = 25

age >= 18?

True

print("Adult")

If the condition is false:

age = 15

age >= 18?

False
Skip the block

Example:

age = 15
if age >= 18:
    print("Adult")

Nothing is printed.

1.4.4Boolean Conditions

Conditions normally produce True or False.

age = 25
print(age >= 18)

Output:

True

Therefore:

if age >= 18:
    print("Adult")

is effectively asking:

Is age >= 18 true?

1.4.5if with Different Comparison Operators

You can use all comparison operators.

Greater than

salary = 70000
if salary > 50000:
    print("High salary")

Less than

age = 17
if age < 18:
    print("Minor")

Equal

status = "SUCCESS"
if status == "SUCCESS":
    print("Pipeline completed")

Not equal

status = "FAILED"
if status != "SUCCESS":
    print("Investigate pipeline")

1.4.6Indentation

Indentation is a fundamental part of Python syntax.

Correct:

age = 25
if age >= 18:
    print("Adult")

The print() statement belongs to the if block.

You can have multiple statements:

age = 25
if age >= 18:
    print("Adult")
print("Eligible for voting")
print("Decision completed")

All three statements are part of the if block.

1.4.7Nested Blocks

You can have additional indentation levels.

age = 25
if age >= 18:
    print("Adult")
if age >= 21:
    print("Above 21")

Execution:

if age >= 18
True
Print Adult
if age >= 21
True
Print Above 21

Nested conditions are useful, but excessive nesting can make code difficult to maintain.

1.4.8The else Statement

else executes when the if condition is false.

Syntax

if condition:
    statement
else:
    statement

Example:

age = 16
if age >= 18:
    print("Adult")
else:
    print("Minor")

Output:

Minor

1.4.9if-else Flow

Condition

┌──────┴──────┐

True False

│ │

if block       else block

│ │

└──────┬────────┘

Continue

Example:

marks = 65
if marks >= 50:
    print("Pass")
else:
    print("Fail")

Output:

Pass

1.4.10The elif Statement

elif means else if.

It allows you to check multiple conditions.

Syntax

if condition1:
    statement
elif condition2:
    statement
elif condition3:
    statement
else:
    statement

Example:

marks = 75
if marks >= 90:
    print("Grade A+")
elif marks >= 75:
    print("Grade A")
elif marks >= 60:
    print("Grade B")
else:
    print("Grade C")

Output:

Grade A

1.4.11How if-elif-else Works

Python evaluates conditions from top to bottom.

marks = 75
if marks >= 90:
    print("A+")
elif marks >= 75:
    print("A")
elif marks >= 60:
    print("B")
else:
    print("C")

Flow:

marks >= 90?

False

marks >= 75?

True
Print A
Stop checking remaining elif conditions

Once a condition is satisfied, Python executes that block and skips the remaining elif and else blocks.

1.4.12Multiple elif Conditions

You can have multiple elif blocks.

temperature = 35
if temperature >= 40:
    print("Very Hot")
elif temperature >= 30:
    print("Hot")
elif temperature >= 20:
    print("Normal")
elif temperature >= 10:
    print("Cool")
else:
    print("Cold")

Output:

Hot

1.4.13Logical Operators with Conditions

You can combine multiple conditions using:

  • and
  • or
  • not
  • Using and
age = 30
salary = 70000
if age >= 18 and salary >= 50000:
    print("Eligible")

Output:

Eligible

Both conditions must be true.

1.4.14Using or

role = "Admin"
if role == "Admin" or role == "Manager":
    print("Access Granted")

Output:

Access Granted

At least one condition must be true.

1.4.15Using not

is_blocked = False
if not is_blocked:
    print("User can access the system")

Output:

User can access the system

1.4.16Combining Multiple Conditions

Consider an employee eligibility rule:

age = 35
salary = 75000
experience = 6
if age >= 21 and salary >= 50000 and experience >= 3:
    print("Eligible")
else:
    print("Not Eligible")

Output:

Eligible

This is a common pattern in business applications.

1.4.17Nested if Statements

A condition can contain another condition.

username = "admin"
password_correct = True
if username == "admin":
    if password_correct:
        print("Login successful")
else:
    print("Incorrect password")
else:
    print("Unknown user")

Output:

Login successful

However

Sometimes nested conditions can be simplified.

Instead of:

if username == "admin":
    if password_correct:
        print("Login successful")

you can write:

if username == "admin" and password_correct:
    print("Login successful")

The second version is often easier to read.

1.4.18Conditional Statements with Strings

Conditions can compare strings.

status = "SUCCESS"
if status == "SUCCESS":
    print("Pipeline completed successfully")
else:
    print("Pipeline failed")

Output:

Pipeline completed successfully

You can also check membership:

role = "Data Engineer"
if "Data" in role:
    print("Data-related role")

Output:

Data-related role

1.4.19Conditional Statements with Lists

You can check whether an item exists in a list.

skills = ["Python", "SQL", "Azure"]
if "Python" in skills:
    print("Python skill available")

Output:

Python skill available

Using not in:

if "Java" not in skills:
    print("Java is not listed")

1.4.20Conditional Statements with Dictionaries

Example:

employee = {
    "name": "Sreehari",
    "department": "Data Engineering",
    "active": True
}

You can check a dictionary value:

if employee["active"]:
    print("Employee is active")

Output:

Employee is active

You can also check whether a key exists:

if "department" in employee:
    print("Department information available")

1.4.21Truthy and Falsy Values

  • Python doesn't require conditions to always explicitly produce True or False.
  • Some values are treated as truthy and some as falsy.
  • Common falsy values include:
  • False
  • None
  • 0
  • 0.0

""

[]

()

{}

set()

Example:

name = ""
if name:
    print("Name exists")
else:
    print("Name is empty")

Output:

Name is empty

Another example:

records = []
if records:
    print("Records available")
else:
    print("No records")

Output:

No records

This is very useful in Python.

1.4.22Checking Whether a List Contains Data

Instead of:

records = []
if len(records) > 0:
    print("Records available")

Python commonly uses:

if records:
    print("Records available")
else:
    print("No records")

This is more Pythonic.

1.4.23Conditional Expression

Python also supports a one-line conditional expression.

Normal form

if age >= 18:
    status = "Adult"
else:
    status = "Minor"

Conditional expression

status = "Adult" if age >= 18 else "Minor"

Example:

age = 25
status = "Adult" if age >= 18 else "Minor"
print(status)

Output:

Adult

This is sometimes called the ternary expression.

Use it for simple decisions, not complicated logic.

1.4.24pass Statement

Sometimes you need a block syntactically, but you don't want it to do anything yet.

Use pass.

age = 25
if age >= 18:
    pass

The program does nothing inside that block.

This can be useful while developing code incrementally.

1.4.25Conditional Statements with User Input

Example:

age = int(input("Enter your age: "))
if age >= 18:
    print("You are an adult")
else:
    print("You are a minor")

Example input:

Enter your age: 25

Output:

You are an adult

1.4.26Practical Example — Grade Calculator

marks = int(input("Enter marks: "))
if marks >= 90:
    grade = "A+"
elif marks >= 80:
    grade = "A"
elif marks >= 70:
    grade = "B"
elif marks >= 60:
    grade = "C"
elif marks >= 50:
    grade = "D"
else:
    grade = "F"
print("Grade:", grade)

Example:

Enter marks: 85

Grade: A

1.4.27Practical Example — Data Pipeline Validation

This is particularly relevant to your Data Engineering work.

Suppose:

expected_records = 100000
actual_records = 100000

We can validate the pipeline:

if actual_records == expected_records:
    print("Pipeline validation successful")
else:
    print("Pipeline validation failed")

Output:

Pipeline validation successful

1.4.28Pipeline Validation with Threshold

Suppose we allow a 1% difference.

expected_records = 100000
actual_records = 99500
difference = abs(expected_records - actual_records)
variance_percentage = difference / expected_records * 100
if variance_percentage <= 1:
    print("Pipeline within acceptable threshold")
else:
    print("Significant variance detected")

Output:

Pipeline within acceptable threshold

This demonstrates how Python conditions can be used for data quality validation.

1.4.29Practical Example — Pipeline Status

pipeline_status = "SUCCESS"
if pipeline_status == "SUCCESS":
    print("Pipeline completed successfully")
elif pipeline_status == "RUNNING":
    print("Pipeline is currently running")
elif pipeline_status == "FAILED":
    print("Pipeline failed")
else:
    print("Unknown pipeline status")

This is a realistic pattern for monitoring systems.

1.4.30Practical Example — ML Prediction

Imagine a classification model predicts whether a customer will churn.

churn_probability = 0.82
if churn_probability >= 0.8:
    print("High churn risk")
elif churn_probability >= 0.5:
    print("Medium churn risk")
else:
    print("Low churn risk")

Output:

High churn risk

In a real ML application, the probability would usually come from the model rather than being manually assigned.

1.4.31Practical Example — Data Quality

Suppose you want to validate customer data.

customer_name = "Ravi"
customer_age = 35
customer_email = "ravi@example.com"
if customer_name and customer_age > 0 and customer_email:
    print("Customer data is valid")
else:
    print("Customer data is invalid")

Output:

Customer data is valid

Notice the use of truthy/falsy values.

1.4.32Avoiding Deeply Nested Conditions

This can become difficult to read:

if user:
    if user["active"]:
        if user["role"] == "admin":
            print("Admin access")

Often you can simplify the logic:

if user and user["active"] and user["role"] == "admin":
    print("Admin access")

As your programs become larger, readable conditions become very important.

1.4.33Common Mistakes

Mistake 1 — Using = instead of ==

Incorrect:

if status = "SUCCESS":
    print("Success")

Correct:

if status == "SUCCESS":
    print("Success")

Remember:

= → assignment

== → comparison

Mistake 2 — Missing colon

Incorrect:

if age >= 18
print("Adult")

Correct:

if age >= 18:
    print("Adult")

Mistake 3 — Incorrect indentation

Incorrect:

if age >= 18:
    print("Adult")

Correct:

if age >= 18:
    print("Adult")

Mistake 4 — Incorrect elif ordering

Consider:

marks = 95
if marks >= 50:
    print("Pass")
elif marks >= 90:
    print("Excellent")

Output:

Pass

  • Why?
  • Because 95 >= 50 is already true.
  • Better:
if marks >= 90:
    print("Excellent")
elif marks >= 50:
    print("Pass")

Output:

Excellent

Order matters.

1.4.34if vs elif vs else

StatementPurpose
ifFirst condition
elifAdditional condition
elseRuns when previous conditions are false

Example:

if condition1:

...

elif condition2:

...

else:

...

Only one branch of this chain executes.

1.4.35Decision-Making Flow

A typical program might look like:

Start
Read input
Check condition

/ \

True False

↓ ↓

Action A Action B

\ /

\ /

End

With multiple conditions:

Start
Condition 1

/ \

True False

↓ ↓

Action A Condition 2

/ \

True False

↓ ↓

Action B Action C

1.4.36Practice Exercises

Exercise 1 — Positive, Negative or Zero

Write a program that accepts a number and prints:

  • Positive
  • Negative
  • Zero
  • Exercise 2 — Even or Odd
  • Ask the user for a number.
  • Print:
  • Even
  • or:
  • Odd
  • Exercise 3 — Largest Number
  • Given:
a = 25
b = 40
c = 15
  • Use conditional statements to find the largest number.
  • Expected:
  • Largest: 40
  • Exercise 4 — Grade Calculator
  • Accept marks and determine:
  • 90–100 → A+

80–89 → A

70–79 → B

60–69 → C

50–59 → D

  • Below 50 → F
  • Exercise 5 — Pipeline Status
  • Given:
status = "FAILED"

Print:

SUCCESS → Pipeline completed

RUNNING → Pipeline running

FAILED → Pipeline failed

Exercise 6 — Data Quality

Given:

source_count = 100000
target_count = 99500
  • Calculate the variance percentage.
  • If variance is:
  • <= 1% → PASS

> 1% → FAIL

Exercise 7 — Customer Churn

Given:

probability = 0.72
  • Classify:
  • >= 0.80 → High Risk
  • 0.50–0.79 → Medium Risk
  • < 0.50 → Low Risk

1.4.37Interview Questions

1. What is a conditional statement?

A conditional statement allows a program to execute different code depending on whether a condition is satisfied.

2. What is the difference between if, elif, and else?

if checks the first condition.
elif checks additional conditions.
else executes when none of the preceding conditions are true.

3. Can you have multiple elif statements?

Yes.

if condition1:

...

elif condition2:

...

elif condition3:

...

else:

...

4. Is else mandatory?

No.

You can write:

if age >= 18:
    print("Adult")

without an else.

5. Can you have an elif without an if?

No. An elif must follow an if or another elif.

6. What is a nested if?

An if statement placed inside another conditional block.

7. What is a ternary expression?

A compact way to write a simple if-else decision:

result = "Pass" if marks >= 50 else "Fail"

8. What are truthy and falsy values?

Python allows objects to be evaluated directly in Boolean contexts. Values such as 0, None, "", [], {}, and False are falsy; most other objects are truthy.

9. Why is indentation important?

Python uses indentation to define code blocks.

10. What happens when multiple elif conditions are true?

Python executes the first matching branch and skips the remaining branches.

1.4.38Key Takeaways

You should now understand:

if
if-else
if-elif-else
  • nested if
  • logical conditions
  • comparison conditions
  • truthy/falsy values
  • conditional expressions
  • pass
  • The basic pattern to remember is:
if condition:
    # execute when true
elif another_condition:
    # execute when first condition is false

# and this condition is true

else:
    # execute when all conditions are false
  • Most important concept
  • Conditional statements are where your Python programs start becoming decision-making programs.
  • For example:
if pipeline_success:
    print("Continue downstream processing")
else:
    print("Stop pipeline and raise alert")

That same fundamental concept will later appear in much more sophisticated forms in data validation, ML classification, model decision logic, API processing, and AI agents.

Next lesson: 1.5 Loops — for, while, range(), nested loops, break, continue, and practical data-processing examples.

Module 1 · Lesson 1.5

Loops

Loops are used to execute a block of code repeatedly.

Instead of writing the same code multiple times:

print("Hello")
print("Hello")
print("Hello")
print("Hello")
print("Hello")

you can use a loop:

for i in range(5):
    print("Hello")

Output:

  • Hello
  • Hello
  • Hello
  • Hello
  • Hello

Loops are extremely important for Data Engineering, Data Science, Machine Learning, and AI, because you frequently need to process multiple records, files, columns, API responses, predictions, or datasets.

1.5.1Why Do We Need Loops?

  • Suppose you have 1,000 customers.
  • Without loops, you would theoretically need to write processing logic 1,000 times.
  • With a loop:
customers = ["Ravi", "Kiran", "Anil", "Suresh"]
for customer in customers:
    print(customer)
  • Python automatically processes each customer.
  • Conceptually:
  • Customer 1 → Process
  • Customer 2 → Process
  • Customer 3 → Process
  • Customer 4 → Process

1.5.2Types of Loops in Python

Python primarily provides two loop statements:

1. for loop

Used when you want to iterate over a sequence or iterable.

for item in collection:

...

2. while loop

Used when you want to continue executing code while a condition remains true.

while condition:

...

  • Python also provides loop-control statements:
  • break
  • continue
  • pass

1.5.3The for Loop

The for loop is used to iterate over items in an iterable.

Basic syntax

for variable in iterable:
    statement

Example:

numbers = [10, 20, 30, 40]
for number in numbers:
    print(number)

Output:

  • 10
  • 20
  • 30
  • 40

1.5.4How a for Loop Works

Consider:

numbers = [10, 20, 30]
for number in numbers:
    print(number)

Execution:

numbers
10 → print
20 → print
30 → print
End

The variable number takes each value from the list one at a time.

1.5.5Looping Through a String

Strings are iterable.

name = "Python"
for character in name:
    print(character)

Output:

  • P
  • y
  • t
  • h
  • o
  • n
  • Each character is processed individually.

1.5.6Looping Through a List

skills = ["Python", "SQL", "Azure", "Power BI"]
for skill in skills:
    print(skill)

Output:

  • Python
  • SQL
  • Azure
  • Power BI

This is very common in Python programs.

1.5.7Looping Through a Tuple

numbers = (10, 20, 30)
for number in numbers:
    print(number)

Output:

  • 10
  • 20
  • 30

1.5.8Looping Through a Set

skills = {"Python", "SQL", "Azure"}
for skill in skills:
    print(skill)

The values are iterated over, but you should not rely on a particular output order for a set.

1.5.9Looping Through a Dictionary

Consider:

employee = {
    "name": "Sreehari",
    "age": 37,
    "role": "Data Engineer"
}

A basic loop iterates through keys:

for key in employee:
    print(key)

Output:

  • name
  • age
  • role
  • Loop Through Values
  • Use .values():
for value in employee.values():
    print(value)

Output:

  • Sreehari
  • 37
  • Data Engineer
  • Loop Through Keys and Values
  • Use .items():
for key, value in employee.items():
    print(key, ":", value)

Output:

  • name : Sreehari
  • age : 37
  • role : Data Engineer

This pattern is extremely useful when processing JSON data.

1.5.10The range() Function

range() is commonly used with for loops.
for i in range(5):
    print(i)

Output:

  • 0
  • 1
  • 2
  • 3
  • 4

Notice that 5 is not included.

1.5.11range(start, stop)

You can specify a starting value.

for i in range(1, 6):
    print(i)

Output:

  • 1
  • 2
  • 3
  • 4
  • 5
  • The stop value is exclusive.
  • Conceptually:
start = 1
stop  = 6

1, 2, 3, 4, 5

1.5.12range(start, stop, step)

You can also specify a step.

for i in range(1, 11, 2):
    print(i)

Output:

  • 1
  • 3
  • 5
  • 7
  • 9
  • Here:
start = 1
stop = 11
step = 2

1.5.13Counting Backwards

You can use a negative step.

for i in range(5, 0, -1):
    print(i)

Output:

  • 5
  • 4
  • 3
  • 2
  • 1
  • Another example:
for i in range(10, 0, -2):
    print(i)

Output:

  • 10
  • 8
  • 6
  • 4
  • 2

1.5.14Calculating a Sum with a Loop

numbers = [10, 20, 30, 40]
total = 0
for number in numbers:
    total = total + number
print(total)

Output:

100

A shorter form:

numbers = [10, 20, 30, 40]
total = 0
for number in numbers:
    total += number
print(total)

1.5.15Finding Even Numbers

numbers = [10, 15, 20, 25, 30]
for number in numbers:
    if number % 2 == 0:
        print(number)

Output:

  • 10
  • 20
  • 30

Notice that we're combining:

for loop
if condition

modulus operator

This combination is fundamental in programming.

1.5.16Finding Odd Numbers

numbers = [10, 15, 20, 25, 30]
for number in numbers:
    if number % 2 != 0:
        print(number)

Output:

15

25

1.5.17The while Loop

A while loop executes as long as a condition is true.

Syntax

while condition:
    statement

Example:

count = 1
while count <= 5:
    print(count)

count += 1

Output:

  • 1
  • 2
  • 3
  • 4
  • 5

1.5.18How a while Loop Works

count = 1

count <= 5?

True
print count
count += 1
Check condition again

...

count = 6
False
Stop

The important thing is that the loop condition eventually needs to become false.

1.5.19Infinite while Loops

Be careful.

This creates an infinite loop:

count = 1
while count <= 5:
    print(count)
  • Why?
  • Because count never changes.
  • The condition:
  • count <= 5
  • remains true forever.
  • Correct:
count = 1
while count <= 5:
    print(count)

count += 1

1.5.20for vs while

forwhile
Usually iterates over an iterableRuns while a condition is true
Useful when processing collectionsUseful when repetition depends on a condition
Common for datasets/lists/filesCommon for condition-based processing
Often has a predictable number of iterationsNumber of iterations may not be known beforehand

Example for:

for customer in customers:
    process(customer)

Example while:

while pipeline_running:
    check_status()

1.5.21The break Statement

break immediately terminates the loop.

Example:

for number in range(1, 11):
    if number == 5:
        break
print(number)

Output:

  • 1
  • 2
  • 3
  • 4

When number becomes 5, the loop stops.

1.5.22break Flow

1
2
3
4
5 → break
Exit loop

break is useful when you've found what you're looking for and don't need to continue.

1.5.23Practical break Example

Suppose you're looking for a particular customer:

customers = ["Ravi", "Kiran", "Anil", "Suresh"]
for customer in customers:
    if customer == "Anil":
        print("Customer found")

break

Output:

Customer found

Once the customer is found, there is no reason to continue searching.

1.5.24The continue Statement

continue skips the current iteration and moves to the next iteration.

Example:

for number in range(1, 6):
    if number == 3:
        continue
print(number)

Output:

  • 1
  • 2
  • 4
  • 5

When the number is 3, Python skips the print() statement.

1.5.25break vs continue

This is an important distinction.

break

Stops the entire loop.

Loop
break
Exit loop

continue

Skips the current iteration.

Loop
continue
Next iteration

1.5.26Practical Data Processing with continue

Suppose some records are invalid.

records = [100, 200, None, 300, None, 400]
for record in records:
    if record is None:
        continue
print(record)

Output:

  • 100
  • 200
  • 300
  • 400

This is a common data-processing pattern.

1.5.27The pass Statement

pass does nothing.

for number in range(5):
    pass

It is mainly useful as a placeholder when you need syntactically valid code but haven't implemented the logic yet.

1.5.28Nested Loops

A loop inside another loop is called a nested loop.

Example:

for i in range(3):
    for j in range(3):
        print(i, j)

Output:

  • 0 0
  • 0 1
  • 0 2
  • 1 0
  • 1 1
  • 1 2
  • 2 0
  • 2 1
  • 2 2

The inner loop completes all its iterations for every iteration of the outer loop.

1.5.29Understanding Nested Loops

Consider:

for i in range(2):
    for j in range(3):
        print(i, j)

Execution:

i = 0
j = 0
j = 1
j = 2
i = 1
j = 0
j = 1
j = 2

Therefore there are:

2 × 3 = 6

iterations of the inner loop.

1.5.30Multiplication Table

Nested loops aren't always necessary for simple tasks, but loops can generate tables easily.

number = 5
for i in range(1, 11):
    print(number, "x", i, "=", number * i)

Output:

  • 5 x 1 = 5
  • 5 x 2 = 10
  • 5 x 3 = 15
  • 5 x 4 = 20
  • 5 x 5 = 25
  • 5 x 6 = 30
  • 5 x 7 = 35
  • 5 x 8 = 40
  • 5 x 9 = 45
  • 5 x 10 = 50

1.5.31enumerate()

When you need both the index and value, enumerate() is very useful.

Instead of:

skills = ["Python", "SQL", "Azure"]
for i in range(len(skills)):
    print(i, skills[i])

Use:

skills = ["Python", "SQL", "Azure"]
for index, skill in enumerate(skills):
    print(index, skill)

Output:

  • 0 Python
  • 1 SQL
  • 2 Azure

You can start counting from 1:

for index, skill in enumerate(skills, start=1):
    print(index, skill)

Output:

  • 1 Python
  • 2 SQL
  • 3 Azure

This is a very Pythonic pattern.

1.5.32zip()

zip() lets you iterate over multiple iterables together.

Example:

names = ["Ravi", "Kiran", "Anil"]
scores = [80, 90, 75]
for name, score in zip(names, scores):
    print(name, score)

Output:

  • Ravi 80
  • Kiran 90
  • Anil 75

This is useful when two collections contain related data.

1.5.33Looping Through Two Lists

Without zip():

names = ["Ravi", "Kiran", "Anil"]
scores = [80, 90, 75]
for i in range(len(names)):
    print(names[i], scores[i])

With zip():

for name, score in zip(names, scores):
    print(name, score)

The second approach is generally clearer.

1.5.34Loop else

Python has an interesting feature: loops can have an else block.

Example:

for number in range(1, 6):
    print(number)
else:
    print("Loop completed")

Output:

  • 1
  • 2
  • 3
  • 4
  • 5
  • Loop completed

The loop's else runs when the loop finishes normally.

1.5.35Loop else with break

Consider:

for number in range(1, 6):
    if number == 3:
        break
print(number)
else:
    print("Loop completed")

Output:

1

  • 2
  • The else does not execute because the loop was terminated by break.
  • This feature is especially useful in search operations.

1.5.36Search Example Using Loop else

numbers = [10, 20, 30, 40]
target = 30
for number in numbers:
    if number == target:
        print("Found")

break

else:
    print("Not Found")

Output:

Found

If target were 50, the result would be:

Not Found

1.5.37Processing Files

Loops are commonly used to process files line by line.

with open("sales.txt") as file:
    for line in file:
        print(line.strip())

This is more memory-efficient than reading an enormous file into memory all at once.

You'll study file handling in detail in 1.10 File Handling.

1.5.38Processing API Results

Suppose an API returns a list of customers:

customers = [
    {"id": 101, "name": "Ravi"},
    {"id": 102, "name": "Kiran"},
    {"id": 103, "name": "Anil"}
]

You can process them:

for customer in customers:
    print(customer["id"], customer["name"])

Output:

  • 101 Ravi
  • 102 Kiran
  • 103 Anil

This pattern is extremely common in API and data-engineering applications.

1.5.39Data Engineering Example

Suppose a pipeline produces several table results:

tables = [
    {"name": "CUSTOMER", "source": 100000, "target": 100000},
    {"name": "ORDERS", "source": 250000, "target": 249900},
    {"name": "PRODUCT", "source": 50000, "target": 50000}
]

You can validate each table:

for table in tables:
    difference = table["source"] - table["target"]
if difference == 0:
    print(table["name"], "PASS")
else:
    print(table["name"], "FAIL - Difference:", difference)

Output:

  • CUSTOMER PASS
  • ORDERS FAIL - Difference: 100
  • PRODUCT PASS

This is a very realistic use of Python loops for data-quality validation.

1.5.40ML Example

Suppose you have predictions:

predictions = [0.92, 0.45, 0.78, 0.23]

You can classify them:

for probability in predictions:
    if probability >= 0.8:
        print("High Risk")
elif probability >= 0.5:
    print("Medium Risk")
else:
    print("Low Risk")

Output:

  • High Risk
  • Low Risk
  • Medium Risk
  • Low Risk
  • This combines:
  • List
for loop
if / elif / else

ML decision logic

1.5.41Accumulator Pattern

One of the most important loop patterns is the accumulator.

Example:

numbers = [10, 20, 30, 40]
total = 0
for number in numbers:
    total += number
print(total)

The variable total accumulates the result.

Another example:

sales = [1000, 2000, 1500, 3000]
total_sales = 0
for sale in sales:
    total_sales += sale
print(total_sales)

Output:

7500

1.5.42Counting Pattern

You can use a loop to count matching records.

numbers = [10, 20, 15, 30, 25]
count = 0
for number in numbers:
    if number >= 20:
        count += 1
print(count)

Output:

3

This means three values are greater than or equal to 20.

1.5.43Finding Maximum

numbers = [10, 50, 20, 80, 30]
maximum = numbers[0]
for number in numbers:
    if number > maximum:
        maximum = number
print(maximum)

Output:

80

Python also provides the built-in max() function, which is often preferable for this simple case:

print(max(numbers))

Learning the loop implementation is still valuable because it teaches the underlying logic.

1.5.44Finding Minimum

numbers = [10, 50, 20, 80, 30]
minimum = numbers[0]
for number in numbers:
    if number < minimum:
        minimum = number
print(minimum)

Output:

10

1.5.45Common Loop Mistakes

Mistake 1 — Forgetting to update a while variable

Bad:

count = 1
while count <= 5:
    print(count)

This produces an infinite loop.

Correct:

count = 1
while count <= 5:
    print(count)
  • count += 1
  • Mistake 2 — Wrong indentation
  • Incorrect:
for number in numbers:
    print(number)

Correct:

for number in numbers:
    print(number)

Mistake 3 — Incorrect range assumption

Remember:

range(5)
  • produces:
  • 0 1 2 3 4
  • not:
  • 0 1 2 3 4 5
  • Mistake 4 — Modifying a collection while iterating over it
  • This can produce unexpected behavior.

Instead of directly modifying a list while iterating over it, consider building a new list or iterating over a copy when appropriate.

For example, rather than:

numbers = [1, 2, 3, 4, 5]
for number in numbers:
    if number % 2 == 0:
        numbers.remove(number)

prefer:

numbers = [1, 2, 3, 4, 5]
numbers = [
    number for number in numbers
    if number % 2 != 0
]

You'll learn this more cleanly in 1.15 List Comprehensions.

1.5.46for Loop vs List Comprehension

Traditional loop:

numbers = [1, 2, 3, 4, 5]
squares = []
for number in numbers:
    squares.append(number ** 2)

List comprehension:

squares = [number ** 2 for number in numbers]

Both produce:

[1, 4, 9, 16, 25]

Don't worry about mastering comprehensions yet. They are covered later.

1.5.47Nested Loop Example — Matrix

Consider:

matrix = [
    [1, 2, 3],
    [4, 5, 6],
    [7, 8, 9]
]

To process every value:

for row in matrix:
    for value in row:
        print(value)

Output:

  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9

This type of nested iteration becomes useful when working with arrays and matrices.

1.5.48Practice Exercises

  • Exercise 1 — Print Numbers
  • Print numbers from 1 to 100 using a for loop.
  • Exercise 2 — Even Numbers
  • Print all even numbers between 1 and 50.
  • Exercise 3 — Sum
  • Calculate the sum of:
numbers = [10, 20, 30, 40, 50]
  • Expected:
  • 150
  • Exercise 4 — Count

Count how many numbers are greater than 50:

numbers = [20, 70, 45, 90, 30, 80]
  • Expected:
  • 3
  • Exercise 5 — Maximum

Find the largest number without using max():

numbers = [15, 72, 34, 91, 28]
  • Expected:
  • 91
  • Exercise 6 — Multiplication Table
  • Ask the user for a number and print its multiplication table from 1 to 10.
  • Exercise 7 — Reverse Counting
  • Use a while loop to print:
  • 10
  • 9
  • 8
  • 7
  • 6
  • 5
  • 4
  • 3
  • 2
  • 1
  • Exercise 8 — Skip Invalid Records
  • Given:
records = [100, None, 200, None, 300, 400]
  • Print only valid records using continue.
  • Exercise 9 — Pipeline Validation
  • Given:
tables = [
    {"name": "CUSTOMER", "source": 100000, "target": 100000},
    {"name": "ORDERS", "source": 250000, "target": 249000},
    {"name": "PRODUCT", "source": 50000, "target": 50000}
]
  • Print:
  • CUSTOMER → PASS
  • ORDERS → FAIL
  • PRODUCT → PASS
  • Exercise 10 — ML Predictions
  • Given:
predictions = [0.95, 0.23, 0.81, 0.62, 0.35]
  • Classify each prediction as:
  • >= 0.80 → High
  • >= 0.50 → Medium

< 0.50 → Low

1.5.49Interview Questions

1. What is a loop?

A loop repeatedly executes a block of code.

2. What types of loops does Python provide?

Primarily:

for
while

3. What is the difference between for and while?

A for loop is generally used to iterate over an iterable, while a while loop continues as long as a condition remains true.

4. What does range() do?

range() produces a sequence of integers commonly used for iteration.
range(5)

represents:

0, 1, 2, 3, 4

5. What does break do?

It immediately terminates the loop.

6. What does continue do?

It skips the current iteration and proceeds to the next iteration.

7. What does pass do?

It performs no operation and acts as a placeholder.

8. What is a nested loop?

A loop inside another loop.

9. What is an infinite loop?

A loop whose termination condition never becomes false.

Example:

while True:
    print("Running")

10. What is enumerate()?

It provides both the index and value while iterating.

for index, value in enumerate(items):
    print(index, value)

11. What is zip()?

It allows multiple iterables to be iterated over together.

12. What is an accumulator?

A variable that progressively collects a result during iteration.

Example:

total = 0
for number in numbers:
    total += number

13. Can a loop have an else block?

Yes. The loop's else executes when the loop terminates normally, but not when it is terminated by break.

1.5.50Key Takeaways

You should now understand:

for loop
while loop
range()
  • break
  • continue
  • pass
  • nested loops
  • enumerate()
  • zip()
  • loop else
  • accumulator patterns
  • counting patterns
  • data processing with loops

The most important patterns to remember are:

Iterate through data

for item in items:
    process(item)

Repeat while a condition is true

while condition:
    process()

Stop early

if condition:
    break

Skip one record

if invalid:
    continue

Process real-world data

for record in records:
    if record is None:
        continue

process(record)

For your AI/ML journey, loops are particularly important because they teach you the fundamental idea of repeating an operation over data. Later, libraries such as NumPy and Pandas will allow you to perform many operations without writing explicit Python loops, but understanding loops first is essential.

Next lesson: 1.6 Functions — defining functions, parameters, arguments, return values, scope, default arguments, keyword arguments, *args, **kwargs, and reusable Data Engineering/ML functions.

Module 1 · Lesson 1.6

Functions

A function is a reusable block of code that performs a specific task.

Instead of writing the same logic repeatedly, you write it once inside a function and call it whenever you need it.

For example:

def greet():
    print("Hello, Python!")
  • greet()
  • greet()
  • greet()

Output:

  • Hello, Python!
  • Hello, Python!
  • Hello, Python!

Functions are one of the most important concepts in Python because they help you write code that is:

  • Reusable
  • Organized
  • Easier to test
  • Easier to maintain
  • Easier to debug
  • Easier to scale

In Data Engineering and AI/ML, almost every real application is built using functions.

1.6.1Why Do We Need Functions?

Imagine you need to calculate the total sales several times.

Without a function:

price1 = 500
quantity1 = 3
total1 = price1 * quantity1
price2 = 1000
quantity2 = 5
total2 = price2 * quantity2
price3 = 750
quantity3 = 4
total3 = price3 * quantity3

There is repeated logic.

With a function:

def calculate_total(price, quantity):
    return price * quantity

Now:

total1 = calculate_total(500, 3)
total2 = calculate_total(1000, 5)
total3 = calculate_total(750, 4)

Much cleaner.

1.6.2Creating a Function

The basic syntax is:

def function_name():
    # function body

Example:

def greet():
    print("Hello!")
  • This defines the function.
  • It doesn't execute yet.
  • To execute it, call the function:
  • greet()

Output:

Hello!

1.6.3Understanding def

The keyword:

def

means:

Define a function.

Example:

def calculate_salary():
    print("Calculating salary")

Here:

def              → keyword

calculate_salary → function name

() → parameter list

: → start of function block

1.6.4Function Calling

Defining:

def greet():
    print("Hello")
  • doesn't run the code.
  • Calling:
  • greet()
  • executes it.
  • Think of it as:
Define
Store the function
Call
Execute the function

1.6.5Function with Parameters

Functions become much more useful when they accept input.

def greet(name):
    print("Hello", name)

Call it:

greet("Sreehari")

Output:

  • Hello Sreehari
  • Call it again:
  • greet("Ravi")
  • greet("Kiran")

Output:

Hello Ravi

Hello Kiran

The function is reusable with different inputs.

1.6.6Parameters vs Arguments

This is an important interview concept.

Consider:

def greet(name):
    print("Hello", name)
  • name is a parameter.
  • When you call:
  • greet("Sreehari")
  • "Sreehari" is an argument.
  • So:
  • Parameter → variable defined in the function

Argument → actual value passed to the function

1.6.7Multiple Parameters

A function can accept multiple parameters.

def add(a, b):
    print(a + b)

Call:

add(10, 20)

Output:

30

Another example:

def employee_details(name, age, department):
    print("Name:", name)
print("Age:", age)
print("Department:", department)
  • Call:
  • employee_details(
  • "Sreehari",
  • 37,
  • "Data Engineering"

)

1.6.8Return Values

A function can return a result using return.

Example:

def add(a, b):
    return a + b

Now:

result = add(10, 20)
print(result)

Output:

30

The function calculates the result and sends it back to the caller.

1.6.9print() vs return

This is one of the most important concepts to understand.

Using print()

def add(a, b):
    print(a + b)

This displays the result.

Using return

def add(a, b):
    return a + b

This sends the result back.

You can then use it:

result = add(10, 20)
print(result)

You can also perform another calculation:

result = add(10, 20)
final_result = result * 2
print(final_result)

Output:

60

A returned value can be stored, reused, passed to another function, or used in an expression.

1.6.10Function Without return

Consider:

def greet():
    print("Hello")

If you do:

result = greet()
print(result)

Output:

Hello

None

If a function doesn't explicitly return a value, Python returns None.

1.6.11Returning Multiple Values

Python allows a function to return multiple values.

def calculate(a, b):
    total = a + b
difference = a - b
return total, difference

Call:

total, difference = calculate(20, 5)

print(total)
print(difference)

Output:

25

15

Technically, Python returns these values together as a tuple.

1.6.12Default Parameters

A parameter can have a default value.

def greet(name="Guest"):
    print("Hello", name)

Without an argument:

greet()

Output:

  • Hello Guest
  • With an argument:
  • greet("Sreehari")

Output:

Hello Sreehari

The supplied argument overrides the default.

1.6.13Example of Default Parameters

def calculate_discount(price, discount=10):
    return price - (price * discount / 100)

Using the default:

print(calculate_discount(1000))

Output:

900.0

Using a custom discount:

print(calculate_discount(1000, 20))

Output:

800.0

1.6.14Keyword Arguments

You can pass arguments using parameter names.

def employee(name, age, department):
    print(name, age, department)

Instead of:

employee("Sreehari", 37, "Data Engineering")

you can write:

employee(

name="Sreehari",
age=37,
department="Data Engineering"

)

This improves readability.

You can also change the order:

employee(

department="Data Engineering",
name="Sreehari",
age=37

)

1.6.15Positional Arguments

When arguments are passed according to their position:

def employee(name, age):
    print(name, age)
  • employee("Sreehari", 37)
  • Here:
  • "Sreehari" → name

37 → age

These are positional arguments.

1.6.16Positional vs Keyword Arguments

  • Positional
  • employee("Sreehari", 37)
  • Keyword
  • employee(
name="Sreehari",
age=37

)

Both are valid.

Keyword arguments are often clearer when a function has several parameters.

1.6.17Mixing Positional and Keyword Arguments

This is valid:

def employee(name, age, department):
    print(name, age, department)

employee(

"Sreehari",

age=37,
department="Data Engineering"

)

But positional arguments must come before keyword arguments.

This is invalid:

employee(

name="Sreehari",
37,
department="Data Engineering"

)

1.6.18Variable-Length Arguments — *args

Sometimes you don't know how many positional arguments the function will receive.

Use *args.

def add_numbers(*numbers):
    total = 0
for number in numbers:
    total += number
return total

Now:

print(add_numbers(10, 20))

Output:

30

And:

print(add_numbers(10, 20, 30, 40))

Output:

100

Inside the function, numbers is a tuple.

1.6.19Understanding *args

Consider:

def calculate_total(*values):
    print(values)
  • Call:
  • calculate_total(10, 20, 30)
  • Output will be similar to:
  • (10, 20, 30)
  • So:
  • *args → collects extra positional arguments into a tuple

The name args is conventional, but not mandatory.

For example:

def calculate_total(*values):

...

works equally well.

1.6.20Variable-Length Keyword Arguments — **kwargs

When you don't know how many keyword arguments will be passed, use **kwargs.

def employee_details(**details):
    print(details)

Call:

employee_details(

name="Sreehari",
age=37,
department="Data Engineering"

)

The function receives a dictionary-like mapping of keyword arguments.

Conceptually:

{

  • "name": "Sreehari",
  • "age": 37,
  • "department": "Data Engineering"

}

1.6.21*args vs **kwargs

Feature*args**kwargs
ArgumentsPositionalKeyword
Inside functionTupleDictionary
Example10, 20, 30name="Sreehari"

Example:

def test(*args, **kwargs):
    print(args)
print(kwargs)
  • Call:
  • test(
  • 10,
  • 20,
name="Sreehari",
age=37

)

Conceptually:

args

→ (10, 20)

kwargs

→ {"name": "Sreehari", "age": 37}

1.6.22Function Scope

Variables created inside a function are generally local variables.

def calculate():
    x = 100
print(x)
  • calculate()
  • Here x exists inside the function's local scope.
  • Trying to access it outside:
print(x)

will normally cause a NameError.

1.6.23Local Variables

def employee():
    salary = 50000
print(salary)

employee()

salary is local to the function.

Each function call gets its own local execution context.

1.6.24Global Variables

A variable defined outside a function is in the surrounding/global scope.

company = "Praval Infotech"
def show_company():
    print(company)

show_company()

Output:

Praval Infotech

The function can read the global variable.

However, relying heavily on global mutable state is generally discouraged because it makes code harder to understand and test.

1.6.25Local vs Global

company = "ABC"
def display():
    department = "Data Engineering"
print(company)
print(department)

Here:

company → global scope

department → local scope

1.6.26The global Keyword

Python allows a function to explicitly refer to a global variable for assignment using global.

count = 0
def increment():
    global count

count += 1

increment()

print(count)

Output:

1

However, prefer returning values or using well-designed objects over excessive use of global.

1.6.27The nonlocal Keyword

nonlocal is used in nested functions when you want an inner function to modify a variable belonging to an enclosing function scope.

Example:

def outer():
    count = 0
def inner():
    nonlocal count

count += 1

return count
return inner

This is more advanced and becomes particularly useful when learning closures and decorators.

1.6.28Function Documentation

You can document a function using a docstring.

def calculate_total(price, quantity):
    """Calculate total price."""
return price * quantity

You can inspect it:

print(calculate_total.__doc__)

Output:

Calculate total price.

For production code, good documentation makes functions easier for other developers to understand.

1.6.29Type Hints

Python allows you to specify expected types using annotations.

def add(a: int, b: int) -> int:
    return a + b
  • Here:
  • a: int
  • b: int
  • → int
  • communicate expected types.

Type hints generally do not enforce types at runtime by themselves.

For example:

def greet(name: str) -> str:
    return "Hello " + name
  • Type hints improve:
  • Readability
  • IDE support
  • Static analysis
  • Maintainability
  • Team collaboration

1.6.30Practical Function — Data Engineering

Suppose you need to validate source and target record counts.

Instead of repeating:

source = 100000
target = 99500
difference = source - target

create a reusable function:

def validate_records(source, target):
    difference = source - target
if difference == 0:
    return "PASS"
return "FAIL"

Use it:

result = validate_records(100000, 100000)
print(result)

Output:

PASS

And:

result = validate_records(100000, 99500)
print(result)

Output:

FAIL

1.6.31Data Quality Function

Let's make the validation more useful.

def validate_records(source, target, threshold=1):
    difference = abs(source - target)
variance = difference / source * 100
if variance <= threshold:
    return "PASS"
return "FAIL"

Use:

print(validate_records(100000, 99500))

Output:

PASS

Because the variance is 0.5%.

You can change the threshold:

print(validate_records(100000, 98000, threshold=1))

Output:

FAIL

1.6.32Processing Multiple Tables

Now combine functions and loops.

def validate_table(table):
    source = table["source"]
target = table["target"]
if source == target:
    return "PASS"
return "FAIL"
tables = [
    {"name": "CUSTOMER", "source": 100000, "target": 100000},
    {"name": "ORDERS", "source": 250000, "target": 249000},
    {"name": "PRODUCT", "source": 50000, "target": 50000}
]
for table in tables:
    result = validate_table(table)
print(table["name"], result)

Output:

  • CUSTOMER PASS
  • ORDERS FAIL
  • PRODUCT PASS

This is an important programming pattern:

Function
Reusable business logic
Loop
Process many records

1.6.33Function Returning a Dictionary

Functions can return complex objects.

def calculate_metrics(source, target):
    difference = abs(source - target)
percentage = target / source * 100
return {
    "source": source,
    "target": target,
    "difference": difference,
    "percentage": percentage
}

Use:

metrics = calculate_metrics(100000, 99500)
print(metrics)

Output:

{

  • 'source': 100000,
  • 'target': 99500,
  • 'difference': 500,
  • 'percentage': 99.5

}

This pattern is very common in APIs and data-processing applications.

1.6.34Function Calling Another Function

Functions can call other functions.

def calculate_total(price, quantity):
    return price * quantity
def calculate_tax(amount):
    return amount * 0.18
total = calculate_total(1000, 2)
tax = calculate_tax(total)
print("Total:", total)
print("Tax:", tax)

Output:

Total: 2000

Tax: 360.0

This lets you break complex applications into smaller components.

1.6.35Functions and Data Pipelines

A production-style data pipeline might be conceptually divided into functions:

extract_data()
validate_data()
transform_data()
load_data()
send_notification()

Python:

def extract_data():

...

def validate_data(data):

...

def transform_data(data):

...

def load_data(data):

...

def send_notification(message):

...

Then:

data = extract_data()
if validate_data(data):
    data = transform_data(data)

load_data(data)

send_notification("Pipeline completed")

This is much easier to maintain than putting hundreds of lines into one giant function.

1.6.36Functions in Machine Learning

Machine Learning applications also use functions extensively.

For example:

def calculate_accuracy(actual, predicted):
    correct = 0
for a, p in zip(actual, predicted):
    if a == p:
        correct += 1
return correct / len(actual)

Then:

actual = [1, 0, 1, 1]
predicted = [1, 0, 0, 1]
accuracy = calculate_accuracy(actual, predicted)
print(accuracy)

Output:

0.75

Later, libraries such as Scikit-learn will provide optimized implementations, but understanding the underlying logic is valuable.

1.6.37Function Design Principle

A good function should generally have a clear responsibility.

Bad:

def process_everything():
    # Read database
  • # Clean data
  • # Train model
  • # Send email
  • # Generate report
  • # Upload to cloud

...

This function does too many unrelated things.

Better:

def extract_data():

...

def clean_data(data):

...

def train_model(data):

...

def generate_report(results):

...

def send_email(report):

...

This follows the principle of separation of responsibilities.

1.6.38Pure Functions

  • A pure function generally:
  • Produces the same output for the same inputs.
  • Doesn't modify external state.

Example:

def add(a, b):
    return a + b
  • For:
  • add(10, 20)
  • the result is always:
  • 30
  • Pure functions are easier to:
  • Test
  • Debug
  • Reuse
  • Reason about

1.6.39Side Effects

A function has a side effect when it changes something outside its local computation.

For example:

total = 0
def update_total(amount):
    global total
  • total += amount
  • The function modifies external state.
  • Another example:
def write_file(data):
    with open("output.txt", "w") as file:
        file.write(data)

Writing to a file is an external side effect.

Side effects aren't inherently bad; they simply need to be managed deliberately.

1.6.40Recap of Function Syntax

Basic function

def greet():
    print("Hello")

Function with parameters

def greet(name):
    print("Hello", name)

Function with return

def add(a, b):
    return a + b

Default parameter

def greet(name="Guest"):
    print(name)

Variable positional arguments

def add(*numbers):

...

Variable keyword arguments

def employee(**details):

...

Type hints

def add(a: int, b: int) -> int:
    return a + b

1.6.41Practice Exercises

Exercise 1 — Greeting Function

Create:

def greet(name):

...

  • Expected:
  • greet("Sreehari")
  • Hello Sreehari
  • Exercise 2 — Calculator
  • Create functions:
  • add()
  • subtract()
  • multiply()
  • divide()
  • Each function should accept two numbers and return the result.
  • Exercise 3 — Even Number
  • Create:
def is_even(number):

...

It should return True if the number is even.

Example:

print(is_even(20))
  • Expected:
  • True
  • Exercise 4 — Maximum
  • Create:
def find_max(numbers):

...

  • Find the largest number without using max().
  • Exercise 5 — Record Validation
  • Create:
def validate_records(source, target):

...

  • Return:
  • "PASS"
  • when the counts match and:
  • "FAIL"
  • otherwise.
  • Exercise 6 — Data Quality
  • Create:
def calculate_variance(source, target):

...

  • Return the percentage difference between source and target.
  • Exercise 7 — Pipeline Status
  • Create:
def get_pipeline_status(status):

...

  • Return:
  • SUCCESS → "Pipeline completed"
  • RUNNING → "Pipeline running"

FAILED → "Pipeline failed"

Exercise 8 — *args

Create:

def calculate_sum(*numbers):

...

  • It should work with:
  • calculate_sum(10, 20)
  • calculate_sum(10, 20, 30)
  • calculate_sum(10, 20, 30, 40, 50)
  • Exercise 9 — **kwargs
  • Create:
def display_employee(**details):

...

Call it with:

display_employee(

name="Sreehari",
age=37,
department="Data Engineering"

)

Exercise 10 — ML Function

Create:

def classify_probability(probability):

...

  • Rules:
  • >= 0.80 → High Risk
  • >= 0.50 → Medium Risk

< 0.50 → Low Risk

1.6.42Interview Questions

1. What is a function?

A function is a reusable block of code designed to perform a specific task.

2. How do you define a function?

Using the def keyword:

def function_name():

...

3. What is the difference between a parameter and an argument?

A parameter is defined in the function signature; an argument is the actual value passed during a call.

4. What does return do?

It sends a value back to the caller and ends the current function execution.

5. What happens if a function doesn't have a return statement?

It returns None implicitly.

6. What is a default parameter?

A parameter with a predefined value:

def greet(name="Guest"):

...

7. What is *args?

It collects additional positional arguments into a tuple.

8. What is **kwargs?

It collects additional keyword arguments into a dictionary.

9. What is variable scope?

Scope determines where a variable can be accessed.

10. What is a local variable?

A variable defined inside a function and normally available within that function's local scope.

11. What is a global variable?

A variable defined in the global/module scope.

12. What is a docstring?

A string literal used to document a function, class, or module.

Example:

def add(a, b):
    """Return the sum of two numbers."""
return a + b

13. What are type hints?

Annotations that communicate the expected types of parameters and return values.

def add(a: int, b: int) -> int:
    return a + b

14. What is a pure function?

A function that, given the same inputs, produces the same result and doesn't rely on or modify external state.

15. Why are functions important in Data Engineering?

They allow repeated processing logic—such as validation, transformation, API processing, and data-quality checks—to be encapsulated and reused.

1.6.43Key Takeaways

You should now understand:

Functions

├── def

  • ├── Function calls
  • ├── Parameters
  • ├── Arguments
  • ├── return
  • ├── Default parameters
  • ├── Positional arguments
  • ├── Keyword arguments

├── *args

├── **kwargs

  • ├── Local scope
  • ├── Global scope
  • ├── Docstrings
  • ├── Type hints
  • └── Pure functions
  • The most important pattern is:
def function_name(parameters):
    # processing
return result

For example:

def validate_pipeline(source, target, threshold=1):
    difference = abs(source - target)
variance = difference / source * 100
if variance <= threshold:
    return "PASS"
return "FAIL"

Then you can reuse it:

result = validate_pipeline(
    source=100000,
    target=99500
)
print(result)

This is the transition from writing individual Python statements to designing reusable Python components.

Next lesson: 1.7 Lambda Functions — anonymous functions, lambda, map(), filter(), sorted(), practical data-processing examples, and when lambda functions should and should not be used.

Module 1 · Lesson 1.7

Lambda Functions

1. Introduction

A Lambda Function is an anonymous (unnamed) function that can have any number of input arguments but only one expression.

Instead of writing:

def square(x):
    return x * x

You can write:

square = lambda x: x * x

Both perform exactly the same task.

3. Why Lambda Functions?

Suppose you only need a function once.

Creating:

def multiply(x):
    return x * 2
  • may be unnecessary.
  • Instead:
  • lambda x: x * 2
  • is shorter.
  • Lambda is useful when:
  • Sorting
  • Filtering
  • Mapping
  • Data analysis
  • GUI programming
  • Event handling

5. Function vs Lambda

Normal Function

def add(a, b):
    return a + b
  • Lambda
  • lambda a, b: a + b
  • Comparison
FeatureFunctionLambda
NameYesNo
Multiple StatementsYesNo
Return KeywordRequiredAutomatic
ReadabilityHighMedium
Best ForLarge LogicSmall Logic

6. Syntax

General Syntax

lambda arguments: expression

Example

  • lambda x: x + 5
  • Multiple Parameters
  • lambda a, b: a + b
  • Three Parameters
  • lambda a, b, c: a + b + c

7. Internal Working

Consider

square = lambda x: x * x

Internally Python creates something similar to:

def square(x):
    return x * x

Lambda is simply syntactic sugar.

8. Memory Model

Memory

square

  • Lambda Object
  • Parameter -> x
  • Expression
  • x*x

9. Execution Flow

Call

square(5)

x = 5
5 * 5
25
Return

10. Beginner Example

Problem

Find square.

square = lambda x: x * x
print(square(5))

Output

25

Line-by-Line

square = lambda x: x*x
  • Creates function.
  • square(5)
  • Calls function.
  • Returns
  • 25

11. Multiple Parameters

add = lambda a, b: a + b
print(add(10, 20))

Output

30

12. Returning Boolean

is_even = lambda x: x % 2 == 0
print(is_even(10))
print(is_even(7))

Output

True

False

13. Conditional Lambda

Python allows if-else.

largest = lambda a, b: a if a > b else b
print(largest(10, 50))

Output

50

14. Lambda with List

numbers = [1,2,3,4]
double = list(map(lambda x: x*2, numbers))
print(double)

Output

[2,4,6,8]

15. Lambda with map()

map() applies a function to every element.

Without Lambda

def square(x):
    return x*x
numbers=[1,2,3]
result=list(map(square,numbers))

Using Lambda

numbers=[1,2,3]
result=list(map(lambda x:x*x,numbers))
print(result)

Output

[1,4,9]

Flow Diagram

List

1 2 3
map()
lambda x:x*x
1 4 9

16. Lambda with filter()

Filter selects matching elements.

numbers=[1,2,3,4,5,6]
even=list(filter(lambda x:x%2==0,numbers))
print(even)

Output

[2,4,6]

Flow

1

False
2
True
Keep
4
Keep
6

Keep

17. Lambda with reduce()

Requires

from functools import reduce

Example

from functools import reduce
numbers=[1,2,3,4]
total=reduce(lambda x,y:x+y,numbers)
print(total)

Output

10

Execution

1+2=3
3+3=6
6+4=10

18. Sorting Using Lambda

Without Lambda

students=[
    ("John",90),
    ("Alex",70),
    ("Bob",85)
]

Sort by marks

students.sort(key=lambda student: student[1])

print(students)

Output

[

('Alex',70),

('Bob',85),

('John',90)

]

Descending

students.sort(

key=lambda x:x[1],
reverse=True

)

19. Sorting Dictionary

employees=[
    {"name":"John","salary":5000},
    {"name":"Alex","salary":7000},
    {"name":"Bob","salary":6000}
]

employees.sort(

key=lambda x:x["salary"]

)

Output

  • John
  • Bob
  • Alex

20. Lambda in Pandas

import pandas as pd
df=pd.DataFrame({
    "name":["A","B"],
    "salary":[5000,6000]
})
  • Increase salary
  • df["salary"]=df["salary"].apply(
  • lambda x:x*1.10

)

  • Result
  • 5500
  • 6600

21. Lambda with max()

employees=[
    ("John",5000),
    ("Alex",7000),
    ("Bob",6500)
]
highest=max(
    employees,
    key=lambda x:x[1]
)
print(highest)

Output

('Alex',7000)

22. Lambda with min()

lowest=min(
    employees,
    key=lambda x:x[1]
)

23. Lambda with sorted()

numbers=[5,2,8,1]
result=sorted(
    numbers,
    key=lambda x:x
)
print(result)

Output

[1,2,5,8]

24. Nested Lambda

multiply=lambda x:(lambda y:x*y)

Example

times5=multiply(5)
print(times5(10))

Output

50

25. Advanced Example

  • Sort employees by:
  • Department
  • Salary
employees=[
    ("HR",5000),
    ("IT",8000),
    ("HR",3000),
    ("IT",6000)
]

employees.sort(

key=lambda x:(x[0],x[1])

)

Output

  • HR 3000
  • HR 5000
  • IT 6000
  • IT 8000

26. Production Example

E-commerce Discount

products=[
    {"name":"Laptop","price":80000},
    {"name":"Phone","price":30000}
]
discount=list(
    map(
        lambda p:{
            "name":p["name"],
            "price":p["price"]*0.90
        },
        products
    )
)
print(discount)

Output

  • Laptop
  • 72000
  • Phone
  • 27000

27. Time Complexity

OperationComplexity
Lambda CallO(1)
map()O(n)
filter()O(n)
reduce()O(n)
sort() with lambdaO(n log n)

28. Best Practices

  • ✔ Keep lambda short.
  • ✔ Use for one-line expressions.
  • ✔ Prefer def for complex logic.
  • ✔ Use meaningful variable names.
  • ✔ Avoid nested lambdas unless necessary.

29. Common Mistakes

1. Writing complex lambdas

  • lambda x: if x > 5
  • Correct:
  • lambda x: x if x > 5 else 0

2. Multiple statements

lambda x:

print(x)
return x

A lambda can contain only a single expression.

3. Overusing lambda

If the function spans multiple operations or needs documentation, use def instead.

30. Security Considerations

  • Do not execute user-provided lambda expressions with eval().
  • Validate input data before applying lambda functions.
  • Prefer named functions when auditability and maintainability are important in enterprise code.

31. Enterprise Applications

  • Lambda functions are commonly used in:
  • Data preprocessing with Pandas
  • ETL pipelines
  • Sorting business records
  • Event-driven programming
  • GUI callbacks
  • Web frameworks
  • Machine Learning feature transformations
  • Cloud data processing (e.g., Spark, Dask)

32. Interview Questions

  • What is a lambda function?
  • Why is it called an anonymous function?
  • Can a lambda contain multiple statements?
  • What is the syntax of a lambda?
  • When would you choose def over lambda?
  • Explain the use of lambda with map().
  • Explain the use of lambda with filter().
  • What does reduce() do?
  • How do you sort a list of dictionaries using a lambda?
  • What are the limitations of lambda functions?

33. MCQs

A lambda function is:

A. A class

B. An anonymous function ✅

C. A module

D. A package

A lambda function can contain:

A. Multiple statements

B. One expression ✅

C. Loops only

D. Classes only

Which built-in function is commonly paired with lambda for transformation?

A. map() ✅

  • B. open()
  • C. print()
  • D. type()
  • (Continue with 12 more MCQs following the same style.)

34. Coding Exercises

  • Write a lambda to calculate the cube of a number.
  • Use map() with lambda to convert temperatures from Celsius to Fahrenheit.
  • Filter all strings longer than five characters.
  • Use reduce() to calculate the product of a list.
  • Sort employees by age.
  • Sort students by marks in descending order.
  • Find the highest-priced product using max() and a lambda.
  • Use apply() with lambda in a Pandas DataFrame.
  • Create a nested lambda that multiplies two numbers.
  • Rewrite a simple named function as a lambda and compare readability.

35. Mini Case Study – Employee Salary Processing

Scenario: An HR system stores employee information in a list of dictionaries. Management requests a 10% salary increase for all employees earning below ₹50,000.

Approach:

  • Use filter() to identify eligible employees.
  • Use map() with a lambda to apply the salary increase.
  • Sort the updated records by salary.

This demonstrates how lambda expressions can simplify data transformation pipelines while keeping the code concise.

36. Lesson Summary

  • In this lesson, you learned:
  • What lambda functions are and why they are useful.
  • The syntax and internal working of lambda expressions.
  • Differences between def and lambda.
  • Practical usage with map(), filter(), reduce(), sorted(), min(), and max().
  • Applications in data processing and enterprise software.
  • Best practices, limitations, interview questions, and hands-on exercises.

Key Takeaways

  • A lambda function is an anonymous function consisting of a single expression.
  • Use lambda for short, simple operations.
  • Prefer named functions (def) for complex business logic.

Lambda expressions are especially powerful when combined with higher-order functions such as map(), filter(), and sorted().

In production code, prioritize readability and maintainability over brevity.

Module 1 · Lesson 1.8

Recursion

1. Introduction

Recursion is one of the most important programming concepts.

A recursive function is a function that calls itself to solve a problem by breaking it into smaller subproblems.

Instead of solving the entire problem at once, recursion repeatedly solves a smaller version until it reaches a stopping condition.

Simple Definition

Recursion is a technique where a function solves a problem by calling itself with a smaller input until a stopping condition is reached.

2. Learning Objectives

After completing this lesson, you will be able to:

  • Understand recursion intuitively
  • Write recursive functions
  • Explain base cases and recursive cases
  • Trace recursive execution
  • Understand Python's call stack
  • Solve common recursive problems
  • Analyze time and space complexity
  • Decide when recursion is appropriate

3. Why Do We Need Recursion?

  • Many problems naturally break into smaller versions of themselves.
  • Examples include:
  • Factorial
  • Fibonacci
  • Binary Search
  • Tree Traversal
  • Directory Traversal
  • Graph Algorithms
  • Quick Sort
  • Merge Sort
  • Backtracking
  • Dynamic Programming

4. Real-World Analogy

Example 1: Russian Dolls (Matryoshka)

Imagine a Russian doll.

┌──────────────┐

│ Large Doll │

│ ┌──────────┐ │

│ │ Medium │ │

│ │ ┌──────┐ │ │

│ │ │Small │ │ │

│ │ └──────┘ │ │

│ └──────────┘ │

└──────────────┘

  • To reach the smallest doll:
  • Open the first doll.
  • Open the next doll.
  • Continue until there is no smaller doll.

Then return outward.

This is exactly how recursion works.

  • Example 2: Climbing Stairs
  • Suppose there are 10 stairs.
  • To reach stair 10:
Reach stair 10
Need stair 9
Need stair 8

...

Need stair 1
Reached

Then you return back:

1
2
3

...

10

5. Basic Syntax

def function(parameters):
    if stopping_condition:
        return value
return function(smaller_problem)
  • Notice two important parts:
  • Base Case
  • Recursive Case

6. Components of Recursion

  • Every recursive function must have two components.
  • Component 1 – Base Case
  • This tells Python when to stop.

Example

if n == 0:
    return
  • Without a base case, recursion never ends.
  • Component 2 – Recursive Case
  • This makes the recursive call.

Example

return function(n - 1)

Each recursive call must move closer to the base case.

7. First Recursive Program

Print numbers from 5 to 1.

def countdown(n):
    if n == 0:
        return
print(n)

countdown(n - 1)

countdown(5)

Output

  • 5
  • 4
  • 3
  • 2
  • 1
  • Step-by-Step Execution
  • Call
  • countdown(5)
  • Python executes
Print 5
countdown(4)

Next

Print 4
countdown(3)

Then

Print 3
countdown(2)

Then

Print 2
countdown(1)

Then

Print 1
countdown(0)
  • Base case reached.
  • Stop.
  • Execution Flow Diagram
countdown(5)
countdown(4)
countdown(3)
countdown(2)
countdown(1)
countdown(0)
STOP

8. What Happens Internally?

Consider

countdown(3)

  • Python does not replace one call with another.
  • Instead, it creates a new function call every time.
  • Each function call gets its own memory.
  • Call Stack
  • Python uses a Call Stack.
  • Think of it as a stack of plates.
  • Top
  • countdown(1)
  • countdown(2)
  • countdown(3)
  • Bottom
  • Whenever a function is called:
  • Python pushes it onto the stack.
  • Whenever it finishes:
  • Python pops it off.
  • Visual Call Stack
  • Suppose
  • countdown(3)

Step 1

Stack

countdown(3)

Step 2

countdown(2)

countdown(3)

Step 3

  • countdown(1)
  • countdown(2)
  • countdown(3)

Step 4

  • countdown(0)
  • countdown(1)
  • countdown(2)
  • countdown(3)
  • Base case reached.

Now Python removes functions one by one.

Stack Unwinding

countdown(0)
Return
countdown(1)
Return
countdown(2)
Return
countdown(3)
Finished

This process is called stack unwinding.

9. Example – Factorial

Mathematics

5!

=

5 × 4 × 3 × 2 × 1

Answer

120

Notice

5!

=

5 × 4!

Similarly

4!

=

4 × 3!

This naturally leads to recursion.

Recursive Formula

n!

=

n × (n−1)!

Stopping condition

0! = 1

Python Program

def factorial(n):
    if n == 0:
        return 1
return n * factorial(n - 1)
print(factorial(5))

Output

120

Dry Run

factorial(5)
5 × factorial(4)
5 × 4 × factorial(3)

5 × 4 × 3 × factorial(2)

5 × 4 × 3 × 2 × factorial(1)

5 × 4 × 3 × 2 × 1 × factorial(0)

Base Case

1

Now returns

1
1
2
6
24
120

Call Stack for Factorial

factorial(5)
factorial(4)
factorial(3)
factorial(2)
factorial(1)
factorial(0)

Now unwind

1
1
2
6
24
120

Memory Diagram

Stack

---------------------

factorial(0)

n = 0

---------------------

factorial(1)

n = 1

---------------------

factorial(2)

n = 2

---------------------

factorial(3)

n = 3

---------------------

factorial(4)

n = 4

---------------------

factorial(5)

n = 5

---------------------

Each function has its own local variable n.

10. Common Mistakes

Mistake 1 – No Base Case

def hello():
    print("Hello")
  • hello()
  • This never stops.
  • Eventually Python raises:
  • RecursionError:
  • maximum recursion depth exceeded
  • Mistake 2 – Base Case Never Reached
def count(n):
    if n == 0:
        return

count(n + 1)

Input

count(5)

Sequence

5
6
7
8
9
  • It moves away from the base case.
  • Infinite recursion.
  • Mistake 3 – Wrong Return Statement
  • Incorrect
def factorial(n):
    if n == 0:
        return 1

factorial(n - 1)

Correct

return n * factorial(n - 1)

Always return the recursive result when the algorithm requires it.

11. Best Practices

  • Always define a clear base case.
  • Ensure each recursive call moves closer to the base case.
  • Keep recursive functions focused on a single task.
  • Use meaningful parameter names.
  • Add comments for complex recursive logic.

Prefer iteration for very deep recursion when performance or recursion limits are concerns.

12. Lesson Summary

In this first part of recursion, you learned:

What recursion is.

  • Why recursion is useful.
  • The importance of the base case and recursive case.
  • How Python manages recursive calls using the call stack.
  • How to trace recursion using execution flow and stack diagrams.
  • How to write and understand recursive solutions such as countdown and factorial.
  • In the next part, we'll cover:
  • Fibonacci recursion
  • Recursive binary search
  • Recursion vs iteration
  • Tail recursion
  • Tree recursion
  • Backtracking
  • Memoization
  • Time and space complexity analysis
  • Enterprise applications of recursion
  • Interview questions, MCQs, and coding exercises.
Module 1 · Lesson 1.9

Modules & Packages

1. Introduction

As programs grow, placing all code in a single file becomes difficult to manage.

Imagine writing an Employee Management System with:

  • Employee logic
  • Payroll
  • Attendance
  • Reports
  • Database
  • Email
  • Logging
  • Would you place everything in one file?
  • employee_system.py
  • This quickly becomes thousands of lines long.
  • Professional software instead splits code into multiple files.
  • Each file performs one specific responsibility.
  • Python provides Modules and Packages to achieve this.

2. Learning Objectives

After completing this lesson, you will be able to:

  • Understand modules
  • Understand packages
  • Create reusable code
  • Import modules
  • Create your own packages
  • Understand Python's import mechanism
  • Build enterprise project structures

3. Why Do We Need Modules?

  • Suppose you create a calculator.
  • Without modules
  • calculator.py
  • add()
  • subtract()
  • multiply()
  • divide()
  • percentage()
  • square()
  • cube()
  • factorial()

...

  • 1000+ lines
  • Everything becomes difficult to maintain.
  • Instead,
  • math_operations.py
  • add()
  • subtract()
  • multiply()
  • divide()

Now another file

  • advanced_math.py
  • factorial()
  • cube()
  • square()
  • Each file performs one task.
  • Advantages:
  • Reusability
  • Maintainability
  • Better testing
  • Easier debugging
  • Team collaboration

4. What is a Module?

Definition

A Module is simply a Python file (.py) containing Python code.

Example

calculator.py

This single file is a module.

Example

# calculator.py

def add(a, b):
    return a + b
def subtract(a, b):
    return a - b
  • This file itself is called a module.
  • Real-World Analogy
  • Think of a toolbox.
  • Toolbox
  • ├── Hammer
  • ├── Screwdriver
  • ├── Wrench
  • ├── Pliers
  • Each tool performs one job.
  • Similarly,
  • Project
  • ├── calculator.py
  • ├── employee.py
  • ├── payroll.py
  • ├── reports.py
  • Each module performs one responsibility.

5. Internal Working of Modules

Suppose

import math

Python performs these steps:

Read import statement
Search module
Load module
Compile if required
Execute module
Store module in memory
Provide reference

Memory Diagram

Program

import math

Memory

math
Functions
  • sqrt()
  • pow()
  • sin()
  • cos()
  • log()
  • The module is loaded only once.
  • Future imports reuse the loaded module.

6. Built-in Modules

Python provides hundreds of built-in modules.

Examples

ModulePurpose
mathMathematics
randomRandom numbers
datetimeDates
statisticsStatistics
osOperating system
sysPython interpreter
jsonJSON processing
csvCSV files
pathlibFile paths
loggingLogging

Example

import math
print(math.sqrt(25))

Output

5.0

Another Example

import random
print(random.randint(1,10))

Possible Output

7

7. Creating Your Own Module

Suppose you create

calculator.py

def add(a,b):
    return a+b
def subtract(a,b):
    return a-b

Another file

main.py

import calculator
print(calculator.add(10,20))

Output

  • 30
  • Project Structure
  • Project
  • ├── calculator.py
  • └── main.py

8. Different Ways to Import

Method 1

import math
  • Usage
  • math.sqrt(16)
  • Method 2
  • Import specific function
from math import sqrt

Now

print(sqrt(16))
  • No need for
  • math.sqrt()
  • Method 3
  • Import multiple functions
from math import sqrt, factorial

Method 4

Import everything

from math import *
Now
  • sqrt(16)
  • factorial(5)
  • Why this is discouraged

It pollutes the namespace and can create name conflicts.

Example:

from math import *
from statistics import *
  • If both modules define the same function name, it becomes unclear which one is being used.
  • Method 5
  • Alias
import numpy as np
  • Usage
  • np.array([1,2,3])
  • Another example
import pandas as pd

9. Module Search Path

When Python executes

import calculator

How does it find the file?

Python searches in this order:

1. Current directory

2. Standard library

3. Installed packages

4. PYTHONPATH

5. Site-packages

  • If not found
  • ModuleNotFoundError
  • View Search Path
import sys
print(sys.path)

Example Output

[

'C:\\Projects',

'C:\\Python312',

...

]

10. The __name__ Variable

Every Python module automatically has a special variable:

__name__

Example

print(__name__)

If executed directly

__main__

  • If imported
  • calculator
  • (the module name)

11. The if __name__ == "__main__": Block

Consider

# calculator.py

def add(a,b):
    return a+b
print("Calculator Loaded")

If imported

import calculator

Output

Calculator Loaded

Sometimes we don't want this code to execute when imported.

Solution:

def add(a,b):
    return a+b
if __name__=="__main__":
    print("Running directly")

Now

Running

python calculator.py

Output

Running directly

But

import calculator
  • produces no output because the condition is false.
  • Internal Working
  • When Python starts
Current File
Assign

__name__

Is file executed directly?

Yes

__main__

Run main block

If imported

__name__

calculator
Skip main block

12. What is a Package?

A package is a directory that contains multiple related modules.

Example

employee_system/
├── employee.py
  • ├── payroll.py
  • ├── reports.py
  • ├── attendance.py

└── __init__.py

  • Here
  • employee_system
  • is a package.
  • Each file inside is a module.
  • Module vs Package
ModulePackage
Single .py fileFolder
Contains functions/classesContains modules
Small unitLarger organization
Example: math.pyExample: numpy

13. The __init__.py File

Traditionally, packages contain:

__init__.py

Example

employee_system/

├── __init__.py

  • ├── employee.py
  • ├── payroll.py
  • The file can be empty:

# __init__.py

Or it can initialize the package:

print("Employee Package Loaded")

In modern Python (3.3+), __init__.py is optional for namespace packages, but it is still widely used to initialize packages and control what they export.

14. Importing from Packages

Project

project/
├── main.py
  • └── employee_system/
  • ├── employee.py
  • └── payroll.py
  • Inside
  • employee.py
def display():
    print("Employee Module")

Main

from employee_system import employee

employee.display()

Output

Employee Module

15. Best Practices

  • ✔ One module should have one responsibility.
  • ✔ Use meaningful module names.
  • ✔ Avoid circular imports.
  • ✔ Prefer explicit imports over wildcard imports.
  • ✔ Group related modules into packages.
  • ✔ Use if __name__ == "__main__": for executable scripts.
  • ✔ Follow a clear project structure.
  • Lesson Summary
  • In this lesson, you learned:
  • What modules are and why they are useful.
  • How to create and import your own modules.
  • Different import styles and their trade-offs.
  • How Python searches for modules.

The purpose of __name__ and the if __name__ == "__main__": idiom.

What packages are and how they organize related modules.

In the next part, you'll explore advanced package organization, relative vs. absolute imports, installing third-party packages with pip, virtual environments, PyPI, namespace packages, and enterprise project layouts.

Module 1 · Lesson 1.10

File Handling

File handling is the process of creating, reading, writing, updating, and managing files using Python.

File handling is especially important in Data Engineering and Data Science, because data often arrives as:

  • CSV files
  • Excel files
  • JSON files
  • Text files
  • Log files
  • Configuration files
  • XML files
  • Parquet files
  • Database extracts

For example, a data pipeline may look like:

Source File
Read File using Python
Validate Data
Transform Data
Load into Database

1.10.1Why File Handling Is Important

  • Suppose you receive:
  • customers.csv
  • containing:
  • CustomerID,Name,Age
  • 101,Ravi,35
  • 102,Kiran,29
  • 103,Anil,42
  • Python can read this file:
with open("customers.csv", "r") as file:
    data = file.read()
print(data)

You can then process the data.

1.10.2Opening a File

  • Python provides the built-in open() function.
  • Basic syntax:
  • open(filename, mode)

Example:

file = open("data.txt", "r")

Here:

data.txt → filename

r → read mode

1.10.3File Modes

The most important file modes are:

ModeMeaning
rRead
wWrite
aAppend
xCreate
bBinary
tText

You can combine modes.

For example:

"rb"

means:

read + binary

1.10.4Read Mode — r

Use r when you want to read an existing file.

file = open("data.txt", "r")
content = file.read()
print(content)

file.close()

Important:

file.close()

releases the file resource.

1.10.5Using with open()

The preferred approach is:

with open("data.txt", "r") as file:
    content = file.read()
print(content)

You don't need to explicitly call:

  • file.close()
  • Python automatically closes the file when the with block finishes.
  • Recommended pattern
with open("data.txt", "r") as file:
    data = file.read()

This is the pattern you should generally use.

1.10.6Reading the Entire File

  • Suppose data.txt contains:
  • Python
  • SQL
  • Azure
  • Machine Learning
  • Use:
with open("data.txt", "r") as file:
    content = file.read()
print(content)

Output:

  • Python
  • SQL
  • Azure
  • Machine Learning

read() reads the entire file into a string.

1.10.7Reading a Specific Number of Characters

You can specify the number of characters:

with open("data.txt", "r") as file:
    content = file.read(10)
print(content)

For a large file, this allows you to read only part of the content.

1.10.8Reading One Line

Use readline():

with open("data.txt", "r") as file:
    line = file.readline()
print(line)

This reads one line at a time.

1.10.9Reading Multiple Lines

You can call readline() repeatedly:

with open("data.txt", "r") as file:
    line1 = file.readline()
line2 = file.readline()
print(line1)
print(line2)

1.10.10readlines()

readlines() reads all lines and returns them as a list.

with open("data.txt", "r") as file:
    lines = file.readlines()
print(lines)

Example result:

[

"Python\n",

"SQL\n",

"Azure\n"

]

Notice the \n.

1.10.11Iterating Through a File

For large files, a very useful pattern is:

with open("data.txt", "r") as file:
    for line in file:
        print(line.strip())

This processes the file line by line.

This is particularly useful for large log files because you don't have to load the entire file into memory.

1.10.12Why Line-by-Line Processing Matters

  • Imagine a log file is:
  • 10 GB
  • Doing:
content = file.read()

may require a large amount of memory.

Instead:

with open("application.log") as file:
    for line in file:
        process(line)

allows Python to process the file incrementally.

Conceptually:

10 GB File
Read Line 1
Process
Read Line 2
Process

...

This is an important Data Engineering concept.

1.10.13Writing to a File

Use mode:

"w"

Example:

with open("output.txt", "w") as file:
    file.write("Hello Python")

If output.txt doesn't exist, Python creates it.

If it already exists, its existing contents are replaced.

This is very important.

1.10.14Writing Multiple Lines

You can write multiple strings:

with open("skills.txt", "w") as file:
    file.write("Python\n")
  • file.write("SQL\n")
  • file.write("Azure\n")
  • The file contains:
  • Python
  • SQL
  • Azure

1.10.15Using writelines()

You can write multiple strings using writelines():

skills = [
    "Python\n",
    "SQL\n",
    "Azure\n"
]
with open("skills.txt", "w") as file:
    file.writelines(skills)

Note that writelines() does not automatically add newline characters.

Therefore:

skills = ["Python", "SQL", "Azure"]
  • would result in:
  • PythonSQLAzure
  • unless you include \n.

1.10.16Append Mode — a

Use a when you want to add content to the end of an existing file.

with open("log.txt", "a") as file:
    file.write("Pipeline completed\n")

Existing content remains.

New content is added at the end.

1.10.17Write vs Append

w

with open("log.txt", "w") as file:
    file.write("New log")

Existing content is overwritten.

a

with open("log.txt", "a") as file:
    file.write("Additional log")
  • Existing content remains and new content is appended.
  • Remember:
  • w → Replace
  • a → Add

1.10.18Create Mode — x

Use x to create a new file.

with open("new_file.txt", "x") as file:
    file.write("Hello")

If the file already exists, Python raises:

FileExistsError

This can be useful when you specifically want to prevent accidental overwriting.

1.10.19File Encoding

Text files have an encoding.

A common encoding is UTF-8.

You can explicitly specify it:

with open(
    "data.txt",
    "r",
    encoding="utf-8"
) as file:
    content = file.read()

For writing:

with open(
    "output.txt",
    "w",
    encoding="utf-8"
) as file:
    file.write("Hello")

Using explicit UTF-8 encoding is a good practice for portable applications.

1.10.20Handling Unicode

  • Suppose your file contains:
  • Hello
  • नमस्ते

తెలుగు

ಕನ್ನಡ

Use UTF-8:

with open(
    "languages.txt",
    "r",
    encoding="utf-8"
) as file:
    content = file.read()
print(content)

UTF-8 is a good default for modern text processing.

1.10.21File Paths

You can specify a relative path:

with open("data/customers.csv") as file:
    data = file.read()

Or an absolute path.

On Windows:

with open(
    r"C:\Data\customers.csv",
    "r"
) as file:
    data = file.read()

The r before the string creates a raw string, which helps avoid backslash escape issues.

1.10.22Windows Paths

This can cause problems:

path = "C:\new\data.txt"

because sequences such as \n can be interpreted as escape characters.

Better:

path = r"C:\new\data.txt"

Or:

path = "C:\\new\\data.txt"

An even better modern approach is using pathlib.

1.10.23pathlib

pathlib provides an object-oriented way to work with paths.

from pathlib import Path
path = Path("data/customers.csv")
print(path)

Check whether it exists:

if path.exists():
    print("File exists")

This is a very useful modern Python approach.

1.10.24Checking Whether a Path Is a File

from pathlib import Path
path = Path("data/customers.csv")
if path.is_file():
    print("This is a file")

1.10.25Checking Whether a Path Is a Directory

from pathlib import Path
path = Path("data")
if path.is_dir():
    print("This is a directory")

1.10.26Creating a Directory

Using pathlib:

from pathlib import Path
folder = Path("output")

folder.mkdir(exist_ok=True)

exist_ok=True prevents an error if the directory already exists.

For nested directories:

folder = Path("data/raw/customers")

folder.mkdir(

parents=True,
exist_ok=True

)

1.10.27Listing Files in a Directory

from pathlib import Path
folder = Path("data")
for file in folder.iterdir():
    print(file)

This lists files and directories inside data.

1.10.28Finding Specific Files

For example, all CSV files:

from pathlib import Path
folder = Path("data")
for file in folder.glob("*.csv"):
    print(file)

For recursive searching:

for file in folder.rglob("*.csv"):
    print(file)

This is extremely useful in data pipelines.

1.10.29Getting File Information

from pathlib import Path
file = Path("data/customers.csv")
print(file.name)
print(file.stem)
print(file.suffix)
  • For:
  • customers.csv
  • you get approximately:

name → customers.csv

stem → customers

suffix → .csv

1.10.30File Size

from pathlib import Path
file = Path("data/customers.csv")
print(file.stat().st_size)

This returns the file size in bytes.

You could convert it:

size_mb = file.stat().st_size / (1024 * 1024)
print(size_mb)

1.10.31Renaming a File

from pathlib import Path
old_file = Path("old_name.txt")
new_file = Path("new_name.txt")

old_file.rename(new_file)

1.10.32Deleting a File

from pathlib import Path
file = Path("old_file.txt")
if file.exists():
    file.unlink()

Be careful with deletion operations, especially in production pipelines.

1.10.33Reading a Text File with pathlib

You can simplify file handling:

from pathlib import Path
file = Path("data.txt")
content = file.read_text(encoding="utf-8")
print(content)
  • Writing:
  • file.write_text(
  • "Hello Python",
encoding="utf-8"

)

For simple text files, this can be convenient.

1.10.34File Pointer

When reading a file, Python maintains a file position.

Example:

with open("data.txt", "r") as file:
    print(file.read(5))
print(file.read(5))

The first read(5) reads the first five characters.

The second starts from the current position.

1.10.35tell()

tell() returns the current file position.

with open("data.txt", "r") as file:
    print(file.tell())

file.read(5)

print(file.tell())

1.10.36seek()

seek() moves the file pointer.

with open("data.txt", "r") as file:
    print(file.read(5))

file.seek(0)

print(file.read(5))

The second read() starts from the beginning again.

1.10.37Binary Files

  • Not every file contains text.
  • Examples:
  • Images
  • PDFs
  • Audio
  • Videos
  • ZIP files
  • Excel workbooks in their underlying binary format
  • For binary files, use b.

Example:

with open("image.jpg", "rb") as file:
    data = file.read()

Writing binary data:

with open("copy.jpg", "wb") as file:
    file.write(data)

1.10.38Copying a Binary File

A simple example:

with open("source.jpg", "rb") as source:
    data = source.read()
with open("destination.jpg", "wb") as destination:
    destination.write(data)

For real applications, Python's shutil module is usually more convenient:

import shutil
  • shutil.copy(
  • "source.jpg",
  • "destination.jpg"

)

1.10.39CSV Files

CSV means:

Comma-Separated Values

Example:

  • id,name,age
  • 101,Ravi,35
  • 102,Kiran,29
  • 103,Anil,42
  • Python provides the csv module.
import csv
with open(
    "customers.csv",
    "r",
    newline="",
    encoding="utf-8"
) as file:
    reader = csv.reader(file)
for row in reader:
    print(row)

Output:

['id', 'name', 'age']

['101', 'Ravi', '35']

['102', 'Kiran', '29']

['103', 'Anil', '42']

1.10.40Reading CSV as Dictionaries

Using csv.DictReader:

import csv
with open(
    "customers.csv",
    "r",
    newline="",
    encoding="utf-8"
) as file:
    reader = csv.DictReader(file)
for row in reader:
    print(row["name"], row["age"])

Output:

  • Ravi 35
  • Kiran 29
  • Anil 42

This is often easier to work with because you can refer to columns by name.

1.10.41Writing CSV

import csv
data = [
    ["id", "name", "age"],
    [101, "Ravi", 35],
    [102, "Kiran", 29]
]
with open(
    "customers.csv",
    "w",
    newline="",
    encoding="utf-8"
) as file:
    writer = csv.writer(file)

writer.writerows(data)

1.10.42JSON Files

  • JSON is extremely common in:
  • APIs
  • Cloud applications
  • Configuration
  • AI applications
  • Data pipelines

Example:

{

  • "id": 101,
  • "name": "Ravi",
  • "age": 35

}

Python provides the json module.

import json
with open(
    "customer.json",
    "r",
    encoding="utf-8"
) as file:
    data = json.load(file)
print(data)

Output:

{'id': 101, 'name': 'Ravi', 'age': 35}

JSON handling is covered more deeply in 1.21 JSON Handling.

1.10.43Writing JSON

import json
customer = {
    "id": 101,
    "name": "Ravi",
    "age": 35
}
with open(
    "customer.json",
    "w",
    encoding="utf-8"
) as file:
    json.dump(
        customer,
        file,
        indent=4
    )

The resulting file is nicely formatted.

1.10.44Log Files

  • File handling is frequently used for simple log processing.
  • Suppose:
  • 2026-08-22 INFO Pipeline started
  • 2026-08-22 INFO Extract completed
  • 2026-08-22 ERROR Load failed

You can find errors:

with open(
    "pipeline.log",
    "r",
    encoding="utf-8"
) as file:
    for line in file:
        if "ERROR" in line:
            print(line.strip())

Output:

2026-08-22 ERROR Load failed

This is a simple but realistic monitoring use case.

1.10.45Exception Handling with Files

Files may not exist.

This:

with open("missing.txt", "r") as file:
    data = file.read()

can raise:

FileNotFoundError

You can handle it:

try:
    with open(
        "missing.txt",
        "r",
        encoding="utf-8"
    ) as file:
        data = file.read()
except FileNotFoundError:
    print("File not found")

Exception handling is covered in detail in 1.11 Exception Handling.

1.10.46Practical Data Engineering Example

Imagine a folder contains daily sales files:

  • data/
  • sales_20260820.csv
  • sales_20260821.csv
  • sales_20260822.csv

You can find all CSV files:

from pathlib import Path
folder = Path("data")
for file in folder.glob("*.csv"):
    print(file)

Output:

  • data/sales_20260820.csv
  • data/sales_20260821.csv
  • data/sales_20260822.csv

You can then process each file:

from pathlib import Path
folder = Path("data")
for file in folder.glob("*.csv"):
    print("Processing:", file)
with open(
    file,
    "r",
    encoding="utf-8"
) as source:
    for line in source:
        # Process each record
print(line.strip())

This is the basic foundation of a file-based ETL pipeline.

1.10.47Practical ETL Example

Let's create a simple pipeline:

CSV Files
Read
Validate
Transform
Output File

Example:

from pathlib import Path
input_file = Path("sales.txt")
output_file = Path("processed_sales.txt")
with open(
    input_file,
    "r",
    encoding="utf-8"
) as source, open(
    output_file,
    "w",
    encoding="utf-8"
) as target:
    for line in source:
        line = line.strip()
if not line:
    continue
processed = line.upper()
  • target.write(processed + "\n")
  • This demonstrates:
  • File reading
  • File writing
  • Looping
  • Conditional processing
  • Transformation

1.10.48Processing Large Files

For large files, avoid:

data = file.read()

when you don't need the entire file in memory.

Prefer:

with open("large_file.txt") as file:
    for line in file:
        process(line)

This gives you a streaming-style processing pattern.

Conceptually:

Large File
Small Chunk / Line
Process
Next Chunk / Line
Process

This becomes important when building scalable Data Engineering solutions.

1.10.49File Handling with Functions

You can combine functions and file handling.

def read_file(filename):
    with open(
        filename,
        "r",
        encoding="utf-8"
    ) as file:
        return file.read()

Use:

content = read_file("data.txt")
print(content)

Another function:

def write_file(filename, content):
    with open(
        filename,
        "w",
        encoding="utf-8"
    ) as file:
        file.write(content)
  • Then:
  • write_file(
  • "output.txt",
  • "Python Data Engineering"

)

This is much easier to reuse.

1.10.50A Reusable File Processing Function

def count_errors(filename):
    count = 0
with open(
    filename,
    "r",
    encoding="utf-8"
) as file:
    for line in file:
        if "ERROR" in line:
            count += 1
return count

Use:

errors = count_errors("pipeline.log")
print("Errors:", errors)

This is a good example of combining:

Function

+

File handling

+

Loop

+

Conditional statement

1.10.51Common Mistakes

Mistake 1 — Forgetting to close a file

Less preferred:

file = open("data.txt")
data = file.read()

Better:

with open("data.txt") as file:
    data = file.read()

Mistake 2 — Using w accidentally

with open("important.txt", "w") as file:
    file.write("Hello")

This can overwrite existing content.

If you want to add content:

with open("important.txt", "a") as file:
    file.write("Hello")

Mistake 3 — Wrong encoding

For modern text files, explicitly using UTF-8 is often a good choice:

with open(
    "data.txt",
    encoding="utf-8"
) as file:

...

Mistake 4 — Loading huge files into memory

Avoid unnecessarily doing:

data = file.read()
for very large files.

Prefer:

for line in file:
    process(line)

1.10.52Important File Methods

MethodPurpose
read()Read entire file or specified number of characters
readline()Read one line
readlines()Read lines into a list
write()Write text
writelines()Write multiple strings
seek()Move file pointer
tell()Get current file position
close()Close file

1.10.53Important pathlib Methods

MethodPurpose
exists()Check whether path exists
is_file()Check whether it is a file
is_dir()Check whether it is a directory
mkdir()Create directory
iterdir()Iterate directory contents
glob()Find matching files
rglob()Recursive file search
rename()Rename/move
unlink()Delete file
read_text()Read text
write_text()Write text

1.10.54Practice Exercises

  • Exercise 1 — Create a File
  • Create:
  • hello.txt
  • and write:
  • Hello Python
  • Welcome to Data Engineering
  • Exercise 2 — Read a File

Read the file and print its contents.

Exercise 3 — Count Lines

Write a program that counts the number of lines in:

  • data.txt
  • Exercise 4 — Count Errors
  • Given:
  • INFO Pipeline started
  • INFO Extraction completed
  • ERROR Database connection failed
  • INFO Retry started
  • ERROR Retry failed
  • Count the number of lines containing ERROR.
  • Expected:
  • 2
  • Exercise 5 — Find CSV Files
  • Create a directory:
  • data/
  • Find all:
  • *.csv
  • files using pathlib.
  • Exercise 6 — Copy a File
  • Copy:
  • source.txt
  • to:
  • backup.txt
  • Exercise 7 — Append Logs
  • Create a function:
def write_log(message):

...

  • that appends a message to:
  • application.log
  • Exercise 8 — Process Sales Files
  • Given:
  • sales/
  • sales1.csv
  • sales2.csv
  • sales3.csv

Write Python code that discovers all CSV files and prints their filenames.

Exercise 9 — Large File Processing

Write a program that reads a large log file line by line and prints only lines containing:

  • ERROR
  • Exercise 10 — Mini ETL
  • Create:
  • input.txt
  • containing:
  • python
  • sql
  • azure
  • machine learning
  • Read it and create:
  • output.txt
  • where every line is uppercase:
  • PYTHON
  • SQL
  • AZURE
  • MACHINE LEARNING

1.10.55Interview Questions

1. What is file handling?

File handling is the process of reading, writing, creating, modifying, and managing files using a programming language.

2. How do you open a file?

file = open("data.txt", "r")

3. Why is with open() preferred?

It automatically manages the file resource and closes the file when the block finishes.

4. What is the difference between r, w, and a?

  • r → Read
  • w → Write/overwrite
  • a → Append

5. What happens when you open an existing file with w?

Its existing contents are replaced.

6. What does read() do?

It reads the entire file or a specified number of characters.

7. What is the difference between read() and readline()?

read() reads file content, while readline() reads one line.

8. Why process large files line by line?

It avoids unnecessarily loading the entire file into memory.

9. What is pathlib?

pathlib is Python's modern standard-library module for working with filesystem paths.

10. What is the difference between text and binary mode?

Text mode handles decoded text, while binary mode handles raw bytes.

"r" # text

"rb" # binary

11. What is UTF-8?

UTF-8 is a widely used Unicode text encoding that can represent characters from many languages.

12. What exception occurs when a file doesn't exist?

Usually:

FileNotFoundError

13. What is the difference between w and x?

w creates a file if needed and can overwrite an existing file; x creates a new file and fails if the file already exists.

1.10.56Key Takeaways

The most important patterns to remember are:

Read a file

with open(
    "data.txt",
    "r",
    encoding="utf-8"
) as file:
    content = file.read()

Read line by line

with open(
    "data.txt",
    encoding="utf-8"
) as file:
    for line in file:
        print(line.strip())

Write

with open(
    "output.txt",
    "w",
    encoding="utf-8"
) as file:
    file.write("Hello")

Append

with open(
    "log.txt",
    "a",
    encoding="utf-8"
) as file:
    file.write("New log\n")

Find files

from pathlib import Path
for file in Path("data").glob("*.csv"):
    print(file)

Process large files

with open(
    "large_file.txt",
    encoding="utf-8"
) as file:
    for line in file:
        process(line)

The Data Engineering pattern

Files
Discover
Read
Validate
Transform
Write
Load to Database / Cloud

Once you understand this lesson, you'll have the foundation needed to work with CSV, Excel, JSON, APIs, logs, and ETL pipelines. The specialized CSV/Excel and JSON techniques will be covered later in 1.20 and 1.21.

Module 1 · Lesson 1.11

Exception Handling

Exception handling is the mechanism Python provides for dealing with errors that occur while a program is running.

Instead of allowing the program to stop unexpectedly, you can detect the error, handle it appropriately, log it, and continue or terminate gracefully.

This is especially important in Data Engineering, APIs, ETL pipelines, Machine Learning, and production applications.

For example:

number = int(input("Enter a number: "))
  • If the user enters:
  • abc
  • Python raises:
  • ValueError
  • Without exception handling, the program stops.
  • With exception handling:
try:
    number = int(input("Enter a number: "))
except ValueError:
    print("Please enter a valid number")

Output:

Please enter a valid number

1.11.1What Is an Exception?

An exception is an event that occurs during program execution that disrupts the normal flow of the program.

Example:

number = 10
result = number / 0
  • Python raises:
  • ZeroDivisionError
  • Another example:
numbers = [10, 20, 30]
print(numbers[10])

Python raises:

IndexError

1.11.2Error vs Exception

You will often hear the terms error and exception used together.

  • In Python, exceptions are objects representing abnormal conditions that can often be caught and handled.
  • Examples:
  • ValueError
  • TypeError
  • IndexError
  • KeyError
  • FileNotFoundError
  • ZeroDivisionError
  • The important idea is:
Normal execution
Something unexpected happens
Exception raised
Python looks for a handler
Handled or program terminates

1.11.3Common Python Exceptions

Some important built-in exceptions are:

ExceptionTypical Cause
ValueErrorInvalid value
TypeErrorIncompatible type/operation
ZeroDivisionErrorDivision by zero
IndexErrorInvalid list/sequence index
KeyErrorMissing dictionary key
FileNotFoundErrorFile doesn't exist
NameErrorUndefined variable
AttributeErrorMissing attribute
ImportErrorImport problem
ModuleNotFoundErrorModule cannot be found
PermissionErrorInsufficient permission
OverflowErrorNumeric operation exceeds limits
RuntimeErrorGeneric runtime problem

1.11.4The try Statement

The basic exception-handling structure is:

try:
    # risky code
except:
    # handling code

Example:

try:
    number = int("abc")
except:
    print("Something went wrong")

Output:

Something went wrong

However, bare except: is usually not the best practice.

Prefer catching the specific exception you expect.

1.11.5Handling ValueError

try:
    number = int("abc")
except ValueError:
    print("Invalid number")

Output:

Invalid number

int("abc") cannot convert "abc" into an integer, so Python raises ValueError.

1.11.6Handling ZeroDivisionError

try:
    result = 10 / 0
except ZeroDivisionError:
    print("Cannot divide by zero")

Output:

Cannot divide by zero

1.11.7Handling IndexError

numbers = [10, 20, 30]
try:
    print(numbers[5])
except IndexError:
    print("Index does not exist")

Output:

Index does not exist

1.11.8Handling KeyError

Consider:

employee = {
    "name": "Ravi",
    "department": "Data Engineering"
}

This causes an exception:

try:
    print(employee["salary"])
except KeyError:
    print("Salary information not available")

Output:

Salary information not available

1.11.9Handling FileNotFoundError

Very useful for Data Engineering.

try:
    with open(
        "customers.csv",
        "r",
        encoding="utf-8"
    ) as file:
        data = file.read()
except FileNotFoundError:
    print("Input file does not exist")

Instead of the program crashing, you can handle the missing file.

1.11.10try and except Flow

Start
Execute try

┌──────┴──────┐

Success Exception

↓ ↓

Continue except block

↓ ↓

└──────┬───────┘

Continue

1.11.11Multiple except Blocks

A single try block can have multiple exception handlers.

try:
    number = int(input("Enter number: "))
result = 100 / number
except ValueError:
    print("Please enter a valid number")
except ZeroDivisionError:
    print("Number cannot be zero")
  • If the user enters:
  • abc
  • you get:
  • Please enter a valid number
  • If the user enters:
  • 0
  • you get:
  • Number cannot be zero

1.11.12Why Specific Exceptions Are Better

Avoid:

try:

...

except:
    print("Error")

Prefer:

try:

...

except FileNotFoundError:

...

except PermissionError:

...

  • Specific exceptions make your program:
  • Easier to understand
  • Easier to debug
  • Safer
  • More maintainable

1.11.13Exception Hierarchy

Python exceptions have a hierarchy.

Conceptually:

BaseException
└── Exception
├── ValueError
  • ├── TypeError
  • ├── KeyError
  • ├── IndexError
  • ├── OSError

│ ├── FileNotFoundError

│ └── PermissionError

└── ...

Most application-level exceptions you handle inherit from Exception.

1.11.14Catching Multiple Exceptions Together

If multiple exceptions should receive the same treatment:

try:
    number = int(input("Enter number: "))
result = 100 / number
except (ValueError, ZeroDivisionError):
    print("Invalid input")

This is useful when the handling logic is identical.

1.11.15The else Block

Python's exception handling supports:

try
except
else

finally

The else block executes only when no exception occurs in the try block.

Example:

try:
    number = int("100")
except ValueError:
    print("Invalid number")
else:
    print("Conversion successful")

Output:

Conversion successful

1.11.16try-except-else

Structure:

try:
    # code that might fail
except SomeException:
    # handle exception
else:
    # execute if no exception

Example:

try:
    number = int(input("Enter number: "))
except ValueError:
    print("Invalid number")
else:
    print("You entered:", number)

This clearly separates successful processing from error handling.

1.11.17The finally Block

finally executes regardless of whether an exception occurs.

Example:

try:
    number = 10 / 2
except ZeroDivisionError:
    print("Cannot divide by zero")

finally:

print("Execution completed")

Output:

Execution completed

1.11.18finally with an Exception

try:
    number = 10 / 0
except ZeroDivisionError:
    print("Cannot divide by zero")

finally:

print("Execution completed")

Output:

  • Cannot divide by zero
  • Execution completed
  • The finally block still runs.

1.11.19Why Is finally Useful?

finally is useful for cleanup operations.

For example:

Open resource
Process
Exception or success
Cleanup
  • Resources can include:
  • Database connections
  • Network connections
  • Files
  • Temporary resources

However, for files, with open() is generally preferable because the context manager handles cleanup automatically.

1.11.20Raising an Exception

You can deliberately raise an exception using raise.

age = -5
if age < 0:
    raise ValueError("Age cannot be negative")

Python raises:

ValueError: Age cannot be negative

This is useful when validating input.

1.11.21Why Use raise?

Suppose you create a function:

def calculate_discount(price, discount):
    if discount < 0:
        raise ValueError("Discount cannot be negative")
return price * (1 - discount / 100)
  • Now:
  • calculate_discount(1000, -10)
  • raises an appropriate error.

This is much better than silently accepting invalid data.

1.11.22Raising Exceptions in Data Validation

Consider a data pipeline:

def validate_record_count(source, target):
    if source < 0:
        raise ValueError("Source count cannot be negative")
if target < 0:
    raise ValueError("Target count cannot be negative")
return source == target

Now invalid input is rejected immediately.

1.11.23Re-Raising an Exception

  • Sometimes you catch an exception to log or inspect it, then want the exception to continue upward.
  • Use:
  • raise

Example:

try:
    process_data()
except ValueError:
    print("Logging validation error")

raise

The exception is re-raised after logging.

This is common in production systems.

1.11.24Exception Object

You can store the exception object using as.

try:
    number = int("abc")
except ValueError as error:
    print("Error:", error)

Output:

Error: invalid literal for int() with base 10: 'abc'

This gives you the actual exception message.

1.11.25Logging Exceptions

In production applications, instead of only:

print("Error")

you'll often use Python's logging module.

Example:

import logging

logging.basicConfig(level=logging.ERROR)

try:
    result = 10 / 0
except ZeroDivisionError as error:
    logging.error("Calculation failed: %s", error)
  • This becomes especially important for:
  • ETL pipelines
  • APIs
  • Cloud applications
  • ML pipelines
  • Production services

You'll study logging in detail in 1.23 Logging.

1.11.26Exception Handling with File Processing

Consider:

filename = "customers.csv"
try:
    with open(
        filename,
        "r",
        encoding="utf-8"
    ) as file:
        for line in file:
            print(line.strip())
except FileNotFoundError:
    print(f"File not found: {filename}")
except PermissionError:
    print(f"Permission denied: {filename}")

This is much safer than assuming the file always exists.

1.11.27Data Engineering Example — File Pipeline

Imagine:

Input CSV
Read
Validate
Transform
Load

A simple Python structure could be:

def process_file(filename):
    try:
        with open(
            filename,
            "r",
            encoding="utf-8"
        ) as file:
            for line in file:
                process_record(line)
except FileNotFoundError:
    print("Input file missing")
except PermissionError:
    print("Cannot access input file")
except UnicodeDecodeError:
    print("File encoding is invalid")

This is the beginning of production-style error handling.

1.11.28Handling Errors Per Record

Suppose one bad record shouldn't stop the entire batch.

records = ["100", "200", "abc", "300"]
for record in records:
    try:
        number = int(record)
print(number * 2)
except ValueError:
    print("Invalid record:", record)

Output:

  • 200
  • 400
  • Invalid record: abc
  • 600

Notice that processing continues after the invalid record.

This pattern can be very useful in data ingestion.

1.11.29Batch-Level vs Record-Level Errors

This is an important Data Engineering concept.

  • Record-level error
  • One record is invalid:
  • Record 1 → PASS
  • Record 2 → PASS
  • Record 3 → FAIL
  • Record 4 → PASS
  • You may want to continue processing.
  • Batch-level error
  • The entire input file is missing:
File missing
Cannot process batch
Stop pipeline

Therefore, exception handling strategy depends on what failed.

1.11.30API Exception Handling

Suppose you call an API:

import requests
try:
    response = requests.get(
        "https://example.com/api/data",
        timeout=10
    )

response.raise_for_status()

except requests.Timeout:
    print("API request timed out")
except requests.RequestException as error:
    print("API request failed:", error)

This is a common production pattern.

The exact exceptions depend on the library you're using.

1.11.31Database Exception Handling

Suppose your application interacts with a database:

try:
    connection = connect_to_database()

execute_query(connection)

except Exception as error:
    print("Database operation failed:", error)

finally:

close_connection(connection)

In production, you would generally catch the specific database exceptions provided by your database driver rather than blindly catching every exception.

1.11.32Custom Exceptions

You can create your own exception classes.

Example:

class DataQualityError(Exception):
    pass

Now:

raise DataQualityError("Record count validation failed")

This allows your application to distinguish data-quality failures from unrelated failures.

1.11.33Custom Exception Example

class PipelineValidationError(Exception):
    pass
def validate_pipeline(source, target):
    if source != target:
        raise PipelineValidationError(
            "Source and target counts do not match"
        )
return True

Use:

try:
    validate_pipeline(100000, 99500)
except PipelineValidationError as error:
    print("Validation failed:", error)

Output:

Validation failed: Source and target counts do not match

This is useful for larger applications.

1.11.34Custom Exceptions with More Information

You can create richer exception classes:

class DataQualityError(Exception):
    def __init__(self, table, source, target):
        self.table = table

self.source = source

self.target = target

super().__init__(

f"Validation failed for {table}"

)

  • Use:
  • raise DataQualityError(
  • "ORDERS",
  • 250000,
  • 249000

)

Now the exception carries structured information.

1.11.35Exception Handling with Functions

Consider:

def divide(a, b):
    try:
        return a / b
except ZeroDivisionError:
    return None

Use:

result = divide(10, 0)
if result is None:
    print("Calculation failed")

However, in larger applications, it may be preferable to let the caller decide how to handle the exception instead of silently converting every failure to None.

1.11.36Don't Hide Errors

Avoid this:

try:
    process_data()
except Exception:
    pass

This is dangerous because it silently ignores failures.

You may never know that something went wrong.

Bad:

try:
    load_data()
except:
    pass

Better:

try:
    load_data()
except Exception as error:
    logging.exception(
        "Data load failed: %s",
        error
    )

raise

The exact strategy depends on whether the application can safely recover.

1.11.37Catching Exception

Sometimes you need a broad safety boundary:

try:
    run_pipeline()
except Exception as error:
    print("Pipeline failed:", error)

This catches most ordinary application exceptions.

However, don't use broad catching everywhere.

Prefer:

except FileNotFoundError:

when you know what failure you expect.

1.11.38Exception Handling Best Practice

A good hierarchy is:

Specific exception
Handle expected problem
Log useful information
Recover if possible
Otherwise re-raise

For example:

try:
    data = read_file()
except FileNotFoundError:
    logging.error("Input file missing")

raise

except PermissionError:
    logging.error("Permission denied")

raise

1.11.39assert vs raise

Python also has assert.

Example:

age = 25

assert age >= 0

If the condition is false, Python raises AssertionError.

You can include a message:

assert age >= 0, "Age cannot be negative"

However, assertions are primarily intended for developer/program invariants, not as the main mechanism for validating untrusted runtime input.

For application validation, prefer:

if age < 0:
    raise ValueError("Age cannot be negative")

1.11.40Exception Chaining

Python can preserve the original cause of an exception.

Example:

try:
    number = int("abc")
except ValueError as error:
    raise RuntimeError(
        "Failed to process customer age"
    ) from error

The resulting traceback shows the relationship between the original ValueError and the new RuntimeError.

This is useful when you want to add higher-level context without losing the original cause.

1.11.41Exception Handling Flow

A complete structure can look like:

try:
    data = read_data()
  • validate_data(data)
  • transform_data(data)
  • load_data(data)
except FileNotFoundError:
    logging.error("Input file missing")
except ValueError:
    logging.error("Data validation failed")
except Exception as error:
    logging.exception("Unexpected pipeline error: %s", error)
else:
    logging.info("Pipeline completed successfully")
  • finally:
  • logging.info("Pipeline execution finished")
  • Conceptually:
START
TRY

┌───────┴────────┐

Success Error

↓ ↓

ELSE EXCEPTION HANDLER

↓ ↓

└────────┬────────┘

FINALLY
END

1.11.42Real-World ETL Example

Let's put several concepts together.

import logging

logging.basicConfig(level=logging.INFO)

def process_file(filename):
    try:
        logging.info("Processing file: %s", filename)
with open(
    filename,
    "r",
    encoding="utf-8"
) as file:
    for line_number, line in enumerate(
        file,
        start=1
    ):
        try:
            value = int(line.strip())
print("Processed:", value)
except ValueError:
    logging.error(
        "Invalid record at line %s",
        line_number
    )
except FileNotFoundError:
    logging.error(
        "File not found: %s",
        filename
    )

raise

except PermissionError:
    logging.error(
        "Permission denied: %s",
        filename
    )
  • raise
  • finally:
  • logging.info("Processing finished")
  • This demonstrates:
Function
File handling
Loop
Record-level exception handling
Batch-level exception handling
Logging
Finally

That's much closer to real Data Engineering code.

1.11.43Best Practices

1. Catch specific exceptions

Prefer:

except FileNotFoundError:
    over:
        except:
            2. Keep try blocks small

Instead of:

try:
    read()
  • transform()
  • save()
  • send_email()
  • consider isolating operations where useful.
  • For example:
try:
    data = read()
except FileNotFoundError:

...

This makes it clearer which operation failed.

3. Don't silently ignore exceptions

Avoid:

except Exception:
    pass

4. Log meaningful context

  • Instead of:
  • logging.error("Error")
  • prefer:
  • logging.error(
  • "Failed to process file %s",
  • filename

)

5. Re-raise when the caller needs to know

except Exception:
    logging.exception("Processing failed")

raise

6. Use finally for cleanup when necessary

  • finally:
  • cleanup()
  • For files, prefer context managers:
with open(...) as file:

...

7. Create custom exceptions for domain-specific failures

For example:

class DataQualityError(Exception):
    pass

1.11.44Common Mistakes

Mistake 1 — Catching everything

try:

...

except:
    print("Error")

You lose useful information about what actually failed.

Mistake 2 — Empty exception handler

try:
    process()
except Exception:
    pass

This can hide serious problems.

Mistake 3 — Returning None for every error

def divide(a, b):
    try:
        return a / b
except:
    return None
  • This may hide the difference between valid results and failures.
  • Mistake 4 — Using exceptions for normal control flow
  • Don't use exceptions where a simple condition is clearer.
  • Instead of:
try:
    value = dictionary["name"]
except KeyError:
    value = "Unknown"

often this is clearer:

value = dictionary.get("name", "Unknown")

Exceptions are best used for genuinely exceptional conditions.

1.11.45Practice Exercises

Exercise 1 — Safe Division

Create:

def divide(a, b):

...

Handle division by zero.

Example:

divide(10, 2)

→ 5

  • divide(10, 0)
  • → Cannot divide by zero
  • Exercise 2 — Safe Integer Conversion
  • Write:
def convert_to_integer(value):

...

  • Handle:
  • "100" → 100
  • "abc" → appropriate error handling
  • Exercise 3 — File Reader
  • Create:
def read_file(filename):

...

  • Handle:
  • FileNotFoundError
  • PermissionError
  • Exercise 4 — Dictionary Access
  • Given:
employee = {
    "name": "Ravi",
    "department": "Data Engineering"
}
  • Safely access:
  • name
  • department
  • salary
  • Handle the missing salary key.
  • Exercise 5 — Record Processing
  • Given:
records = ["100", "200", "abc", "300", "xyz"]
  • Convert every valid record to an integer while allowing invalid records to be skipped.
  • Expected:
  • 100
  • 200
  • 300
  • Exercise 6 — Data Quality Validation
  • Create:
def validate_counts(source, target):

...

  • Raise a custom DataQualityError when:
  • source != target
  • Exercise 7 — Pipeline
  • Create:
  • read_file()
  • validate_data()
  • transform_data()
  • load_data()

Use exception handling to handle failures at the appropriate level.

1.11.46Interview Questions

1. What is exception handling?

It is the mechanism used to detect and handle runtime exceptions so that applications can respond gracefully to failures.

2. What is the purpose of try?

It contains code that may raise an exception.

3. What is except?

It defines how a particular exception should be handled.

4. What is else?

It executes when the try block completes without an exception.

5. What is finally?

It executes regardless of whether an exception occurred.

6. What is raise?

It explicitly raises an exception.

raise ValueError("Invalid data")

7. What is the difference between raise and raise ... from?

raise raises or re-raises an exception. raise ... from ... explicitly establishes another exception as the cause, preserving exception context.

8. Can you have multiple except blocks?

Yes.

try:

...

except ValueError:

...

except TypeError:

...

9. Can one except handle multiple exceptions?

Yes:

except (ValueError, TypeError):

...

10. What is a custom exception?

A user-defined exception class, typically derived from Exception.

class DataQualityError(Exception):
    pass

11. Why shouldn't you use a bare except everywhere?

Because it can catch unexpected problems and hide useful debugging information.

12. What is exception chaining?

It allows you to preserve the relationship between a higher-level exception and its original cause:

raise RuntimeError("Processing failed") from error

1.11.47Key Takeaways

The most important structure is:

try:
    risky_operation()
except SpecificException as error:
    handle_error(error)
else:
    handle_success()
  • finally:
  • cleanup()
  • Remember the roles:
try

Code that might fail

except

Handle the failure

else

Run when no exception occurred

finally
Always run cleanup/finalization
raise
Explicitly generate or re-raise an exception

For your Data Engineering and AI/ML journey, a particularly important pattern is:

try:
    data = read_data()

validate_data(data)

transformed_data = transform_data(data)

load_data(transformed_data)

except FileNotFoundError:
    logging.error("Input file missing")
except ValueError:
    logging.error("Data validation failed")
except Exception:
    logging.exception("Unexpected pipeline failure")
  • raise
  • finally:
  • logging.info("Pipeline execution finished")

This is the foundation for building reliable ETL pipelines, APIs, ML pipelines, and production AI applications.

Next relevant lesson: 1.12 Object-Oriented Programming (OOP) — classes, objects, constructors, instance attributes, methods, inheritance, encapsulation, polymorphism, and practical Data Engineering examples.

Module 1 · Lesson 1.12

Object-Oriented Programming (OOP)

Object-Oriented Programming (OOP) is a programming approach where we organize code around objects that contain:

Data → attributes/properties

Behavior → methods/functions

Python is a powerful object-oriented language, and OOP becomes very important when you start building larger applications, Data Engineering frameworks, APIs, ML systems, and AI applications.

1.12.1Why Do We Need OOP?

Imagine you want to store information about employees.

Without OOP:

employee1_name = "Sreehari"
employee1_age = 37
employee1_department = "Data Engineering"
employee2_name = "Ravi"
employee2_age = 32
employee2_department = "Analytics"

As the number of employees grows, this becomes difficult to manage.

With OOP:

class Employee:
    pass

You can create objects:

employee1 = Employee()
employee2 = Employee()

Each object can contain its own data.

Conceptually:

Employee Class
├── employee1

│ ├── name

│ ├── age

│ └── department

└── employee2

├── name

├── age

└── department

1.12.2What Is a Class?

A class is a blueprint or template for creating objects.

Example:

class Employee:
    pass
  • The class itself doesn't represent a specific employee.
  • It defines what an employee object can be.
  • Think of:
Class = Blueprint
Object = Actual thing created from blueprint
  • For example:
  • Class → Employee
  • Objects → Sreehari, Ravi, Kiran

1.12.3What Is an Object?

An object is an instance of a class.

Example:

class Employee:
    pass
employee1 = Employee()
employee2 = Employee()
  • Here:
  • Employee → Class
  • employee1 → Object
  • employee2 → Object

You can verify:

print(type(employee1))

You will get something similar to:

<class '__main__.Employee'>

1.12.4Adding Attributes

We can add data to an object.

class Employee:
    pass
employee = Employee()
  • employee.name = "Sreehari"
  • employee.age = 37
  • employee.department = "Data Engineering"
  • Now:
print(employee.name)
print(employee.age)
print(employee.department)

Output:

  • Sreehari
  • 37
  • Data Engineering

1.12.5The __init__() Method

Instead of manually assigning attributes after creating an object, Python provides the special method:

__init__()

Example:

class Employee:
    def __init__(self, name, age, department):
        self.name = name

self.age = age

self.department = department

Now create an object:

employee = Employee(
    "Sreehari",
    37,
    "Data Engineering"
)

The values are automatically assigned.

print(employee.name)
print(employee.age)
print(employee.department)

Output:

  • Sreehari
  • 37
  • Data Engineering

1.12.6Understanding self

This is one of the most important concepts in Python OOP.

Consider:

class Employee:
    def __init__(self, name, age):
        self.name = name
  • self.age = age
  • Here:
  • self.name
  • self.age

refer to attributes belonging to the current object.

When we create:

employee1 = Employee("Sreehari", 37)

Python effectively associates the data with employee1.

When we create:

employee2 = Employee("Ravi", 32)
  • the data belongs to employee2.
  • Conceptually:
  • employee1
  • ├── name = Sreehari
  • └── age = 37
  • employee2
  • ├── name = Ravi
  • └── age = 32
  • self refers to the current instance.

1.12.7Instance Attributes

Attributes created using self are usually called instance attributes.

class Employee:
    def __init__(self, name, salary):
        self.name = name

self.salary = salary

Create:

employee = Employee(
    "Sreehari",
    100000
)
  • Here:
  • self.name
  • self.salary
  • are instance attributes.

1.12.8Methods

A function defined inside a class is called a method.

Example:

class Employee:
    def __init__(self, name, salary):
        self.name = name

self.salary = salary

def display(self):
    print(self.name)
print(self.salary)

Call:

employee = Employee("Sreehari", 100000)

employee.display()

Output:

Sreehari

100000

1.12.9Method with Parameters

Methods can accept parameters.

class Employee:
    def __init__(self, name, salary):
        self.name = name

self.salary = salary

def give_bonus(self, percentage):
    bonus = self.salary * percentage / 100
return bonus

Use:

employee = Employee(
    "Sreehari",
    100000
)
bonus = employee.give_bonus(10)
print(bonus)

Output:

10000.0

1.12.10Class Attributes

A class can also have attributes shared by the class.

class Employee:
    company = "ABC Technologies"
def __init__(self, name):
    self.name = name

Now:

employee1 = Employee("Sreehari")
employee2 = Employee("Ravi")
print(employee1.company)
print(employee2.company)
  • Both can access:
  • ABC Technologies
  • Conceptually:
Employee Class
└── company = ABC Technologies

┌─────┴─────┐

│ │

employee1 employee2

1.12.11Instance vs Class Attributes

FeatureInstance AttributeClass Attribute
Belongs toIndividual objectClass
Exampleself.namecompany
Can differ per object?YesUsually shared
DefinedUsually in __init__Directly in class

Example:

class Employee:
    company = "ABC"
def __init__(self, name):
    self.name = name

Here:

company → class attribute

name → instance attribute

1.12.12Four Main Principles of OOP

The four commonly taught pillars are:

OOP

┌───────┼────────┐

│ │ │

Encapsulation Inheritance

│ │ │

Abstraction Polymorphism

Let's understand each.

1.12.13Encapsulation

Encapsulation means keeping data and the operations that work on that data together, while controlling how the internal state is accessed or modified.

Example:

class BankAccount:
    def __init__(self, balance):
        self.balance = balance
def deposit(self, amount):
    if amount > 0:
        self.balance += amount

Instead of allowing arbitrary modifications through many places in your application, you can provide controlled methods such as:

account.deposit(5000)

1.12.14Public Attributes

Python doesn't enforce traditional private/public access modifiers in the same way as some languages.

A normal attribute is generally considered public:

class Employee:
    def __init__(self, name):
        self.name = name

You can access:

employee.name

1.12.15Protected Convention — _

A single underscore is commonly used as a convention indicating an internal attribute:

class Employee:
    def __init__(self):
        self._salary = 50000

The underscore communicates:

This is intended for internal use.

It is not a strict access restriction.

1.12.16Name Mangling — __

Double underscore at the beginning triggers name mangling.

class Employee:
    def __init__(self):
        self.__salary = 50000
  • Direct access:
  • employee.__salary
  • will normally fail.

Python internally transforms the name approximately to:

_Employee__salary

This mechanism is called name mangling.

It is useful for avoiding accidental name collisions in subclasses, rather than providing absolute security.

1.12.17Properties

Python provides property for controlled attribute access.

Example:

class Employee:
    def __init__(self, salary):
        self._salary = salary

@property

def salary(self):
    return self._salary

Now:

employee = Employee(100000)
print(employee.salary)

You access it like an attribute while the class controls how the value is retrieved.

1.12.18Property Setter

You can also control assignment.

class Employee:
    def __init__(self, salary):
        self.salary = salary

@property

def salary(self):
    return self._salary

@salary.setter

def salary(self, value):
    if value < 0:
        raise ValueError(
            "Salary cannot be negative"
        )

self._salary = value

Now:

employee = Employee(100000)
  • employee.salary = 120000
  • works.
  • But:
  • employee.salary = -5000
  • raises:
  • ValueError

This is a practical example of encapsulation.

1.12.19Inheritance

Inheritance allows one class to reuse or extend another class.

Example:

class Employee:
    def work(self):
        print("Employee is working")
class DataEngineer(Employee):
    def build_pipeline(self):
        print("Building data pipeline")

Now:

engineer = DataEngineer()

engineer.work()

engineer.build_pipeline()

Output:

  • Employee is working
  • Building data pipeline
  • DataEngineer inherits from Employee.

1.12.20Parent and Child Classes

Terminology:

Employee
Parent / Base / Superclass
DataEngineer
Child / Derived / Subclass

The child can use functionality defined by the parent.

1.12.21Overriding Methods

A child class can provide its own implementation of a parent method.

class Employee:
    def work(self):
        print("Employee working")
class DataEngineer(Employee):
    def work(self):
        print("Data Engineer building pipelines")

Now:

employee = Employee()
engineer = DataEngineer()

employee.work()

engineer.work()

Output:

Employee working

Data Engineer building pipelines

This is method overriding.

1.12.22super()

super() allows a child class to call functionality from its parent class.

class Employee:
    def __init__(self, name):
        self.name = name
class DataEngineer(Employee):
    def __init__(self, name, technology):
        super().__init__(name)

self.technology = technology

Create:

engineer = DataEngineer(
    "Sreehari",
    "Azure"
)

Now:

print(engineer.name)
print(engineer.technology)

Output:

Sreehari

Azure

1.12.23Why Use super()?

  • Suppose the parent class already performs initialization.
  • Instead of duplicating:
  • self.name = name
  • the child can reuse:
  • super().__init__(name)
  • This reduces duplicated code.

1.12.24Multiple Inheritance

Python allows a class to inherit from multiple classes.

class Employee:
    def work(self):
        print("Working")
class AzureSkill:
    def deploy(self):
        print("Deploying to Azure")
class DataEngineer(Employee, AzureSkill):
    pass

Now:

engineer = DataEngineer()

engineer.work()

engineer.deploy()

Output:

Working

Deploying to Azure

Python supports multiple inheritance, although it should be used thoughtfully.

1.12.25Method Resolution Order — MRO

When multiple inheritance is involved, Python needs to determine which method to use.

This is called Method Resolution Order (MRO).

You can inspect it:

print(DataEngineer.mro())

For example, Python may show a sequence similar to:

  • DataEngineer
  • Employee
  • AzureSkill
  • object

The exact order depends on the inheritance structure.

1.12.26Polymorphism

Polymorphism means that different objects can respond to the same operation in their own way.

Example:

class CSVProcessor:
    def process(self):
        print("Processing CSV")
class JSONProcessor:
    def process(self):
        print("Processing JSON")

Now:

processors = [
    CSVProcessor(),
    JSONProcessor()
]
for processor in processors:
    processor.process()

Output:

  • Processing CSV
  • Processing JSON
  • The same method call:
  • processor.process()
  • behaves differently depending on the object.

1.12.27Duck Typing

Python often relies on duck typing.

  • The idea is:
  • If an object supports the required operation, we can use it.
  • For example:
def process_data(processor):
    processor.process()
  • Any object with a compatible process() method can potentially be passed in.
  • process_data(CSVProcessor())
  • process_data(JSONProcessor())

Python doesn't require both classes to inherit from the same base class just for this simple interface.

This is an important Python concept.

1.12.28Abstraction

  • Abstraction means exposing the important interface while hiding implementation details.
  • For example, when you call:
  • model.predict(data)

you don't need to know every internal mathematical operation performed by the model.

Python supports formal abstraction using the abc module.

1.12.29Abstract Base Class

from abc import ABC, abstractmethod
class DataProcessor(ABC):
    @abstractmethod
def process(self, data):
    pass

A subclass must implement process():

class CSVProcessor(DataProcessor):
    def process(self, data):
        print("Processing CSV")

And:

class JSONProcessor(DataProcessor):
    def process(self, data):
        print("Processing JSON")

This establishes a common interface.

1.12.30Creating an Abstract Class

from abc import ABC, abstractmethod
class Pipeline(ABC):
    @abstractmethod
def run(self):
    pass

Now:

class SalesPipeline(Pipeline):
    def run(self):
        print("Running sales pipeline")

You can create:

pipeline = SalesPipeline()

pipeline.run()

But you generally cannot instantiate the abstract class itself:

Pipeline()

because it contains an abstract method.

1.12.31Composition

Another important OOP concept is composition.

Instead of inheriting, one object can contain another object.

Example:

class Database:
    def connect(self):
        print("Database connected")
class DataPipeline:
    def __init__(self):
        self.database = Database()
def run(self):
    self.database.connect()
print("Pipeline running")

Now:

pipeline = DataPipeline()

pipeline.run()

Conceptually:

DataPipeline
└── Database

This represents a has-a relationship.

Inheritance often represents an is-a relationship.

1.12.32Inheritance vs Composition

  • Inheritance
  • DataEngineer IS AN Employee
  • Composition
  • DataPipeline HAS A Database
  • A useful rule is:

IS-A → consider inheritance

HAS-A → consider composition

In production software, composition is often preferred when inheritance isn't a natural relationship.

1.12.33Static Methods

  • Sometimes a method doesn't need access to the object or class.
  • Use:
  • @staticmethod

Example:

class MathUtils:
    @staticmethod
def add(a, b):
    return a + b

Call:

result = MathUtils.add(10, 20)
print(result)

Output:

30

No object is required.

1.12.34Class Methods

  • A class method receives the class itself as the first argument, conventionally called cls.
  • Use:
  • @classmethod

Example:

class Employee:
    company = "ABC"

@classmethod

def get_company(cls):
    return cls.company

Call:

print(Employee.get_company())

Output:

ABC

1.12.35Instance vs Class vs Static Methods

MethodFirst argumentAccess
Instance methodselfObject data
Class methodclsClass data
Static methodNone automaticallyIndependent utility

Example:

class Example:
    def instance_method(self):
        pass

@classmethod

def class_method(cls):
    pass

@staticmethod

def static_method():
    pass

1.12.36Magic / Dunder Methods

Python has special methods surrounded by double underscores.

Examples:

__init__

__str__

__repr__

__len__

__eq__

These are often called dunder methods.

1.12.37__str__()

__str__() controls the user-friendly string representation of an object.

Without it:

class Employee:
    def __init__(self, name):
        self.name = name

Printing the object gives a generic representation.

With:

class Employee:
    def __init__(self, name):
        self.name = name
def __str__(self):
    return f"Employee: {self.name}"

Now:

employee = Employee("Sreehari")
print(employee)

Output:

Employee: Sreehari

1.12.38__repr__()

__repr__() is intended to provide a useful representation of an object, particularly for developers and debugging.

Example:

class Employee:
    def __init__(self, name, age):
        self.name = name

self.age = age

def __repr__(self):
    return (
        f"Employee(name={self.name!r}, "
            f"age={self.age!r})"
    )

Then:

employee = Employee("Sreehari", 37)
print(employee)

Depending on how __str__ is defined, Python may use the appropriate representation.

1.12.39__len__()

You can define how len() behaves for your object.

class DataSet:
    def __init__(self, records):
        self.records = records
def __len__(self):
    return len(self.records)

Now:

data = DataSet([10, 20, 30, 40])
print(len(data))

Output:

4

1.12.40__eq__()

You can define how equality works.

class Employee:
    def __init__(self, name):
        self.name = name
def __eq__(self, other):
    return self.name == other.name

Then:

e1 = Employee("Sreehari")
e2 = Employee("Sreehari")
print(e1 == e2)

Output:

True

1.12.41Dataclasses

Python provides dataclasses for classes that mainly store data.

Example:

from dataclasses import dataclass

@dataclass

class Employee:
    name: str
  • age: int
  • department: str
  • Now:
employee = Employee(
    "Sreehari",
    37,
    "Data Engineering"
)

You automatically get useful functionality such as an initializer and a readable representation.

print(employee)

Output will be similar to:

Employee(name='Sreehari', age=37, department='Data Engineering')

Dataclasses are very useful for clean data models.

1.12.42OOP Example — Data Engineering

Let's design a simple pipeline object.

class DataPipeline:
    def __init__(self, name):
        self.name = name

self.status = "NOT_STARTED"

def start(self):
    self.status = "RUNNING"
print(f"{self.name} started")
def complete(self):
    self.status = "SUCCESS"
print(f"{self.name} completed")
def fail(self):
    self.status = "FAILED"
print(f"{self.name} failed")

Use:

pipeline = DataPipeline("Customer_Load")

pipeline.start()

pipeline.complete()

print(pipeline.status)

Output:

  • Customer_Load started
  • Customer_Load completed
  • SUCCESS

1.12.43OOP ETL Pipeline

We can go further.

class ETLPipeline:
    def __init__(self, name):
        self.name = name
def extract(self):
    print("Extracting data")
def transform(self):
    print("Transforming data")
def load(self):
    print("Loading data")
def run(self):
    self.extract()
  • self.transform()
  • self.load()
  • Now:
pipeline = ETLPipeline("Sales Pipeline")

pipeline.run()

Output:

  • Extracting data
  • Transforming data
  • Loading data

This is much cleaner than putting everything into one large procedural script.

1.12.44Adding Exception Handling to OOP

We can combine OOP with the previous lesson.

class ETLPipeline:
    def __init__(self, name):
        self.name = name
def extract(self):
    print("Extracting")
def transform(self):
    print("Transforming")
def load(self):
    print("Loading")
def run(self):
    try:
        self.extract()

self.transform()

self.load()

except Exception as error:
    print(
        f"{self.name} failed: {error}"
    )
else:
    print(
        f"{self.name} completed successfully"
    )

Use:

pipeline = ETLPipeline("Customer Pipeline")
  • pipeline.run()
  • This combines:
  • OOP

+

Functions

+

Exception Handling

1.12.45Data Source Abstraction

Imagine your system supports multiple sources.

class DataSource:
    def read(self):
        raise NotImplementedError

CSV:

class CSVSource(DataSource):
    def read(self):
        print("Reading CSV")

Database:

class DatabaseSource(DataSource):
    def read(self):
        print("Reading Database")

API:

class APISource(DataSource):
    def read(self):
        print("Reading API")

Now:

sources = [
    CSVSource(),
    DatabaseSource(),
    APISource()
]
for source in sources:
    source.read()

Output:

  • Reading CSV
  • Reading Database
  • Reading API
  • This demonstrates polymorphism.

1.12.46OOP in Machine Learning

OOP is also common in ML.

Imagine a generic model:

class Model:
    def train(self, data):
        print("Training model")
def predict(self, data):
    print("Generating predictions")

A specialized model:

class ChurnModel(Model):
    def train(self, data):
        print("Training churn model")

Now:

model = ChurnModel()

model.train(data)

model.predict(data)

This pattern resembles how many ML frameworks organize models and components.

1.12.47OOP in APIs

Suppose you're building an API client.

class APIClient:
    def __init__(self, base_url):
        self.base_url = base_url
def get(self, endpoint):
    print(
        f"GET {self.base_url}{endpoint}"
    )

Use:

client = APIClient(
    "https://api.example.com"
)

client.get("/customers")

This lets you encapsulate API configuration and behavior inside an object.

1.12.48OOP in AI Applications

A simple AI assistant could be modeled as:

class AIAssistant:
    def __init__(self, model):
        self.model = model
def ask(self, question):
    return self.model.generate(question)

Then:

assistant = AIAssistant(model)
response = assistant.ask(
    "What is machine learning?"
)

This type of object-oriented structure becomes useful when building larger AI systems.

1.12.49Composition in an AI Pipeline

A realistic architecture might look like:

AIApplication
├── DocumentLoader
├── TextProcessor
├── EmbeddingModel
├── VectorStore

└── LLM

Each component has its own responsibility.

For example:

class RAGPipeline:
    def __init__(
        self,
        loader,
        embedder,
        vector_store,
        llm
    ):
        self.loader = loader
  • self.embedder = embedder
  • self.vector_store = vector_store
  • self.llm = llm

This is composition and becomes extremely useful when you study Generative AI, RAG, LangChain, LangGraph, and AI Agents later in your course.

1.12.50SOLID Principles

  • As you become more advanced, you'll encounter the SOLID principles.
  • They help design maintainable object-oriented software.
  • S → Single Responsibility Principle
  • O → Open/Closed Principle
  • L → Liskov Substitution Principle
  • I → Interface Segregation Principle
  • D → Dependency Inversion Principle
  • You don't need to master all of these immediately.

For now, remember the most important one:

  • Single Responsibility Principle
  • A class should have a clear responsibility rather than doing everything.
  • Bad:
class DataSystem:
    def read_database(self):

...

def send_email(self):

...

def train_model(self):

...

def generate_report(self):

...

def upload_to_azure(self):

...

  • Better:
  • DatabaseReader
  • EmailService
  • ModelTrainer
  • ReportGenerator
  • AzureUploader
  • Each class has a clear responsibility.

1.12.51When Should You Use OOP?

  • OOP becomes particularly useful when:
  • Your application is large
  • You have many related objects
  • State needs to be maintained
  • You need reusable components
  • You need clear abstractions
  • Multiple developers work on the project
  • You are building frameworks or services
  • For a tiny script:
print("Hello")
  • creating a class would usually be unnecessary.
  • Don't use OOP simply because you can.
  • Use it when it makes the design clearer.

1.12.52OOP vs Procedural Programming

Procedural

def extract():

...

def transform():

...

def load():

...

  • extract()
  • transform()
  • load()
  • Object-Oriented
class Pipeline:
    def extract(self):

...

def transform(self):

...

def load(self):

...

def run(self):
    self.extract()
  • self.transform()
  • self.load()
  • Neither approach is universally better.
  • Python supports both styles.

For larger applications, OOP can provide useful structure.

1.12.53Important OOP Terminology

TermMeaning
ClassBlueprint for objects
ObjectInstance of a class
AttributeData stored on an object/class
MethodFunction defined inside a class
selfCurrent instance
__init__Initializer method
EncapsulationOrganizing/protecting state and behavior
InheritanceReusing/extending another class
PolymorphismSame interface, different behavior
AbstractionExposing essential interface
CompositionBuilding objects from other objects
super()Access parent-class behavior
@staticmethodMethod without automatic self/cls
@classmethodMethod receiving the class as cls
PropertyControlled attribute-style access
DataclassConvenient data-oriented class

1.12.54Practice Exercise 1 — Employee

Create a class:

class Employee:

...

  • It should contain:
  • name
  • age
  • department
  • salary
  • Create two employee objects.

1.12.55Practice Exercise 2 — Salary

  • Add:
  • calculate_bonus()
  • The method should accept a percentage.

Example:

employee.calculate_bonus(10)

1.12.56Practice Exercise 3 — Bank Account

Create:

class BankAccount:
    with:
        account_number
  • holder_name
  • balance
  • Methods:
  • deposit()
  • withdraw()
  • check_balance()

Do not allow withdrawal if there isn't enough balance.

1.12.57Practice Exercise 4 — Data Pipeline

Create:

class DataPipeline:
    with methods:
        extract()
  • transform()
  • load()
  • run()
  • run() should call:
extract()
transform()
load()

1.12.58Practice Exercise 5 — Inheritance

Create:

Employee
DataEngineer
  • Employee should have:
  • work()
  • DataEngineer should have:
  • build_pipeline()

1.12.59Practice Exercise 6 — Polymorphism

  • Create:
  • CSVProcessor
  • JSONProcessor
  • ExcelProcessor
  • Each should have:
  • process()
  • Then:
processors = [
    CSVProcessor(),
    JSONProcessor(),
    ExcelProcessor()
]
for processor in processors:
    processor.process()

1.12.60Practice Exercise 7 — Custom Exception + OOP

Create:

class DataQualityError(Exception):
    pass

Then create:

class DataValidator:

...

with:
    validate_record_count()

Raise DataQualityError when source and target counts don't match.

1.12.61Interview Questions

1. What is OOP?

OOP is a programming paradigm that organizes software around objects containing data and behavior.

2. What is a class?

A class is a blueprint for creating objects.

3. What is an object?

An object is an instance of a class.

4. What is self?

self refers to the current object instance in an instance method.

5. What is __init__()?

It is an initializer method that runs when an object is created.

6. What is inheritance?

Inheritance allows one class to reuse or extend another class.

7. What is polymorphism?

It allows different objects to provide different implementations of a common operation/interface.

8. What is encapsulation?

It means combining data and related behavior and controlling access to internal state.

9. What is abstraction?

Abstraction exposes essential functionality while hiding implementation details.

10. What is super()?

super() provides access to methods and initialization logic from a parent class.

11. What is method overriding?

When a child class provides its own implementation of a method inherited from a parent class.

12. What is composition?

Composition is building a class using instances of other classes.

13. What is the difference between @staticmethod and @classmethod?

A static method doesn't receive an automatic self or cls; a class method receives the class as cls.

14. What is multiple inheritance?

A class inheriting from more than one parent class.

15. What is duck typing?

Python's approach of focusing on whether an object supports the required behavior rather than requiring a specific declared type.

1.12.62The Most Important OOP Example

Put everything together:

class DataPipeline:
    pipeline_count = 0
def __init__(self, name):
    self.name = name

self.status = "NOT_STARTED"

DataPipeline.pipeline_count += 1

def start(self):
    self.status = "RUNNING"
print(f"{self.name} started")
def complete(self):
    self.status = "SUCCESS"
print(f"{self.name} completed")
def fail(self):
    self.status = "FAILED"
print(f"{self.name} failed")
def __str__(self):
    return (
        f"Pipeline("
            f"name={self.name}, "
            f"status={self.status})"
    )

Create objects:

pipeline1 = DataPipeline("Customer Load")
pipeline2 = DataPipeline("Orders Load")
  • Run them:
  • pipeline1.start()
  • pipeline1.complete()
  • pipeline2.start()
  • pipeline2.fail()
  • Print:
print(pipeline1)
print(pipeline2)

Output:

  • Customer Load started
  • Customer Load completed
  • Orders Load started
  • Orders Load failed
  • Pipeline(name=Customer Load, status=SUCCESS)
  • Pipeline(name=Orders Load, status=FAILED)
  • And:
print(DataPipeline.pipeline_count)
  • gives:
  • 2
  • This one example demonstrates:
Class
Objects

__init__

self
Instance attributes
Class attributes
Methods

__str__

1.12.63OOP Mental Model

Keep this model in your mind:

CLASS

┌────────┴────────┐

│ │

DATA BEHAVIOR

│ │

Attributes Methods

│ │

└────────┬────────┘

OBJECT

┌────────┼────────┐

↓ ↓ ↓

Object 1 Object 2 Object 3

And the four major OOP concepts:

OOP

┌───────────┼───────────┐

↓ ↓ ↓

Encapsulation Inheritance Polymorphism

Abstraction

For your Data Engineering path, focus especially on:

  • Classes and objects
  • __init__() and self
  • Instance vs class attributes
  • Methods
  • Inheritance
  • Method overriding
  • super()
  • Polymorphism
  • Composition
  • Exception handling inside classes
  • Properties
  • Dataclasses

These concepts will become very useful later when you work with APIs, ML pipelines, FastAPI, Flask, MLflow, LangChain, LangGraph, RAG systems, and AI agents.

Next lesson: 1.13 Iterators & Generators — iter(), next(), custom iterators, yield, generator functions, generator expressions, memory efficiency, and how generators are useful for processing large datasets.

Module 1 · Lesson 1.13

Iterators & Generators

1.13 Iterators & Generators

Iterators and generators are important Python concepts for processing data one item at a time instead of loading everything into memory.

This becomes especially useful in Data Engineering, ETL, log processing, APIs, Machine Learning, and large-data applications.

The central idea is:

Large Dataset
One item
Process
Next item
Process

...

Instead of:

Large Dataset
Load everything into memory
Process

1.13.1Why Do We Need Iterators?

Suppose you have:

numbers = [10, 20, 30, 40, 50]

A list stores all these values in memory.

You can loop through it:

for number in numbers:
    print(number)
  • Python internally uses an iterator to move through the elements.
  • Conceptually:
  • List

├── 10

├── 20

├── 30

├── 40

└── 50

Iterator

10 → 20 → 30 → 40 → 50

1.13.2Iterable vs Iterator

These two terms are often confused.

  • Iterable
  • An iterable is an object that can provide its elements one at a time.
  • Examples:
  • list
  • tuple
  • string
  • set
  • dictionary
  • file

Example:

numbers = [10, 20, 30]
for number in numbers:
    print(number)

The list is iterable.

Iterator

An iterator is an object that keeps track of the current position while producing values one at a time.

You can create an iterator using:

iter()

Example:

numbers = [10, 20, 30]
iterator = iter(numbers)

Now iterator is an iterator.

1.13.3The iter() Function

The built-in iter() function converts an iterable into an iterator.

numbers = [10, 20, 30]
iterator = iter(numbers)

Now:

print(iterator)

will show an iterator object rather than the list itself.

1.13.4The next() Function

The next() function retrieves the next value from an iterator.

numbers = [10, 20, 30]
iterator = iter(numbers)
print(next(iterator))
print(next(iterator))
print(next(iterator))

Output:

  • 10
  • 20
  • 30
  • Each call moves the iterator forward.
  • Conceptually:
  • iterator

10 → 20 → 30
Current position

1.13.5What Happens After the Last Item?

Consider:

numbers = [10, 20, 30]
iterator = iter(numbers)
print(next(iterator))
print(next(iterator))
print(next(iterator))
print(next(iterator))
  • The fourth call raises:
  • StopIteration
  • This exception means:

There are no more items to produce.

1.13.6for Loop and Iterators

When you write:

numbers = [10, 20, 30]
for number in numbers:
    print(number)

Python effectively performs iterator-based processing.

Conceptually:

iterator = iter(numbers)
while True:
    try:
        number = next(iterator)
print(number)
except StopIteration:
    break

You normally don't write this manually.

The for loop handles it for you.

1.13.7Why Iterators Are Useful

Iterators allow values to be processed sequentially.

For example:

numbers = [10, 20, 30, 40, 50]
iterator = iter(numbers)
while True:
    try:
        number = next(iterator)
print(number)
except StopIteration:
    break

Only the current value needs to be processed at each step.

1.13.8Creating a Custom Iterator

You can create your own iterator class.

A Python iterator class typically implements:

__iter__()

__next__()

Example:

class CountUp:
    def __init__(self, limit):
        self.current = 1

self.limit = limit

def __iter__(self):
    return self
def __next__(self):
    if self.current > self.limit:
        raise StopIteration
value = self.current

self.current += 1

return value

Use:

counter = CountUp(5)
for number in counter:
    print(number)

Output:

  • 1
  • 2
  • 3
  • 4
  • 5

1.13.9Understanding __iter__()

The __iter__() method returns an iterator.

In our example:

def __iter__(self):
    return self

The object itself is the iterator.

1.13.10Understanding __next__()

__next__() returns the next value.

def __next__(self):
    if self.current > self.limit:
        raise StopIteration
value = self.current

self.current += 1

return value

Important rule:

Value available

return value
No value available
raise StopIteration

1.13.11Iterator Protocol

An object is an iterator if it follows the iterator protocol.

It should provide:

__iter__()

__next__()

Example:

class MyIterator:
    def __iter__(self):
        return self
def __next__(self):

...

This is called the iterator protocol.

1.13.12Iterators and Memory

Consider:

numbers = list(range(1000000))

This creates a list containing one million values.

A generator can produce values when needed:

numbers = range(1000000)

or:

def generate_numbers():
    for number in range(1000000):
        yield number

The generator doesn't need to construct the entire output list first.

This is one of the biggest advantages of generators.

1.13.13What Is a Generator?

A generator is a special kind of iterator that produces values lazily, usually using the yield keyword.

Example:

def numbers():
    yield 10
  • yield 20
  • yield 30
  • Call:
generator = numbers()

Then:

print(next(generator))
print(next(generator))
print(next(generator))

Output:

  • 10
  • 20
  • 30

1.13.14yield vs return

This is extremely important.

return
def example():
    return 10

Once return executes, the function ends.

yield

def example():
    yield 10

yield 20

  • yield 30
  • yield produces a value and pauses the function.
  • The next call continues from where it paused.
  • Conceptually:
yield 10
Pause
next()
yield 20
Pause
next()
yield 30

1.13.15Generator Function

A function containing yield is a generator function.

def generate_numbers():
    yield 1
  • yield 2
  • yield 3
  • Calling it:
result = generate_numbers()

doesn't immediately execute all the code and produce a list.

It creates a generator object.

You can iterate over it:

for number in result:
    print(number)

Output:

  • 1
  • 2
  • 3

1.13.16Generator Execution

Consider:

def numbers():
    print("Start")

yield 10

print("Middle")

yield 20

print("End")

yield 30

Now:

generator = numbers()

At this point, the generator function body hasn't run through to completion.

Then:

print(next(generator))

Output:

  • Start
  • 10
  • Next:
print(next(generator))

Output:

  • Middle
  • 20
  • Next:
print(next(generator))

Output:

End

30

The generator resumes from where it previously paused.

1.13.17Generator with a Loop

A much more practical example:

def generate_numbers(limit):
    for number in range(1, limit + 1):
        yield number

Use:

for number in generate_numbers(5):
    print(number)

Output:

  • 1
  • 2
  • 3
  • 4
  • 5

1.13.18Generator vs List

List

numbers = [x * 2 for x in range(10)]

The results are created and stored.

Generator

numbers = (x * 2 for x in range(10))
  • The results are generated when requested.
  • Conceptually:
  • List
  • → Calculate everything
  • → Store everything
  • → Process
  • Generator
  • → Calculate one
  • → Process
  • → Calculate next
  • → Process

1.13.19Generator Expressions

A generator expression looks similar to a list comprehension.

List comprehension

numbers = [x * 2 for x in range(10)]

Generator expression

numbers = (x * 2 for x in range(10))
  • Notice:
  • [ ] → List
  • ( ) → Generator expression

1.13.20Checking the Difference

numbers_list = [x * 2 for x in range(5)]
numbers_generator = (
    x * 2
    for x in range(5)
)

You can inspect:

print(numbers_list)
print(numbers_generator)

The list contains the values immediately.

The generator is an object that will produce them when iterated.

1.13.21Consuming a Generator

You can use:

for number in numbers_generator:
    print(number)

Output:

  • 0
  • 2
  • 4
  • 6
  • 8

Once a generator is exhausted, it doesn't restart automatically.

1.13.22Generators Are One-Time Iteration

Example:

def generate_numbers():
    yield 1

yield 2

yield 3

numbers = generate_numbers()
for number in numbers:
    print(number)
print("Again:")
for number in numbers:
    print(number)

The second loop produces nothing because the generator has already been exhausted.

If you need to iterate again, create a new generator.

1.13.23Generator with return

A generator can also use return.

Example:

def generate():
    yield 1

yield 2

return

The return signals that the generator is finished.

A normal for loop handles the resulting StopIteration.

1.13.24Generator for Large Files

This is one of the most useful real-world applications.

Suppose you have a huge log file.

Instead of:

def read_file(filename):
    with open(filename) as file:
        return file.readlines()

which creates a list of all lines, use:

def read_file(filename):
    with open(
        filename,
        "r",
        encoding="utf-8"
    ) as file:
        for line in file:
            yield line.strip()

Now:

for line in read_file("application.log"):
    print(line)

The function produces one line at a time.

1.13.25Why This Matters in Data Engineering

Imagine a 20 GB file.

Bad approach:

20 GB File
Read everything
Memory requirement becomes huge

Better:

20 GB File
Read one record
Process
Read next record
Process

This is why generators are extremely valuable in Data Engineering.

1.13.26Generator for CSV Processing

You can create a generator that reads records:

import csv
def read_customers(filename):
    with open(
        filename,
        "r",
        newline="",
        encoding="utf-8"
    ) as file:
        reader = csv.DictReader(file)
for row in reader:
    yield row

Now:

for customer in read_customers(
    "customers.csv"
):
    print(customer["name"])

Only the current row needs to be processed by your application at each iteration.

1.13.27Generator for Data Transformation

You can combine reading and transformation:

def clean_customers(rows):
    for row in rows:
        row["name"] = row["name"].strip()

yield row

Then:

customers = read_customers(
    "customers.csv"
)
cleaned_customers = clean_customers(
    customers
)
for customer in cleaned_customers:
    print(customer)

This creates a streaming-style pipeline:

CSV
read_customers()
clean_customers()
process

1.13.28Generator Pipeline

This is a very important Data Engineering pattern.

def extract():
    for i in range(1, 6):
        yield i
def transform(data):
    for value in data:
        yield value * 10
def filter_data(data):
    for value in data:
        if value >= 30:
            yield value

Now:

data = extract()
data = transform(data)
data = filter_data(data)
for value in data:
    print(value)

Output:

  • 30
  • 40
  • 50

The data flows through the pipeline lazily.

Conceptually:

Extract
Transform
Filter
Output

This is a powerful pattern.

1.13.29Generator Pipeline with Large Data

  • Suppose:
  • 10 million records
  • Instead of creating:
  • 10 million extracted records

+

10 million transformed records

+

10 million filtered records

generators can allow processing to flow incrementally:

Record 1
Transform
Filter
Output
Record 2
Transform
Filter
Output

This can significantly reduce intermediate memory usage.

1.13.30Generator vs Iterator

  • A generator is an iterator.
  • But not every iterator is a generator.
  • Custom iterator
class Counter:
    def __iter__(self):
        return self
def __next__(self):

...

Generator

def counter():
    yield 1

yield 2

The generator automatically implements the iterator behavior for you.

1.13.31Iterator vs Generator Comparison

FeatureIteratorGenerator
Implements iterator protocolYesYes
Usually uses __iter__()YesAutomatically
Usually uses __next__()YesAutomatically
Uses yieldNoYes
Code complexityHigherLower
Memory efficientYesYes
Good for streamingYesYes

1.13.32Custom Iterator vs Generator

Custom iterator

class EvenNumbers:
    def __init__(self, limit):
        self.current = 0

self.limit = limit

def __iter__(self):
    return self
def __next__(self):
    if self.current > self.limit:
        raise StopIteration
value = self.current

self.current += 2

return value

Generator

Much simpler:

def even_numbers(limit):
    current = 0
while current <= limit:
    yield current

current += 2

The generator version is easier to write and maintain.

1.13.33Infinite Generators

Generators can produce an unlimited sequence.

Example:

def infinite_numbers():
    number = 1
while True:
    yield number

number += 1

Use carefully:

numbers = infinite_numbers()
print(next(numbers))
print(next(numbers))
print(next(numbers))

Output:

  • 1
  • 2
  • 3
  • It can continue indefinitely.

1.13.34Limiting an Infinite Generator

You can stop after a certain number of values:

def infinite_numbers():
    number = 1
while True:
    yield number

number += 1

numbers = infinite_numbers()
for _ in range(5):
    print(next(numbers))

Output:

  • 1
  • 2
  • 3
  • 4
  • 5

1.13.35Generator for Batch Processing

Suppose you want to process records in batches.

def batches(data, batch_size):
    for i in range(
        0,
        len(data),
        batch_size
    ):
        yield data[i:i + batch_size]

Use:

data = list(range(1, 11))
for batch in batches(data, 3):
    print(batch)

Output:

[1, 2, 3]

[4, 5, 6]

[7, 8, 9]

[10]

This concept is very useful when working with APIs, databases, and ML datasets.

1.13.36Generator for Database Records

A database query may return a large number of records.

Conceptually:

def fetch_records(cursor):
    for row in cursor:
        yield row

Then:

for row in fetch_records(cursor):
    process(row)

This allows the application to process rows incrementally, depending on how the database driver handles cursor fetching.

1.13.37Generator for API Pagination

Suppose an API returns pages of data.

Conceptually:

def fetch_all_pages():
    page = 1
while True:
    data = fetch_page(page)
if not data:
    break
for record in data:
    yield record

page += 1

Then:

for record in fetch_all_pages():
    process(record)

This is a very common pattern in real-world data ingestion.

1.13.38yield from

Python provides yield from for delegating iteration to another iterable or generator.

Example:

def numbers1():
    yield 1

yield 2

def numbers2():
    yield 3

yield 4

def all_numbers():
    yield from numbers1()

yield from numbers2()

Now:

for number in all_numbers():
    print(number)

Output:

  • 1
  • 2
  • 3
  • 4

1.13.39Why yield from Is Useful

Without yield from:

def all_numbers():
    for number in numbers1():
        yield number
for number in numbers2():
    yield number

With:

def all_numbers():
    yield from numbers1()

yield from numbers2()

The second version is cleaner.

1.13.40Generator Expressions with Conditions

You can add filtering:

numbers = (
    x
    for x in range(100)
    if x % 2 == 0
)

Then:

for number in numbers:
    print(number)

It generates only even numbers.

1.13.41Generator Expressions vs List Comprehensions

List comprehension

squares = [
    x * x
    for x in range(1000000)
]

Creates the complete list.

Generator expression

squares = (
    x * x
    for x in range(1000000)
)

Produces values lazily.

For large sequences where you only need sequential processing, the generator can be much more memory-efficient.

1.13.42Memory Example

Consider:

numbers = [x for x in range(1000000)]

This creates and stores one million values.

With:

numbers = (x for x in range(1000000))

the generator stores the mechanism needed to produce values rather than a million-element list.

The exact memory behavior depends on the expression and objects involved, but the key principle is lazy evaluation.

1.13.43Lazy Evaluation

Lazy evaluation means:

Don't calculate something until it is actually needed.

Example:

def numbers():
    for i in range(5):
        print("Generating", i)

yield i

When you create:

data = numbers()
  • the values aren't all generated.
  • When you request:
  • next(data)
  • the next value is generated.

This is why generators are called lazy.

1.13.44Eager vs Lazy

Eager

Create everything
Store everything
Process

Lazy

Request value
Generate value
Process
Request next value

1.13.45Generator with File Handling

Let's combine your previous lessons.

def error_lines(filename):
    with open(
        filename,
        "r",
        encoding="utf-8"
    ) as file:
        for line in file:
            if "ERROR" in line:
                yield line.strip()

Now:

for error in error_lines(
    "application.log"
):
    print(error)

This is a very practical example of:

File Handling

+

Generators

+

Loops

+

Conditional Statements

1.13.46Generator with Exception Handling

You can also combine exception handling:

def read_numbers(filename):
    try:
        with open(
            filename,
            "r",
            encoding="utf-8"
        ) as file:
            for line in file:
                try:
                    yield int(line.strip())
except ValueError:
    print(
        "Invalid record:",
        line.strip()
    )
except FileNotFoundError:
    print("File not found")

Now:

for number in read_numbers(
    "numbers.txt"
):
    print(number)

This is approaching real-world ETL programming.

1.13.47Generator-Based ETL

Let's build a small ETL pipeline.

Extract

def extract(filename):
    with open(
        filename,
        encoding="utf-8"
    ) as file:
        for line in file:
            yield line.strip()

Transform

def transform(records):
    for record in records:
        yield record.upper()

Filter

def filter_records(records):
    for record in records:
        if record:
            yield record

Pipeline

records = extract("input.txt")
records = transform(records)
records = filter_records(records)
for record in records:
    print(record)

The entire pipeline is lazy.

1.13.48Mental Model for Generator Pipelines

Think of generators like a conveyor belt:

Conveyor Belt

File

Extractor

Transformer

Validator

Loader

  • Database
  • Data doesn't necessarily need to be stored at every stage.
  • One item can flow through the entire pipeline.

1.13.49When Should You Use Generators?

  • Generators are particularly useful when:
  • Processing large files
  • Reading logs
  • Processing database records
  • Processing API pages
  • Streaming data
  • Creating large sequences
  • Building ETL pipelines
  • Processing batches
  • Memory usage matters

1.13.50When Should You Use a List?

  • Use a list when:
  • You need random access
  • You need to iterate multiple times
  • The dataset is reasonably small
  • You need to know its length immediately
  • You need list-specific operations such as indexing or mutation

Example:

numbers = [10, 20, 30, 40]

You can:

print(numbers[2])

Output:

30

A generator isn't designed for random indexing like that.

1.13.51Iterator vs List vs Generator

FeatureListIteratorGenerator
Stores valuesYesUsually noUsually no
LazyNoYesYes
next()NoYesYes
Random indexingYesNoNo
Reusable iterationYesDependsUsually no
Memory efficientLess for huge dataYesYes
Easy to createYesModerateVery easy

1.13.52Common Mistakes

Mistake 1 — Expecting a generator to behave like a list

This doesn't work:

numbers = (x for x in range(10))
print(numbers[0])

Generators don't support normal list indexing.

Mistake 2 — Trying to iterate twice

numbers = (x for x in range(5))
for x in numbers:
    print(x)
for x in numbers:
    print(x)
  • The second loop produces nothing because the generator has already been consumed.
  • Mistake 3 — Converting a huge generator to a list
  • You might write:
data = list(generate_large_dataset())
  • This defeats much of the memory advantage because now all values are materialized into a list.
  • Mistake 4 — Forgetting yield
  • This:
def numbers():
    return [1, 2, 3]

returns a list.

This:

def numbers():
    yield 1
  • yield 2
  • yield 3
  • creates a generator.

1.13.53Practice Exercise 1 — Basic Generator

Create:

def generate_numbers(n):

...

  • It should generate:
  • 1
  • 2
  • 3

...

n

using yield.

1.13.54Practice Exercise 2 — Even Numbers

Create:

def even_numbers(n):

...

  • For:
  • even_numbers(10)
  • expected:
  • 2
  • 4
  • 6
  • 8
  • 10

1.13.55Practice Exercise 3 — Squares

Create:

def squares(n):

...

  • Expected:
  • 1
  • 4
  • 9
  • 16
  • 25

1.13.56Practice Exercise 4 — File Generator

Create:

def read_lines(filename):

...

It should yield one line at a time.

1.13.57Practice Exercise 5 — Error Generator

Create:

def error_lines(filename):

...

It should yield only lines containing:

ERROR

1.13.58Practice Exercise 6 — Batch Generator

Create:

def batches(data, size):

...

Input:

data = list(range(1, 11))

For size 3, expected:

[1, 2, 3]

[4, 5, 6]

[7, 8, 9]

[10]

1.13.59Practice Exercise 7 — Custom Iterator

Create:

class Countdown:

...

Example:

for number in Countdown(5):
    print(number)
  • Expected:
  • 5
  • 4
  • 3
  • 2
  • 1
  • Implement:

__iter__()

__next__()

1.13.60Practice Exercise 8 — ETL Generator

  • Create:
  • extract()
  • transform()
  • filter_data()
  • Use generators to create:
Extract
Transform
Filter

without materializing intermediate lists.

1.13.61Practice Exercise 9 — Large File

  • Assume:
  • sales.log
  • contains millions of lines.
  • Create a generator that yields only:
  • ERROR
  • records.
  • Do not use:
  • readlines()

1.13.62Interview Questions

1. What is an iterator?

An iterator is an object that produces values one at a time and implements the iterator protocol.

2. What is an iterable?

An object that can be iterated over, such as a list, tuple, string, set, dictionary, or file.

3. What is the difference between iterable and iterator?

An iterable can provide an iterator; an iterator maintains iteration state and provides the next value.

4. What does iter() do?

It obtains an iterator from an iterable.

iterator = iter(numbers)

5. What does next() do?

It retrieves the next item from an iterator.

6. What happens when an iterator has no more items?

It raises:

StopIteration

7. What is a generator?

A generator is a convenient way of creating an iterator, typically using yield.

8. What is yield?

yield produces a value and suspends the generator's execution until the next value is requested.

9. What is the difference between yield and return?

return ends a normal function; yield pauses a generator and allows it to resume later.

10. Why are generators memory efficient?

They generate values lazily instead of materializing the entire sequence at once.

11. Can a generator be iterated multiple times?

Normally no. Once exhausted, you need to create a new generator.

12. What is a generator expression?

A compact syntax for creating a generator:

(x * 2 for x in range(10))

13. What is yield from?

It delegates yielding to another iterable or generator.

14. Are all iterators generators?

No.

A custom iterator can implement __iter__() and __next__() without using yield.

15. Why are generators useful in Data Engineering?

They allow large datasets, files, API results, and database records to be processed incrementally with lower memory usage.

1.13.63Most Important Concepts

Remember these five:

1. Iterable

numbers = [1, 2, 3]

Can be iterated.

2. Iterator

iterator = iter(numbers)

Produces values using:

next(iterator)

3. Generator

def numbers():
    yield 1
  • yield 2
  • yield 3
  • Automatically behaves as an iterator.

4. Lazy evaluation

Generate only when needed

5. Memory efficiency

Large data
One item
Process
Next item

1.13.64The Big Picture

You have now learned:

1.10 File Handling

Read large files

1.13 Iterators & Generators

Process one record at a time
Memory-efficient processing

For your Data Engineering path, this is a very important progression:

Python
Functions
OOP
Iterators
Generators
Streaming Data
ETL Pipelines

A pattern worth remembering is:

def process_file(filename):
    with open(
        filename,
        encoding="utf-8"
    ) as file:
        for line in file:
            yield line.strip()

Then:

for record in process_file("sales.csv"):
    process(record)

This simple pattern is the foundation for memory-efficient file processing and streaming-style ETL.

Next lesson: 1.14 Decorators — function wrapping, @decorator syntax, *args/**kwargs, functools.wraps, timing/logging decorators, authentication decorators, and practical Data Engineering examples.

Module 1 · Lesson 1.14

Decorators

Decorators are one of Python's most powerful features.

A decorator allows you to add or modify the behavior of a function or class without changing its original source code.

  • They are heavily used in:
  • APIs
  • Web applications
  • Logging
  • Authentication
  • Performance monitoring
  • Caching
  • Retry mechanisms
  • Data Engineering pipelines
  • Flask / FastAPI
  • ML and AI frameworks

You have already learned functions, OOP, and exception handling. Decorators build directly on those concepts.

1.14.1What Is a Decorator?

Suppose we have:

def greet():
    print("Hello Sreehari")

We want to add:

Before function → "Function started"

After function → "Function completed"

  • without modifying greet().
  • A decorator allows us to do this.
  • Conceptually:
Original Function
Decorator

┌───────────┴───────────┐

↓ ↓

Before Function After Function

↓ ↓

└────── Original ───────┘

1.14.2Functions Are Objects

Before understanding decorators, you need to understand an important Python concept:

Functions are first-class objects.

This means you can:

  • Store a function in a variable
  • Pass a function as an argument
  • Return a function from another function
  • Store functions in lists/dictionaries

Example:

def greet():
    print("Hello")

You can assign it:

message = greet

message()

Output:

  • Hello
  • Both:
  • greet
  • and:
  • message
  • refer to the same function object.

1.14.3Passing a Function as an Argument

Example:

def greet():
    print("Hello")
def execute(function):
    function()

Now:

execute(greet)

Output:

Hello

Here:

greet
passed into execute()
function()
Hello

This ability is the foundation of decorators.

1.14.4Returning a Function

A function can also return another function.

def outer():
    def inner():
        print("Hello")
return inner

Now:

result = outer()

result()

Output:

Hello

Notice:

return inner

not:

return inner()

The first returns the function itself.

1.14.5Your First Decorator

Let's create a simple decorator.

def my_decorator(function):
    def wrapper():
        print("Before function")

function()

print("After function")
return wrapper

Now decorate a function:

def greet():
    print("Hello")

Apply the decorator:

greet = my_decorator(greet)

Now:

greet()

Output:

  • Before function
  • Hello
  • After function

1.14.6How Does This Work?

Originally:

greet
Hello

After:

greet = my_decorator(greet)

the structure becomes:

greet
wrapper
Before
original greet()
Hello
After

The decorator has wrapped the original function.

That's why the internal function is commonly called:

wrapper

1.14.7The @ Syntax

Python provides a convenient syntax.

Instead of:

def greet():
    print("Hello")
greet = my_decorator(greet)

you can write:

@my_decorator

def greet():
    print("Hello")

Then:

greet()

The @ syntax is called decorator syntax.

1.14.8What Does @decorator Actually Mean?

This:

@my_decorator

def greet():
    print("Hello")

is essentially equivalent to:

def greet():
    print("Hello")
greet = my_decorator(greet)

This is extremely important to understand.

1.14.9Basic Decorator Example

def log_function(function):
    def wrapper():
        print("Function started")

function()

print("Function completed")
return wrapper

@log_function

def process_data():
    print("Processing data")

Call:

process_data()

Output:

  • Function started
  • Processing data
  • Function completed

1.14.10The Problem with Arguments

The previous decorator works only with functions that don't require arguments.

Suppose:

def greet(name):
    print(f"Hello {name}")

If we use the previous decorator:

@log_function

def greet(name):
    print(f"Hello {name}")
  • and call:
  • greet("Sreehari")
  • the wrapper doesn't accept name.
  • We need a more flexible decorator.

1.14.11*args and **kwargs

The solution is:

def log_function(function):
    def wrapper(*args, **kwargs):
        print("Function started")
result = function(
    *args,
    **kwargs
)
print("Function completed")
return result
return wrapper

Now it works with many different function signatures.

1.14.12Decorator with Arguments

def log_function(function):
    def wrapper(*args, **kwargs):
        print(
            f"Calling {function.__name__}"
        )
result = function(
    *args,
    **kwargs
)
print(
    f"{function.__name__} completed"
)
return result
return wrapper

Use:

@log_function

def greet(name):
    print(f"Hello {name}")

Call:

greet("Sreehari")

Output:

  • Calling greet
  • Hello Sreehari
  • greet completed

1.14.13Returning the Function Result

This is important.

Suppose:

def add(a, b):
    return a + b

Your decorator should preserve the return value.

def log_function(function):
    def wrapper(*args, **kwargs):
        print("Starting")
result = function(
    *args,
    **kwargs
)
print("Finished")
return result
return wrapper

Now:

@log_function

def add(a, b):
    return a + b

Call:

result = add(10, 20)
print(result)

Output:

  • Starting
  • Finished
  • 30
  • If you forget:
return result

the decorated function would return None.

1.14.14Understanding *args

*args captures positional arguments.

Example:

def example(*args):
    print(args)

Call:

example(10, 20, 30)

Output:

(10, 20, 30)

1.14.15Understanding **kwargs

**kwargs captures keyword arguments.

def example(**kwargs):
    print(kwargs)

Call:

example(

name="Sreehari",
age=37

)

Output:

{'name': 'Sreehari', 'age': 37}

1.14.16Why Decorators Use Both

A general-purpose decorator uses:

def wrapper(*args, **kwargs):
    because it can work with functions like:
        func()
  • func(10)
  • func(10, 20)
  • func(name="Sreehari")
  • func(10, name="Sreehari")
  • This makes decorators reusable.

1.14.17functools.wraps

There is an important issue with decorators.

Consider:

def log_function(function):
    def wrapper(*args, **kwargs):
        return function(*args, **kwargs)
return wrapper

Decorating:

@log_function

def greet():
    """Greets the user."""
print("Hello")

Now:

print(greet.__name__)
  • may show:
  • wrapper
  • instead of:
  • greet

This is because greet now refers to the wrapper.

1.14.18Using functools.wraps

Python provides:

from functools import wraps

Use it like this:

from functools import wraps
def log_function(function):
    @wraps(function)
def wrapper(*args, **kwargs):
    print("Starting")
result = function(
    *args,
    **kwargs
)
print("Finished")
return result
return wrapper

Now:

@log_function

def greet():
    """Greets the user."""
print("Hello")

You can check:

print(greet.__name__)
print(greet.__doc__)

Output:

  • greet
  • Greets the user.
  • Best practice
  • For normal production decorators, use:
  • @wraps(function)

1.14.19Timing Decorator

Decorators are excellent for measuring execution time.

import time
from functools import wraps
def timer(function):
    @wraps(function)
def wrapper(*args, **kwargs):
    start = time.perf_counter()
result = function(
    *args,
    **kwargs
)
end = time.perf_counter()
print(
    f"{function.__name__} "
    f"took {end - start:.4f} seconds"
)
return result
return wrapper

Use:

@timer

def process_data():
    time.sleep(2)
print("Processing...")
  • Call:
  • process_data()
  • Output will be similar to:
  • Processing...
  • process_data took 2.00 seconds

1.14.20Why This Is Useful in Data Engineering

Suppose you have:

@timer

def extract_data():

...

@timer

def transform_data():

...

@timer

def load_data():

...

Now you can measure:

  • Extract → 12.4 seconds
  • Transform → 45.8 seconds
  • Load → 20.2 seconds

without putting timing code manually inside every function.

That's a major advantage of decorators.

1.14.21Logging Decorator

You can create a reusable logging decorator.

from functools import wraps
def log_execution(function):
    @wraps(function)
def wrapper(*args, **kwargs):
    print(
        f"Starting: {function.__name__}"
    )
result = function(
    *args,
    **kwargs
)
print(
    f"Completed: {function.__name__}"
)
return result
return wrapper

Use:

@log_execution

def load_customer_data():
    print("Loading customers")

Output:

  • Starting: load_customer_data
  • Loading customers
  • Completed: load_customer_data

1.14.22Exception Logging Decorator

You can combine decorators with exception handling.

from functools import wraps
def log_errors(function):
    @wraps(function)
def wrapper(*args, **kwargs):
    try:
        return function(
            *args,
            **kwargs
        )
except Exception as error:
    print(
        f"Error in "
        f"{function.__name__}: "
        f"{error}"
    )

raise

return wrapper

Use:

@log_errors

def divide(a, b):
    return a / b

Then:

divide(10, 0)

The decorator logs the error and re-raises it.

1.14.23Why raise Is Important Here

Notice:

except Exception as error:
    print(error)
  • raise
  • The raise sends the exception back to the caller.
  • Without it, you could accidentally hide failures.

In production data pipelines, silently swallowing errors is usually dangerous.

1.14.24Authentication Decorator

Decorators are heavily used for authorization.

Conceptual example:

from functools import wraps
def require_admin(function):
    @wraps(function)
def wrapper(user, *args, **kwargs):
    if user != "admin":
        raise PermissionError(
            "Admin access required"
        )
return function(
    user,
    *args,
    **kwargs
)
return wrapper

Use:

@require_admin

def delete_data(user):
    print("Data deleted")
  • Then:
  • delete_data("admin")
  • works.
  • But:
  • delete_data("guest")
  • raises a permission error.

This is conceptually similar to authentication/authorization decorators used in web frameworks.

1.14.25Decorator with Its Own Arguments

This is an advanced but very important pattern.

Suppose you want:

@repeat(3)

def greet():
    print("Hello")

Here repeat itself receives an argument.

We need three levels of functions.

from functools import wraps
def repeat(times):
    def decorator(function):
        @wraps(function)
def wrapper(*args, **kwargs):
    for _ in range(times):
        function(
            *args,
            **kwargs
        )
return wrapper
return decorator

Use:

@repeat(3)

def greet():
    print("Hello")

Then:

greet()

Output:

  • Hello
  • Hello
  • Hello

1.14.26Understanding Three Levels

This can look confusing initially.

repeat(times)
decorator(function)
wrapper(*args, **kwargs)
  • Think of it as:
  • Level 1 → Configure decorator
  • Level 2 → Receive function
  • Level 3 → Execute function

1.14.27Another Decorator with Configuration

Example:

@retry(attempts=3)

def load_data():

...

The structure is:

def retry(attempts):
    def decorator(function):
        @wraps(function)
def wrapper(*args, **kwargs):

...

return wrapper
return decorator

This pattern is extremely useful in real applications.

1.14.28Retry Decorator

Let's create a practical retry decorator.

from functools import wraps
import time
def retry(attempts):
    def decorator(function):
        @wraps(function)
def wrapper(*args, **kwargs):
    for attempt in range(1, attempts + 1):
        try:
            return function(
                *args,
                **kwargs
            )
except Exception as error:
    print(
        f"Attempt {attempt} failed: "
        f"{error}"
    )
if attempt == attempts:
    raise

time.sleep(1)

return wrapper
return decorator

Use:

@retry(attempts=3)

def connect_to_database():
    print("Connecting...")

...

This is a very practical pattern for:

  • API calls
  • Database connections
  • Network operations
  • Cloud services
  • ETL pipelines

1.14.29Multiple Decorators

You can apply more than one decorator.

@timer

@log_execution

def process_data():
    print("Processing")

Python applies them from the bottom upward.

Conceptually:

process_data
log_execution
timer
final wrapped function

Equivalent conceptually to:

process_data = timer(
    log_execution(process_data)
)

1.14.30Order Matters

  • Consider:
  • @decorator1
  • @decorator2
def function():
    pass

This means approximately:

function = decorator1(
    decorator2(function)
)

Therefore:

decorator2
decorator1

The order can change behavior.

1.14.31Decorators and OOP

Decorators can also be used with methods inside classes.

from functools import wraps
def log_method(function):
    @wraps(function)
def wrapper(*args, **kwargs):
    print(
        f"Calling {function.__name__}"
    )
return function(
    *args,
    **kwargs
)
return wrapper

Use:

class DataPipeline:
    @log_method
def run(self):
    print("Pipeline running")

Now:

pipeline = DataPipeline()

pipeline.run()

Output:

Calling run

Pipeline running

1.14.32Decorators and self

Notice:

def wrapper(*args, **kwargs):
    For a method:
        pipeline.run()
  • the self object is included in args.
  • Conceptually:
  • args

(self, ...)

That's another reason why general-purpose method decorators use:

*args, **kwargs

1.14.33Class Decorators

Decorators aren't limited to functions.

They can also decorate classes.

Example:

def add_message(cls):
    cls.message = "Hello"
return cls

Use:

@add_message

class Employee:
    pass

Now:

employee = Employee()
print(employee.message)

Output:

Hello

The decorator modified the class.

1.14.34Decorator for Data Validation

Suppose you have a function that requires positive numbers.

from functools import wraps
def positive_numbers(function):
    @wraps(function)
def wrapper(a, b):
    if a <= 0 or b <= 0:
        raise ValueError(
            "Numbers must be positive"
        )
return function(a, b)
return wrapper

Use:

@positive_numbers

def multiply(a, b):
    return a * b

Then:

print(multiply(10, 5))

Output:

  • 50
  • But:
  • multiply(-10, 5)
  • raises:
  • ValueError

1.14.35Data Engineering Example

Imagine several ETL functions:

def extract():

...

def transform():

...

def load():

...

You want logging and timing on all of them.

Instead of:

def extract():
    start = time.perf_counter()
print("Starting extract")

...

print("Finished extract")
print(time.perf_counter() - start)

you can write:

@timer

@log_execution

def extract():

...

@timer

@log_execution

def transform():

...

@timer

@log_execution

def load():

...

The actual business logic remains clean.

This is one of the main reasons decorators are useful.

1.14.36Decorators Separate Concerns

Suppose your function does:

Business Logic

You don't necessarily want it mixed with:

  • Logging
  • Timing
  • Authentication
  • Retry
  • Monitoring
  • Validation
  • Decorators allow these concerns to be separated.
  • Conceptually:
  • Function

┌─────────┼─────────┐

↓ ↓ ↓

Logging Timing Retry

│ │ │

└─────────┼─────────┘

Business Logic

This is an important software engineering concept called separation of concerns.

1.14.37Decorators in Flask

If you later study Flask, you'll see code like:

@app.route("/customers")

def customers():
    return "Customer Data"

The:

@app.route(...)

syntax is a decorator.

It tells Flask:

Associate this function with this URL route.

1.14.38Decorators in FastAPI

In FastAPI, you'll see:

@app.get("/customers")

def get_customers():
    return {"status": "success"}

Again:

@app.get(...)

is decorator syntax.

It registers the function as an API endpoint.

This is why understanding decorators now will make later API lessons much easier.

1.14.39Decorators in Python Libraries

  • You'll encounter decorators such as:
  • @property
  • @staticmethod
  • @classmethod
  • @dataclass
  • and framework-specific decorators such as:

@app.route(...)

or:

@app.get(...)

You are already using the concept throughout Python.

1.14.40@property Is a Decorator

Earlier in OOP you saw:

@property

def salary(self):
    return self._salary

property is implemented using Python's descriptor machinery, and the @property syntax provides decorator-like transformation of the method into a property.

So decorators aren't just a theoretical concept—they're part of everyday Python.

1.14.41A Complete Logging + Timing Decorator

Here's a production-style basic version worth understanding:

import time
from functools import wraps
def monitor(function):
    @wraps(function)
def wrapper(*args, **kwargs):
    start = time.perf_counter()
print(
    f"START: {function.__name__}"
)
try:
    result = function(
        *args,
        **kwargs
    )
return result
except Exception as error:
    print(
        f"ERROR: {function.__name__} "
        f"-> {error}"
    )

raise

finally:

elapsed = (
    time.perf_counter()
    - start
)
print(
    f"END: {function.__name__} "
    f"({elapsed:.4f}s)"
)
return wrapper

Use:

@monitor

def process_customer_data():
    print("Processing customer data")
  • Then:
  • process_customer_data()
  • Possible output:
  • START: process_customer_data
  • Processing customer data
  • END: process_customer_data (0.0002s)
  • This combines:
  • Decorator

+

Functions

+

Exception Handling

+

Timing

+

Logging

1.14.42Important Rules for Writing Decorators

Rule 1

Accept the original function:

def decorator(function):
    Rule 2

Create a wrapper:

def wrapper(*args, **kwargs):
    Rule 3
  • Call the original function:
  • function(*args, **kwargs)
  • Rule 4
  • Return its result:
return result
  • Rule 5
  • Preserve metadata:
  • @wraps(function)

So the common pattern is:

from functools import wraps
def decorator(function):
    @wraps(function)
def wrapper(*args, **kwargs):
    # before
result = function(
    *args,
    **kwargs
)

# after

return result
return wrapper

Memorize this structure.

1.14.43Practice Exercise 1 — Logging

Create:

@log_function

def greet(name):
    print(f"Hello {name}")
  • Expected:
  • Starting greet
  • Hello Sreehari
  • Completed greet

1.14.44Practice Exercise 2 — Timer

Create:

@timer

def calculate():

...

  • Measure how long it takes.
  • Use:
  • time.perf_counter()

1.14.45Practice Exercise 3 — Authentication

Create:

@require_login

def dashboard(user):
    print("Welcome to dashboard")

Allow only:

logged_in = True

1.14.46Practice Exercise 4 — Retry

Create:

@retry(3)

def connect():

...

The function should be attempted a maximum of three times.

1.14.47Practice Exercise 5 — Data Engineering Monitor

Create:

@monitor

def load_customer_data():

...

  • Your decorator should:
  • Log start time
  • Execute function
  • Log success
  • Log failure
  • Log execution time

This is a very useful exercise for your Data Engineering work.

1.14.48Practice Exercise 6 — Multiple Decorators

  • Create:
  • @timer
  • @log_function
def transform_data():

...

Observe the order in which the decorators execute.

1.14.49Interview Questions

1. What is a decorator?

A decorator is a callable that modifies or extends the behavior of another function or class without changing its source code.

2. Why are decorators useful?

They allow reusable cross-cutting functionality such as:

  • Logging
  • Timing
  • Authentication
  • Authorization
  • Validation
  • Caching
  • Retry

3. What does @decorator mean?

It is shorthand for:

function = decorator(function)

4. Why do decorators use *args and **kwargs?

To allow the wrapper to work with functions having different positional and keyword arguments.

5. Why use functools.wraps?

It preserves metadata such as the original function's name and documentation.

6. What is a wrapper function?

A function inside the decorator that adds behavior around the original function.

7. Can decorators accept arguments?

Yes.

Example:

@retry(attempts=3)

This requires an additional outer function.

8. Can classes be decorated?

Yes.

9. Can multiple decorators be applied?

  • Yes.
  • @decorator1
  • @decorator2
def function():
    pass

10. What is the execution order of multiple decorators?

They are applied from the bottom upward:

function = decorator1(
    decorator2(function)
)

11. Can a decorator modify the return value?

Yes.

def wrapper(*args, **kwargs):
    result = function(*args, **kwargs)
return result

The decorator can modify result before returning it.

12. Can decorators handle exceptions?

Yes. A wrapper can use try/except/finally.

1.14.50Decorator Mental Model

Remember this:

DECORATOR

Original Function

┌───────┴───────┐

↓ ↓

Before After

│ │

└───────┬───────┘

Function Result

And memorize this basic pattern:

from functools import wraps
def decorator(function):
    @wraps(function)
def wrapper(*args, **kwargs):
    # Before
result = function(
    *args,
    **kwargs
)

# After

return result
return wrapper
  • Once this pattern is clear, most Python decorators become much easier to understand.
  • What you should be able to do after this lesson
  • You should now be comfortable with:
Functions as objects
Passing functions
Returning functions
Wrapper functions
@decorator syntax
*args / **kwargs
functools.wraps
Decorators with arguments
Multiple decorators
Logging / timing / retry
Real-world ETL decorators

A particularly important connection for your course

Later, when you reach:

1.22 APIs in Python → 6.17 Power BI/analysis → 7.x ML → 10.x FastAPI/MLOps → 11.x GenAI/AI Agents

you will repeatedly encounter decorator syntax such as:

  • @app.get("/data")
  • @property
  • @classmethod
  • @staticmethod
  • @cache

Understanding today's lesson will make those topics significantly easier.

Module 1 · Lesson 1.15

List Comprehensions

List comprehension is a concise way to create a new list from an existing iterable.

Instead of writing several lines with a for loop, you can often create the same list in a single readable expression.

It is one of the most commonly used Python features, especially in Data Engineering, Data Analysis, Pandas, and Machine Learning.

1.15.1Why List Comprehensions?

Suppose you want the squares of numbers from 1 to 5.

Traditional approach:

squares = []
for number in range(1, 6):
    squares.append(number ** 2)
print(squares)

Output:

[1, 4, 9, 16, 25]

Using list comprehension:

squares = [number ** 2 for number in range(1, 6)]
print(squares)

Output:

[1, 4, 9, 16, 25]

The second version is shorter and idiomatic Python.

1.15.2Basic Syntax

  • The basic syntax is:
  • [expression for item in iterable]
  • For example:
numbers = [1, 2, 3, 4, 5]
squares = [number ** 2 for number in numbers]

Break it down:

[ expression for item in iterable ]

↓ ↓ ↓

number ** 2 number numbers

  • So:
  • [number ** 2 for number in numbers]
  • means:

For every number in numbers, calculate number ** 2 and put the result into a new list.

1.15.3Simple Example

numbers = [1, 2, 3, 4, 5]
doubled = [number * 2 for number in numbers]
print(doubled)

Output:

[2, 4, 6, 8, 10]

1.15.4Converting Strings

names = ["sreehari", "ravi", "kiran"]
uppercase_names = [
    name.upper()
    for name in names
]
print(uppercase_names)

Output:

['SREEHARI', 'RAVI', 'KIRAN']

This type of transformation is extremely common when cleaning data.

1.15.5Traditional Loop vs List Comprehension

Traditional

numbers = [1, 2, 3, 4, 5]
squares = []
for number in numbers:
    squares.append(number ** 2)

List comprehension

squares = [
    number ** 2
    for number in numbers
]

Both produce the same result.

1.15.6List Comprehension with if

You can filter elements using an if condition.

Syntax:

[expression for item in iterable if condition]

Example:

numbers = [1, 2, 3, 4, 5, 6]
even_numbers = [
    number
    for number in numbers
    if number % 2 == 0
]
print(even_numbers)

Output:

[2, 4, 6]

Meaning:

Take each number
Check if even
If yes → add to list

1.15.7Odd Numbers

numbers = [1, 2, 3, 4, 5, 6]
odd_numbers = [
    number
    for number in numbers
    if number % 2 != 0
]
print(odd_numbers)

Output:

[1, 3, 5]

1.15.8Squares of Even Numbers

You can combine transformation and filtering.

numbers = range(1, 11)
squares = [
    number ** 2
    for number in numbers
    if number % 2 == 0
]
print(squares)

Output:

[4, 16, 36, 64, 100]

Notice:

Input
Filter even numbers
Calculate square
New list

1.15.9if-else in List Comprehension

  • There are two different forms.
  • Filtering
  • [expression for item in iterable if condition]
  • Conditional expression
  • [expression_if_true if condition else expression_if_false
for item in iterable]

Example:

numbers = [1, 2, 3, 4, 5]
result = [
    "Even" if number % 2 == 0 else "Odd"
    for number in numbers
]
print(result)

Output:

['Odd', 'Even', 'Odd', 'Even', 'Odd']

1.15.10Important Difference

These two are different:

Filter

[

number

for number in numbers
if number > 3

]

Output:

[4, 5]

If-else transformation

[

"High" if number > 3 else "Low"

for number in numbers

]

Output:

['Low', 'Low', 'Low', 'High', 'High']

The first removes elements.

The second transforms every element.

1.15.11String Filtering

names = [
    "Sreehari",
    "Ravi",
    "Anil",
    "Suresh"
]
long_names = [
    name
    for name in names
    if len(name) > 5
]
print(long_names)

Output:

['Sreehari', 'Suresh']

1.15.12String Cleaning

Suppose:

names = [
    " Sreehari ",
    " Ravi",
    "Kiran "
]

You can clean them:

clean_names = [
    name.strip()
    for name in names
]

Output:

['Sreehari', 'Ravi', 'Kiran']

This is very common in data preprocessing.

1.15.13Converting Data Types

Suppose you have strings:

values = ["10", "20", "30", "40"]

Convert them to integers:

numbers = [
    int(value)
    for value in values
]

Output:

[10, 20, 30, 40]

1.15.14Applying a Function

You can call a function inside a comprehension.

def clean_name(name):
    return name.strip().title()
names = [
    " sreehari ",
    " ravi ",
    " kiran "
]
clean_names = [
    clean_name(name)
    for name in names
]

Output:

['Sreehari', 'Ravi', 'Kiran']

1.15.15Nested List Comprehensions

You can use more than one for.

Example:

matrix = [
    [1, 2, 3],
    [4, 5, 6],
    [7, 8, 9]
]

Flatten it:

flattened = [
    number
    for row in matrix
    for number in row
]

Output:

[1, 2, 3, 4, 5, 6, 7, 8, 9]

This is equivalent to:

flattened = []
for row in matrix:
    for number in row:
        flattened.append(number)

1.15.16Understanding Nested Comprehension

This:

[

number

for row in matrix
for number in row

]

should be read as:

For each row
For each number in that row
Add number

The order is the same as nested for loops.

1.15.17Creating a Matrix

You can also create a matrix.

matrix = [
    [0 for column in range(3)]
    for row in range(3)
]
print(matrix)

Output:

[

[0, 0, 0],

[0, 0, 0],

[0, 0, 0]

]

1.15.18Multiplication Table

table = [
    number * 5
    for number in range(1, 11)
]
print(table)

Output:

[5, 10, 15, 20, 25, 30, 35, 40, 45, 50]

1.15.19Multiple Conditions

You can use multiple conditions.

numbers = range(1, 21)
result = [
    number
    for number in numbers
    if number % 2 == 0
    if number > 10
]

Output:

[12, 14, 16, 18, 20]

This is similar to:

for number in numbers:
    if number % 2 == 0:
        if number > 10:

...

1.15.20Using and

You can also write:

result = [
    number
    for number in range(1, 21)
    if number % 2 == 0 and number > 10
]

This is often more readable when conditions belong together.

1.15.21Dictionary Comprehensions

Python also supports dictionary comprehensions.

Syntax:

{key: value for item in iterable}

Example:

numbers = [1, 2, 3, 4, 5]
squares = {
    number: number ** 2
    for number in numbers
}

Output:

{

1: 1,

2: 4,

3: 9,

4: 16,

5: 25

}

1.15.22Set Comprehensions

You can create sets too.

numbers = [1, 2, 2, 3, 3, 4]
unique_squares = {
    number ** 2
    for number in numbers
}

Output:

{1, 4, 9, 16}

Duplicate results are automatically removed because it's a set.

1.15.23Generator Expressions

Remember the previous lesson?

A similar syntax creates a generator:

numbers = (
    number ** 2
    for number in range(1, 6)
)

Compare:

# List

numbers = [
    number ** 2
    for number in range(1, 6)
]
with:
    # Generator
numbers = (
    number ** 2
    for number in range(1, 6)
)
  • The brackets matter:
  • [ ] → List comprehension
  • ( ) → Generator expression
  • { } → Set/dictionary comprehension

1.15.24List Comprehension with Functions

Example:

def square(number):
    return number ** 2
numbers = [1, 2, 3, 4, 5]
result = [
    square(number)
    for number in numbers
]

Output:

[1, 4, 9, 16, 25]

1.15.25Data Engineering Example — Cleaning Columns

Suppose:

columns = [
    " customer_id ",
    " customer_name ",
    " order_date ",
    " total_amount "
]

Clean them:

clean_columns = [
    column.strip().lower()
    for column in columns
]

Output:

[

  • 'customer_id',
  • 'customer_name',
  • 'order_date',
  • 'total_amount'

]

This is a very common ETL operation.

1.15.26Data Engineering Example — Filter Null Values

Suppose:

values = [
    10,
    None,
    20,
    None,
    30
]

Remove nulls:

valid_values = [
    value
    for value in values
    if value is not None
]

Output:

[10, 20, 30]

1.15.27Data Engineering Example — Clean Records

Suppose:

records = [
    {"name": " Sreehari ", "age": 37},
    {"name": " Ravi ", "age": 32},
    {"name": " Kiran ", "age": 28}
]

Clean names:

clean_records = [
    {
        **record,
        "name": record["name"].strip()
    }
    for record in records
]

Output:

[

{'name': 'Sreehari', 'age': 37},

{'name': 'Ravi', 'age': 32},

{'name': 'Kiran', 'age': 28}

]

The **record syntax copies the existing dictionary entries.

1.15.28Filtering Records

Suppose:

employees = [
    {"name": "Sreehari", "salary": 100000},
    {"name": "Ravi", "salary": 70000},
    {"name": "Kiran", "salary": 50000}
]

Find employees earning more than ₹60,000:

high_salary = [
    employee
    for employee in employees
    if employee["salary"] > 60000
]

Output:

[

{'name': 'Sreehari', 'salary': 100000},

{'name': 'Ravi', 'salary': 70000}

]

1.15.29Data Engineering Example — Selecting Columns

Suppose:

records = [
    {
        "id": 1,
        "name": "Sreehari",
        "salary": 100000
    },
    {
        "id": 2,
        "name": "Ravi",
        "salary": 70000
    }
]

Extract only names:

names = [
    record["name"]
    for record in records
]

Output:

['Sreehari', 'Ravi']

1.15.30Data Engineering Example — Transformation

Suppose:

prices = [100, 200, 300, 400]

Add 18% tax:

prices_with_tax = [
    price * 1.18
    for price in prices
]

Output:

[118.0, 236.0, 354.0, 472.0]

1.15.31Data Engineering Example — Categorization

scores = [45, 67, 82, 39, 91]
categories = [
    "Pass" if score >= 50 else "Fail"
    for score in scores
]

Output:

['Fail', 'Pass', 'Pass', 'Fail', 'Pass']

1.15.32Data Engineering Example — File Names

Suppose:

files = [
    "sales.csv",
    "customers.csv",
    "orders.json",
    "products.csv",
    "employees.xlsx"
]

Find CSV files:

csv_files = [
    file
    for file in files
    if file.endswith(".csv")
]

Output:

[

  • 'sales.csv',
  • 'customers.csv',
  • 'products.csv'

]

1.15.33Data Engineering Example — Pipeline Names

pipelines = [
    "PL_CUSTOMERS",
    "PL_ORDERS",
    "PL_PRODUCTS",
    "DIM_CUSTOMER"
]

Find pipeline objects:

pipeline_names = [
    name
    for name in pipelines
    if name.startswith("PL_")
]

Output:

[

  • 'PL_CUSTOMERS',
  • 'PL_ORDERS',
  • 'PL_PRODUCTS'

]

1.15.34Nested Data

Suppose you have:

departments = [
    ["HR", "Finance"],
    ["IT", "Data"],
    ["Sales", "Marketing"]
]

Flatten:

all_departments = [
    department
    for group in departments
    for department in group
]

Output:

[

'HR',

'Finance',

'IT',

  • 'Data',
  • 'Sales',
  • 'Marketing'

]

1.15.35List Comprehension with enumerate()

You can combine comprehensions with enumerate().

names = ["Sreehari", "Ravi", "Kiran"]
result = [
    f"{index}: {name}"
    for index, name in enumerate(names, start=1)
]

Output:

[

  • '1: Sreehari',
  • '2: Ravi',
  • '3: Kiran'

]

1.15.36List Comprehension with zip()

Suppose:

names = ["Sreehari", "Ravi", "Kiran"]
ages = [37, 32, 28]

Combine them:

employees = [
    {"name": name, "age": age}
    for name, age in zip(names, ages)
]

Output:

[

{'name': 'Sreehari', 'age': 37},

{'name': 'Ravi', 'age': 32},

{'name': 'Kiran', 'age': 28}

]

This is useful for constructing structured data.

1.15.37Nested Conditions

You can have more complex conditions.

numbers = range(1, 21)
result = [
    number
    for number in numbers
    if number % 2 == 0
    and number % 3 == 0
]

Output:

[6, 12, 18]

1.15.38Avoid Overly Complex Comprehensions

Just because Python allows complex comprehensions doesn't mean you should always use them.

This can become difficult to read:

result = [
    x * y
    for x in range(10)
    for y in range(10)
    if x % 2 == 0
    if y % 3 == 0
]
  • Sometimes normal loops are clearer.
  • Good Python code prioritizes:
  • Readability over cleverness.

1.15.39When NOT to Use List Comprehension

  • Avoid a comprehension when:
  • The logic is very complicated
  • Multiple side effects are involved
  • The expression becomes difficult to understand
  • You need extensive exception handling
  • The result isn't actually a list
  • For example, don't write:

[

print(x)
for x in numbers

]

Use:

for x in numbers:
    print(x)

The comprehension should primarily be used to create a collection, not just to execute side effects.

1.15.40Performance

List comprehensions are often faster than equivalent Python-level for loops for simple transformations because the construct is implemented efficiently.

For example:

squares = [
    x * x
    for x in range(100000)
]

is generally an efficient Python pattern.

However, don't sacrifice readability for tiny performance gains.

1.15.41List Comprehension vs map()

Traditional functional approach:

numbers = [1, 2, 3, 4]
squares = list(
    map(
        lambda x: x ** 2,
        numbers
    )
)

List comprehension:

squares = [
    x ** 2
    for x in numbers
]

For many Python developers, the comprehension is easier to read.

1.15.42List Comprehension vs filter()

Using filter():

numbers = [1, 2, 3, 4, 5, 6]
even_numbers = list(
    filter(
        lambda x: x % 2 == 0,
        numbers
    )
)

Using comprehension:

even_numbers = [
    x
    for x in numbers
    if x % 2 == 0
]

Again, the comprehension is often more readable.

1.15.43Comprehensions and Pandas

  • Once you reach Pandas, you'll see vectorized operations that are often preferable for DataFrame columns.
  • For example:
  • df["name"] = df["name"].str.strip()
  • is generally preferable to manually iterating over DataFrame rows.
  • So:

List comprehensions are excellent for Python collections, but don't automatically use them where Pandas vectorization is more appropriate.

1.15.44Three Types You Should Know

  • Python supports several comprehension forms.
  • List
  • [x * 2 for x in numbers]
  • Set
  • {x * 2 for x in numbers}
  • Dictionary
  • {x: x * 2 for x in numbers}
  • And generator expression:
  • (x * 2 for x in numbers)
  • Remember:
  • [ ] → List
  • {value} → Set
  • {key: value} → Dictionary
  • ( ) → Generator expression

1.15.45Practical Example — Data Cleaning Pipeline

Let's combine everything.

Input:

raw_names = [
    " sreehari ",
    "",
    " RAVI",
    "kiran ",
    None,
    " anil "
]
  • We want:
  • Remove None
  • Remove empty values
  • Remove whitespace
  • Convert to title case
clean_names = [
    name.strip().title()
    for name in raw_names
    if name is not None and name.strip()
]

Output:

[

  • 'Sreehari',
  • 'Ravi',
  • 'Kiran',
  • 'Anil'

]

This is a realistic data-cleaning use case.

1.15.46Practical Example — ETL Transformation

Suppose:

sales = [
    {"product": "Laptop", "amount": 50000},
    {"product": "Phone", "amount": 30000},
    {"product": "Tablet", "amount": 20000}
]

Calculate amounts with 18% tax:

sales_with_tax = [
    {
        **sale,
        "amount_with_tax": sale["amount"] * 1.18
    }
    for sale in sales
]

Result:

[

{

  • 'product': 'Laptop',
  • 'amount': 50000,
  • 'amount_with_tax': 59000

},

...

]

This kind of transformation is common in ETL work.

1.15.47Practical Example — Data Validation

Suppose:

records = [
    {"id": 1, "name": "Sreehari"},
    {"id": 2, "name": ""},
    {"id": 3, "name": "Ravi"},
    {"id": None, "name": "Kiran"}
]

Find valid records:

valid_records = [
    record
    for record in records
    if record["id"] is not None
    and record["name"]
]

Output:

[

{'id': 1, 'name': 'Sreehari'},

{'id': 3, 'name': 'Ravi'}

]

1.15.48Practice Exercise 1 — Squares

Given:

numbers = [1, 2, 3, 4, 5]

Create:

[1, 4, 9, 16, 25]

using a list comprehension.

1.15.49Practice Exercise 2 — Even Numbers

Given:

numbers = range(1, 21)

Create a list containing only even numbers.

Expected:

[2, 4, 6, ..., 20]

1.15.50Practice Exercise 3 — Uppercase

Given:

names = [
    "sreehari",
    "ravi",
    "kiran"
]

Create:

['SREEHARI', 'RAVI', 'KIRAN']

1.15.51Practice Exercise 4 — Clean Names

Given:

names = [
    " Sreehari ",
    " Ravi",
    "Kiran ",
    None,
    ""
]
  • Return only valid cleaned names.
  • Expected:
  • ['Sreehari', 'Ravi', 'Kiran']

1.15.52Practice Exercise 5 — Employees

Given:

employees = [
    {"name": "Sreehari", "salary": 100000},
    {"name": "Ravi", "salary": 70000},
    {"name": "Kiran", "salary": 50000}
]

Return employees whose salary is greater than 60000.

1.15.53Practice Exercise 6 — Nested Lists

Given:

numbers = [
    [1, 2],
    [3, 4],
    [5, 6]
]

Flatten it to:

[1, 2, 3, 4, 5, 6]

1.15.54Practice Exercise 7 — Dictionary Comprehension

Given:

numbers = [1, 2, 3, 4, 5]

Create:

{

1: 1,

2: 4,

3: 9,

4: 16,

5: 25

}

1.15.55Practice Exercise 8 — ETL Transformation

Given:

sales = [
    {"product": "Laptop", "amount": 50000},
    {"product": "Phone", "amount": 30000},
    {"product": "Tablet", "amount": 20000}
]
  • Create a new list containing only products whose amount is greater than 25000.
  • Expected products:
  • Laptop
  • Phone

1.15.56Practice Exercise 9 — Conditional Transformation

Given:

scores = [35, 48, 67, 82, 91]
  • Generate:
  • ['Fail', 'Fail', 'Pass', 'Pass', 'Pass']
  • using a list comprehension.

1.15.57Interview Questions

1. What is a list comprehension?

A concise Python syntax for creating a list from an iterable.

2. What is the basic syntax?

[expression for item in iterable]

3. How do you filter elements?

[

item

for item in items
if condition

]

4. Can list comprehensions contain if-else?

Yes.

[

"Even" if x % 2 == 0 else "Odd"

for x in numbers

]

5. Can you have nested loops?

Yes.

[

value

for row in matrix
for value in row

]

6. What is a dictionary comprehension?

{

key: value

for item in iterable

}

7. What is a set comprehension?

{

expression

for item in iterable

}

8. What is a generator expression?

(

expression

for item in iterable

)

It produces values lazily.

9. Is a list comprehension always better than a normal loop?

No. Readability is more important.

10. Are list comprehensions memory efficient?

They create and store the resulting list. For very large lazy sequences, a generator expression may be more memory efficient.

1.15.58The Most Important Pattern

Memorize this:

result = [
    expression
    for item in iterable
    if condition
]

For example:

squares_of_even_numbers = [
    x ** 2
    for x in range(1, 11)
    if x % 2 == 0
]
  • Result:
  • [4, 16, 36, 64, 100]
  • And remember the difference:
  • List comprehension

[ expression for item in iterable ]

Generator expression

( expression for item in iterable )

Connection to what you've learned

Your Python concepts are now building nicely:

Functions
OOP
Iterators & Generators
Decorators
List Comprehensions
Data Transformation

For your Data Engineering + AI/ML path, list comprehensions will become particularly useful when working with Python collections, API responses, JSON data, file processing, feature preparation, and preprocessing logic.

Next lesson: 1.16 Virtual Environments — venv, environment isolation, pip, requirements.txt, package installation, activation/deactivation, and how to create a clean Python project environment.

Module 1 · Lesson 1.16

Virtual Environments

Lesson focus: This lesson is part of Module 1 — Python Programming. Detailed lesson content can be added here from the corresponding source material.
Module 1 · Lesson 1.17

NumPy Basics

NumPy (Numerical Python) is one of the most important Python libraries for data analysis, machine learning, scientific computing, and AI.

If normal Python lists are the basic building blocks for data, NumPy arrays are the foundation for numerical computing in Python.

You will use NumPy heavily later with:

  • Pandas
  • Machine Learning
  • Scikit-learn
  • TensorFlow
  • PyTorch
  • Statistics
  • Data Analysis

1.17.1What Is NumPy?

  • NumPy stands for:
  • Numerical Python
  • It provides:
  • Multidimensional arrays
  • Fast numerical operations
  • Mathematical functions
  • Matrix operations
  • Statistical operations
  • Random number generation
  • Linear algebra
  • Efficient handling of numerical data
  • Import it using:
import numpy as np

The np alias is the standard convention.

1.17.2Installing NumPy

If NumPy isn't installed:

pip install numpy

Check the installation:

import numpy as np
print(np.__version__)

1.17.3Python List vs NumPy Array

Python list:

numbers = [1, 2, 3, 4, 5]

NumPy array:

numbers = np.array([1, 2, 3, 4, 5])
  • The NumPy version is:
  • NumPy ndarray
  • ndarray means:
  • N-dimensional array.

1.17.4Creating Your First NumPy Array

import numpy as np
numbers = np.array([10, 20, 30, 40, 50])
print(numbers)

Output:

[10 20 30 40 50]

Check its type:

print(type(numbers))

Output:

<class 'numpy.ndarray'>

1.17.5Why NumPy Arrays?

Consider normal Python:

numbers = [1, 2, 3, 4]
result = []
for number in numbers:
    result.append(number * 2)

With NumPy:

numbers = np.array([1, 2, 3, 4])
result = numbers * 2

Output:

[2 4 6 8]

This is called vectorized computation.

1.17.6Vectorization

Instead of:

for number in numbers:
    number * 2

NumPy can operate on the entire array:

numbers * 2

Conceptually:

[1, 2, 3, 4]

× 2

[2, 4, 6, 8]

This is one of NumPy's most important features.

1.17.7NumPy Array Attributes

NumPy arrays have useful properties.

arr = np.array([10, 20, 30, 40])

Type

print(type(arr))

Data type

print(arr.dtype)

Number of dimensions

print(arr.ndim)

Shape

print(arr.shape)

Number of elements

print(arr.size)

1.17.8Understanding ndim

For:

arr = np.array([1, 2, 3, 4])
print(arr.ndim)

Output:

1

This is a 1-dimensional array.

1.17.9Two-Dimensional Array

arr = np.array([
    [1, 2, 3],
    [4, 5, 6]
])

This represents:

1 2 3

4 5 6

Check:

print(arr.ndim)

Output:

2

1.17.10Shape

For:

arr = np.array([
    [1, 2, 3],
    [4, 5, 6]
])
print(arr.shape)

Output:

(2, 3)

  • Meaning:
  • 2 rows
  • 3 columns
  • Think:
  • (shape)
  • (rows, columns)

1.17.11Three-Dimensional Array

You can have more dimensions:

arr = np.array([
    [
        [1, 2],
        [3, 4]
    ],
    [
        [5, 6],
        [7, 8]
    ]
])

Check:

print(arr.ndim)

Output:

3

And:

print(arr.shape)

Output:

(2, 2, 2)

You don't need to master high-dimensional arrays immediately. The important concepts are:

  • 1D → vector-like data
  • 2D → matrix/table-like data
  • 3D+ → higher-dimensional data

1.17.12Array Data Types

NumPy arrays have a fixed data type for their elements in many common cases.

arr = np.array([10, 20, 30])
print(arr.dtype)
  • Possible output:
  • int64
  • For decimal values:
arr = np.array([10.5, 20.5, 30.5])
print(arr.dtype)

Possible output:

float64

1.17.13Common NumPy Data Types

  • You'll commonly see:
  • int8
  • int16
  • int32
  • int64
  • float32
  • float64
  • bool
  • complex
  • For ML, you will frequently encounter:
  • float32
  • float64
  • int32
  • int64

1.17.14Creating Arrays from Lists

arr = np.array([1, 2, 3, 4, 5])

Two-dimensional:

arr = np.array([
    [1, 2],
    [3, 4],
    [5, 6]
])

1.17.15np.zeros()

Creates an array filled with zeros.

zeros = np.zeros(5)
print(zeros)

Output:

[0. 0. 0. 0. 0.]

1.17.16Two-Dimensional Zeros

zeros = np.zeros((3, 4))
print(zeros)

Output:

[

[0. 0. 0. 0.],

[0. 0. 0. 0.],

[0. 0. 0. 0.]

]

Shape:

print(zeros.shape)

Output:

(3, 4)

1.17.17np.ones()

Creates an array filled with ones.

ones = np.ones(5)
print(ones)

Output:

[1. 1. 1. 1. 1.]

Two-dimensional:

ones = np.ones((2, 3))

1.17.18np.full()

Creates an array filled with a specific value.

arr = np.full(5, 7)
print(arr)

Output:

[7 7 7 7 7]

Two-dimensional:

arr = np.full((2, 3), 10)

Output:

[

[10 10 10],

[10 10 10]

]

1.17.19np.arange()

np.arange() is similar to Python's range().

arr = np.arange(1, 10)
print(arr)

Output:

[1 2 3 4 5 6 7 8 9]

Notice that 10 isn't included.

1.17.20arange() with Step

arr = np.arange(0, 20, 2)
print(arr)

Output:

[0 2 4 6 8 10 12 14 16 18]

Syntax:

np.arange(start, stop, step)

1.17.21np.linspace()

linspace() creates evenly spaced values between two points.

arr = np.linspace(0, 10, 5)
print(arr)

Output:

[0. 2.5 5. 7.5 10.]

It creates exactly 5 values.

This is very useful in mathematical calculations and plotting.

1.17.22arange() vs linspace()

  • arange()
  • You specify the step:
  • np.arange(0, 10, 2)
  • linspace()
  • You specify the number of values:
  • np.linspace(0, 10, 6)
  • Remember:
  • arange → step size
  • linspace → number of values

1.17.23Identity Matrix

NumPy can create an identity matrix.

identity = np.eye(3)
print(identity)

Output:

[

[1. 0. 0.],

[0. 1. 0.],

[0. 0. 1.]

]

  • This becomes important later when studying:
  • Linear algebra
  • Matrices
  • Machine Learning
  • Deep Learning

1.17.24Random Numbers

NumPy provides random-number functionality.

random_numbers = np.random.rand(5)
print(random_numbers)
  • You'll get five random floating-point numbers between 0 and 1.
  • For example:
  • [0.31 0.82 0.14 0.67 0.45]
  • The exact values change.

1.17.25Random Integers

numbers = np.random.randint(
    1,
    100,
    5
)
print(numbers)

This generates 5 random integers from 1 through 99.

1.17.26Random Seed

  • Random results normally change each time.
  • For reproducible results:
  • np.random.seed(42)
numbers = np.random.randint(
    1,
    100,
    5
)
print(numbers)

Using the same seed produces the same pseudo-random sequence with the same NumPy random API behavior.

This is important in:

  • Machine Learning experiments
  • Testing
  • Debugging
  • Reproducible analysis

1.17.27Indexing

NumPy indexing is similar to Python lists.

arr = np.array([10, 20, 30, 40, 50])
print(arr[0])

Output:

10

print(arr[2])

Output:

30

1.17.28Negative Indexing

arr = np.array([10, 20, 30, 40, 50])
print(arr[-1])

Output:

50

print(arr[-2])

Output:

40

1.17.292D Array Indexing

Consider:

arr = np.array([
    [10, 20, 30],
    [40, 50, 60]
])

Access row 1, column 2:

print(arr[0, 1])

Output:

20

Access:

print(arr[1, 2])

Output:

  • 60
  • The format is:
  • array[row, column]

1.17.30Slicing

arr = np.array([10, 20, 30, 40, 50])
print(arr[1:4])

Output:

  • [20 30 40]
  • Just like Python slicing:
  • [start : stop]
  • The stop index is excluded.

1.17.31Slicing with Step

arr = np.array([10, 20, 30, 40, 50, 60])
print(arr[::2])

Output:

[10 30 50]

1.17.322D Slicing

arr = np.array([
    [1, 2, 3],
    [4, 5, 6],
    [7, 8, 9]
])

First two rows:

print(arr[:2])

Output:

[

[1 2 3],

[4 5 6]

]

1.17.33Selecting Columns

To select the second column:

print(arr[:, 1])

Output:

[2 5 8]

Meaning:

:

all rows

1
column index 1

1.17.34Selecting a Submatrix

print(arr[:2, 1:])

Output:

[

[2 3],

[5 6]

]

This means:

first 2 rows

+

columns from index 1 onward

1.17.35Array Arithmetic

NumPy makes arithmetic easy.

a = np.array([10, 20, 30])
b = np.array([1, 2, 3])

Addition:

print(a + b)

Output:

[11 22 33]

Subtraction:

print(a - b)

Output:

[9 18 27]

Multiplication:

print(a * b)

Output:

[10 40 90]

Division:

print(a / b)

Output:

[10. 10. 10.]

1.17.36Scalar Operations

arr = np.array([10, 20, 30])

Multiply:

print(arr * 5)

Output:

[50 100 150]

Add:

print(arr + 10)

Output:

[20 30 40]

Every element is affected.

1.17.37Mathematical Functions

NumPy provides many mathematical functions.

arr = np.array([1, 4, 9, 16])

Square root:

print(np.sqrt(arr))

Output:

[1. 2. 3. 4.]

1.17.38Exponential

arr = np.array([1, 2, 3])
print(np.exp(arr))
  • Calculates:

1.17.39Logarithm

arr = np.array([1, 10, 100])
print(np.log(arr))

np.log() calculates the natural logarithm.

For base 10:

print(np.log10(arr))
  • Logarithms become very important later in:
  • Machine Learning
  • Probability
  • Loss functions
  • Optimization
  • Neural networks

1.17.40Trigonometric Functions

angles = np.array([
    0,
    np.pi / 2,
    np.pi
])
print(np.sin(angles))
  • Other functions include:
  • np.cos()
  • np.tan()

1.17.41Aggregation Functions

NumPy can calculate statistics very efficiently.

numbers = np.array([
    10, 20, 30, 40, 50
])

Sum:

print(np.sum(numbers))

Output:

150

Mean:

print(np.mean(numbers))

Output:

30.0

Minimum:

print(np.min(numbers))

Output:

10

Maximum:

print(np.max(numbers))

Output:

50

1.17.42Median

numbers = np.array([
    10, 20, 30, 40, 50
])
print(np.median(numbers))

Output:

30.0

1.17.43Standard Deviation

numbers = np.array([
    10, 20, 30, 40, 50
])
print(np.std(numbers))

Standard deviation measures the spread of values around the mean.

You'll use this extensively in your Statistics and Machine Learning modules.

1.17.44Variance

print(np.var(numbers))

Variance is related to standard deviation:

standard deviation = √variance

1.17.45argmin() and argmax()

These return the index of the minimum/maximum value.

arr = np.array([10, 50, 20, 80, 30])
print(np.argmax(arr))

Output:

  • 3
  • Because:
  • index 0 → 10
  • index 1 → 50
  • index 2 → 20

index 3 → 80 ← maximum

index 4 → 30

Similarly:

print(np.argmin(arr))

Output:

0

1.17.46Boolean Comparisons

NumPy can compare every element.

arr = np.array([10, 20, 30, 40, 50])
print(arr > 25)

Output:

[False False True True True]

This is called boolean masking.

1.17.47Boolean Filtering

You can use that mask to filter values.

arr = np.array([10, 20, 30, 40, 50])
result = arr[arr > 25]
print(result)

Output:

[30 40 50]

This is extremely important.

You'll use this technique constantly in data analysis.

1.17.48Multiple Conditions

Use parentheses around individual conditions.

arr = np.array([
    10, 20, 30, 40, 50
])
result = arr[
    (arr > 20) &
    (arr < 50)
]
print(result)

Output:

  • [30 40]
  • For NumPy arrays, use:
  • & → AND

| → OR

~ → NOT

rather than Python's and, or, and not for element-wise array conditions.

1.17.49Reshaping

Suppose:

arr = np.arange(1, 7)
print(arr)

Output:

[1 2 3 4 5 6]

Reshape:

matrix = arr.reshape(2, 3)

Output:

[

[1 2 3],

[4 5 6]

]

The number of elements must remain the same:

2 × 3 = 6

1.17.50Another Reshape

arr = np.arange(1, 13)
matrix = arr.reshape(3, 4)

Shape:

print(matrix.shape)

Output:

(3, 4)

1.17.51Flattening an Array

Convert a multi-dimensional array into one dimension.

arr = np.array([
    [1, 2, 3],
    [4, 5, 6]
])
flat = arr.flatten()
print(flat)

Output:

[1 2 3 4 5 6]

1.17.52ravel()

Another option:

flat = arr.ravel()
  • Both flatten() and ravel() create a one-dimensional representation, but their memory behavior can differ.
  • For now, remember:
  • flatten() → flatten array

ravel() → flatten when possible without copying

1.17.53Transpose

Transpose swaps rows and columns.

arr = np.array([
    [1, 2, 3],
    [4, 5, 6]
])
print(arr.T)

Output:

[

[1 4],

[2 5],

[3 6]

]

Original shape:

(2, 3)

After transpose:

(3, 2)

Transpose becomes very important in linear algebra and ML.

1.17.54Combining Arrays

Use np.concatenate().

a = np.array([1, 2, 3])
b = np.array([4, 5, 6])
result = np.concatenate(
    [a, b]
)
print(result)

Output:

[1 2 3 4 5 6]

1.17.55Vertical Stack

a = np.array([1, 2, 3])
b = np.array([4, 5, 6])
result = np.vstack([a, b])
print(result)

Output:

[

[1 2 3],

[4 5 6]

]

1.17.56Horizontal Stack

result = np.hstack([a, b])
print(result)

Output:

[1 2 3 4 5 6]

1.17.57Splitting Arrays

arr = np.array([
    1, 2, 3, 4, 5, 6
])
parts = np.split(arr, 3)
print(parts)

Conceptually:

[1, 2]

[3, 4]

[5, 6]

1.17.58Copy vs View

This is an important NumPy concept.

Consider:

arr = np.array([1, 2, 3, 4])
view = arr[:2]
  • A slice often creates a view sharing underlying data.
  • Changing it can affect the original:
  • view[0] = 100
print(arr)

Depending on the operation, you may see:

[100 2 3 4]

To explicitly create an independent copy:

copy = arr[:2].copy()

Then modifications to copy don't change arr.

1.17.59Why Views Matter

  • Views can save memory because NumPy doesn't always need to duplicate data.
  • But they can also cause unexpected changes if you're not aware of shared memory.
  • Remember:
View
May share data
Copy
Independent data

1.17.60Broadcasting

Broadcasting is one of NumPy's most powerful features.

Suppose:

arr = np.array([
    [1, 2, 3],
    [4, 5, 6]
])

Add:

10

result = arr + 10

Output:

[

[11 12 13],

[14 15 16]

]

NumPy effectively applies 10 to every element.

1.17.61Broadcasting with Arrays

arr = np.array([
    [10, 20, 30],
    [40, 50, 60]
])
values = np.array([
    1, 2, 3
])
result = arr + values

Output:

[

[11 22 33],

[41 52 63]

]

NumPy broadcasts:

[1, 2, 3]

across each row.

1.17.62Broadcasting Mental Model

Think:

[1, 2, 3]

[10, 20, 30] → [11, 22, 33]

[40, 50, 60] → [41, 52, 63]

Broadcasting allows compatible arrays of different shapes to participate in arithmetic operations.

1.17.63Matrix Multiplication

  • Normal element-wise multiplication:
  • A * B
  • is different from matrix multiplication.
  • For matrix multiplication use:
  • A @ B

Example:

A = np.array([
    [1, 2],
    [3, 4]
])
B = np.array([
    [5, 6],
    [7, 8]
])
result = A @ B
print(result)

Output:

[

[19 22],

[43 50]

]

  • This becomes extremely important in:
  • Linear algebra
  • Machine Learning
  • Neural Networks
  • Deep Learning

1.17.64Dot Product

For vectors:

a = np.array([1, 2, 3])
b = np.array([4, 5, 6])

Use:

result = np.dot(a, b)

Output:

32

Because:

1×4 + 2×5 + 3×6

= 4 + 10 + 18

= 32

You can also use:

a @ b

for compatible vector/matrix operations.

1.17.65Sorting

arr = np.array([
    50, 10, 40, 20, 30
])
result = np.sort(arr)
print(result)

Output:

[10 20 30 40 50]

1.17.66Unique Values

arr = np.array([
    10, 20, 20, 30, 30, 30
])
print(np.unique(arr))

Output:

[10 20 30]

This is useful in data analysis.

1.17.67Counting Unique Values

values, counts = np.unique(

arr,

return_counts=True

)

print(values)
print(counts)
  • For:
  • [10, 20, 20, 30, 30, 30]
  • you'll get conceptually:
  • values:
  • [10 20 30]
  • counts:
  • [1 2 3]

1.17.68Handling Missing Values

NumPy commonly uses np.nan for missing floating-point values.

arr = np.array([
    10,
    20,
    np.nan,
    40
])

Mean:

print(np.mean(arr))

This will produce nan because the array contains nan.

Instead:

print(np.nanmean(arr))

Output:

  • 23.333333...
  • Other useful functions include:
  • np.nansum()
  • np.nanmean()
  • np.nanmin()
  • np.nanmax()

1.17.69Checking for NaN

arr = np.array([
    10,
    np.nan,
    30
])
print(np.isnan(arr))

Output:

[False True False]

You can use this as a mask:

print(arr[np.isnan(arr)])

1.17.70NumPy and Data Engineering

  • Suppose you receive:
  • Sales:
  • 100
  • 200
  • 300
  • 400
  • 500
  • You could calculate:
sales = np.array([
    100,
    200,
    300,
    400,
    500
])
total = np.sum(sales)
average = np.mean(sales)
maximum = np.max(sales)
minimum = np.min(sales)

This provides a simple numerical-analysis foundation.

1.17.71NumPy and Machine Learning

Suppose:

X = np.array([
    [25, 50000],
    [30, 60000],
    [35, 80000]
])

This could represent:

Age Salary

25 50000

30 60000

35 80000

Then:

print(X.shape)

gives:

(3, 2)

  • Meaning:
  • 3 observations
  • 2 features

This is exactly the type of structure you'll encounter in Machine Learning.

1.17.72NumPy and ML Features

Suppose:

X = np.array([
    [25, 50000],
    [30, 60000],
    [35, 80000],
    [40, 90000]
])

Then:

X[:, 0]

  • selects age:
  • [25 30 35 40]
  • And:

X[:, 1]

selects salary:

[50000 60000 80000 90000]

This kind of slicing becomes very important when preparing ML data.

1.17.73Standardization Preview

Later in Machine Learning, you'll learn feature scaling.

A simple mathematical example:

X = np.array([
    10,
    20,
    30,
    40,
    50
])
mean = np.mean(X)
std = np.std(X)
standardized = (
    X - mean
) / std

This transforms the data around a mean of approximately zero.

You'll study the mathematics behind this in the Statistics and Machine Learning modules.

1.17.74NumPy Functions Worth Memorizing

  • You don't need to memorize everything.
  • Start with these:
  • Creation
  • np.array()
  • np.zeros()
  • np.ones()
  • np.full()
  • np.arange()
  • np.linspace()
  • np.eye()
  • Statistics
  • np.sum()
  • np.mean()
  • np.median()
  • np.std()
  • np.var()
  • np.min()
  • np.max()
  • Shape
  • arr.shape
  • arr.ndim
  • arr.size
  • arr.reshape()
  • arr.flatten()
  • arr.T
  • Math
  • np.sqrt()
  • np.exp()
  • np.log()
  • np.log10()
  • np.sin()
  • np.cos()
  • Array operations
  • np.concatenate()
  • np.vstack()
  • np.hstack()
  • np.split()
  • np.sort()
  • np.unique()
  • Random
  • np.random.rand()
  • np.random.randint()
  • np.random.seed()

1.17.75NumPy Cheat Sheet

import numpy as np

# Create

a = np.array([1, 2, 3])
  • # Zeros
  • np.zeros(5)
  • # Ones
  • np.ones(5)
  • # Range
  • np.arange(1, 10)
  • # Even numbers
  • np.arange(0, 10, 2)
  • # Evenly spaced
  • np.linspace(0, 10, 5)
  • # Identity matrix
  • np.eye(3)
  • # Shape
  • a.shape
  • # Dimensions
  • a.ndim
  • # Number of elements
  • a.size
  • # Data type
  • a.dtype
  • # Reshape
  • a.reshape(3, 1)
  • # Sum
  • np.sum(a)
  • # Mean
  • np.mean(a)
  • # Min / Max
  • np.min(a)
  • np.max(a)
  • # Standard deviation
  • np.std(a)
  • # Square root
  • np.sqrt(a)
  • # Sorting
  • np.sort(a)
  • # Unique
  • np.unique(a)
  • # Random integer
  • np.random.randint(1, 100, 5)

1.17.76Practice Exercise 1

Create:

arr = np.array([
    10, 20, 30, 40, 50
])
  • Find:
  • Sum
  • Mean
  • Minimum
  • Maximum
  • Standard deviation
  • Variance

1.17.77Practice Exercise 2

Create:

arr = np.arange(1, 21)
  • Using NumPy, extract:
  • Even numbers
  • Odd numbers
  • Numbers > 10
  • Numbers between 5 and 15

1.17.78Practice Exercise 3

Create:

arr = np.arange(1, 13)
  • Reshape it into:
  • 3 × 4
  • Expected:

[

[1, 2, 3, 4],

[5, 6, 7, 8],

[9, 10, 11, 12]

]

Then extract:

  • First row
  • Last row
  • First column
  • Last column
  • Middle values

1.17.79Practice Exercise 4

Create:

sales = np.array([
    10000,
    15000,
    12000,
    18000,
    25000
])
  • Calculate:
  • Total sales
  • Average sales
  • Highest sales
  • Lowest sales
  • Sales standard deviation

1.17.80Practice Exercise 5 — ML Dataset

Create:

X = np.array([
    [25, 50000],
    [30, 60000],
    [35, 75000],
    [40, 90000],
    [45, 100000]
])
  • Find:
  • Shape
  • Number of rows
  • Number of columns
  • All ages
  • All salaries
  • Average age
  • Average salary
  • Maximum salary

1.17.81Practice Exercise 6 — Matrix

Create:

A = np.array([
    [1, 2],
    [3, 4]
])
B = np.array([
    [5, 6],
    [7, 8]
])
  • Calculate:
  • A + B
  • A - B
  • A * B
  • A @ B
  • A.T

Pay special attention to the difference between:

  • A * B
  • and:
  • A @ B

1.17.82Interview Questions

1. What is NumPy?

A Python library for numerical computing that provides efficient multidimensional arrays and mathematical operations.

2. What is an ndarray?

NumPy's multidimensional array data structure.

3. Why is NumPy faster than normal Python loops for numerical operations?

It uses optimized implementations and vectorized operations, reducing the need for Python-level loops for many numerical workloads.

4. What is vectorization?

Performing operations on entire arrays instead of explicitly looping through individual elements in Python.

5. What is broadcasting?

A mechanism that allows NumPy to perform operations between arrays with compatible shapes.

6. What does shape represent?

The size of an array along each dimension.

(3, 4)

  • means:
  • 3 rows
  • 4 columns
for a 2D array.

7. What does ndim represent?

The number of dimensions of the array.

8. What does dtype represent?

The data type of the array elements.

9. Difference between arange() and linspace()?

arange → specify step

linspace → specify number of values

10. What is boolean masking?

Using a Boolean array to select elements.

arr[arr > 10]

11. What is the difference between * and @?

* → element-wise multiplication

@ → matrix multiplication

12. What is reshaping?

Changing the dimensions of an array without changing the total number of elements.

13. What is the difference between a view and a copy?

A view may share the underlying data with the original array; a copy has independent data.

1.17.83The Big Picture

You can now see how NumPy fits into your course:

Python
Lists / Dictionaries
NumPy
Numerical Arrays
Vectorization
Statistics
Pandas
Data Analysis
Machine Learning
Deep Learning

The five NumPy concepts I recommend mastering first are:

1. ndarray

2. Indexing & slicing

3. Shape & reshape

4. Vectorized operations

5. Boolean masking

If these five become natural to you, the transition into Pandas and Machine Learning will be much easier.

Next in your syllabus: 1.18 Pandas Basics — Series, DataFrames, reading data, selecting rows/columns, filtering, sorting, missing values, grouping, and the core operations you'll use in Data Engineering and Data Analysis.

Module 1 · Lesson 1.18

Pandas Basics

  • Pandas is one of the most important Python libraries for data analysis, data engineering, machine learning, and ETL.
  • If NumPy is mainly used for numerical arrays, Pandas is designed to work with structured/tabular data.
  • Think of Pandas as:
  • Excel + SQL + Python

You can use Pandas to:

  • Read CSV, Excel, JSON, and database data
  • Filter records
  • Select columns
  • Clean data
  • Handle missing values
  • Sort data
  • Group and aggregate data
  • Join datasets
  • Transform columns
  • Analyze datasets
  • Prepare data for Machine Learning

1.18.1Installing Pandas

Install using:

pip install pandas

Import it using the standard alias:

import pandas as pd

Check the version:

print(pd.__version__)

1.18.2What Is Pandas?

Pandas provides two primary data structures:

Series
One-dimensional data
DataFrame
Two-dimensional tabular data

For example:

Customer Age Salary

Sreehari 37 100000

Ravi 32 70000

Kiran 28 50000

This can be represented using a Pandas DataFrame.

1.18.3Pandas Series

A Series is a one-dimensional labeled data structure.

Create one:

import pandas as pd
ages = pd.Series([37, 32, 28])
print(ages)

Output:

0 37

1 32

2 28

dtype: int64

The left side is the index.

Index Value

0 37

1 32

2 28

1.18.4Series with Custom Index

You can specify your own index.

ages = pd.Series(
    [37, 32, 28],
    index=["Sreehari", "Ravi", "Kiran"]
)
print(ages)

Output:

Sreehari 37

Ravi 32

Kiran 28

Now the index contains names.

1.18.5Accessing Series Values

ages = pd.Series(
    [37, 32, 28],
    index=["Sreehari", "Ravi", "Kiran"]
)

Access by label:

print(ages["Sreehari"])

Output:

37

1.18.6Accessing by Position

You can use:

print(ages.iloc[0])

Output:

37

Remember:

loc → label-based

iloc → position-based

1.18.7Creating a DataFrame

A DataFrame is a two-dimensional table.

data = {
    "Name": ["Sreehari", "Ravi", "Kiran"],
    "Age": [37, 32, 28],
    "Salary": [100000, 70000, 50000]
}
df = pd.DataFrame(data)
print(df)

Output:

Name Age Salary

0 Sreehari 37 100000

1 Ravi 32 70000

2 Kiran 28 50000

This is the most important Pandas object.

1.18.8Understanding DataFrame Structure

Think of a DataFrame as:

Columns

Name Age Salary

↓ ↓ ↓

Row 0 Sreehari 37 100000

Row 1 Ravi 32 70000

Row 2 Kiran 28 50000

  • A DataFrame contains:
  • Rows
  • Columns
  • Index
  • Values
  • Data Types

1.18.9Checking DataFrame Type

print(type(df))

Output:

<class 'pandas.core.frame.DataFrame'>

1.18.10head()

head() displays the first rows.

print(df.head())

By default, it shows the first five rows.

You can specify a number:

print(df.head(2))

1.18.11tail()

Displays the last rows:

print(df.tail())

Or:

print(df.tail(2))

1.18.12shape

print(df.shape)

For our DataFrame:

(3, 3)

  • Meaning:
  • 3 rows
  • 3 columns

1.18.13columns

Get column names:

print(df.columns)

Output:

Index(['Name', 'Age', 'Salary'], dtype='object')

Convert them to a list:

columns = df.columns.tolist()

1.18.14index

Get the row index:

print(df.index)

Output will look similar to:

RangeIndex(start=0, stop=3, step=1)

1.18.15dtypes

Check column data types:

print(df.dtypes)

Example:

Name object

Age int64

Salary int64

dtype: object

Depending on your Pandas/NumPy version and data, exact dtype names may differ.

1.18.16info()

  • One of the most useful commands:
  • df.info()
  • It gives information about:
  • Number of rows
  • Columns
  • Non-null values
  • Data types
  • Memory usage

For data engineering, you should become very comfortable with:

df.info()

1.18.17describe()

Get statistical information:

print(df.describe())
  • For numeric columns, you'll typically get:
  • count
  • mean
  • std
  • min
  • 25%
  • 50%
  • 75%
  • max
  • For example:

Age Salary

count 3.000000 3.000000

mean 32.333333 73333.333333

...

This is extremely useful during Exploratory Data Analysis (EDA).

1.18.18Selecting a Column

Select one column:

print(df["Name"])

Output:

0 Sreehari

1 Ravi

2 Kiran

This returns a Series.

1.18.19Selecting Multiple Columns

print(
    df[
        ["Name", "Salary"]
    ]
)

Output:

Name Salary

0 Sreehari 100000

1 Ravi 70000

2 Kiran 50000

This returns a DataFrame.

1.18.20Dot Notation

You can sometimes write:

  • df.Name
  • instead of:
  • df["Name"]

However, prefer:

df["Name"]

because it works reliably even when column names contain spaces or conflict with DataFrame attributes/methods.

1.18.21Selecting Rows with iloc

  • Suppose:
  • df.iloc[0]
  • This selects the first row.
  • df.iloc[1]
  • selects the second row.

1.18.22Selecting Multiple Rows

df.iloc[0:2]

Gets rows 0 and 1.

Output:

Name Age Salary

0 Sreehari 37 100000

1 Ravi 32 70000

1.18.23Selecting Specific Row and Column

  • df.iloc[0, 1]
  • Meaning:
  • row 0
  • column 1

Output:

37

1.18.24loc

  • loc is label-based.
  • df.loc[0]
  • gets row label 0.

You can select specific columns:

df.loc[0, "Name"]

Output:

Sreehari

1.18.25iloc vs loc

This is important.

iloc
integer position
loc
label

Example:

df.iloc[0]

  • means:
  • Give me the first row by position.
  • df.loc[0]
  • means:

Give me the row whose label is 0.

These can differ when you use a custom index.

1.18.26Filtering Rows

  • Suppose:
  • df
  • contains:

Name Age Salary

0 Sreehari 37 100000

1 Ravi 32 70000

2 Kiran 28 50000

Find employees older than 30:

result = df[df["Age"] > 30]
print(result)

Output:

Name Age Salary

0 Sreehari 37 100000

1 Ravi 32 70000

This is called Boolean filtering.

1.18.27Multiple Conditions

  • Suppose you want:
  • Age > 30
  • AND
  • Salary > 60000
  • Use:
result = df[
    (df["Age"] > 30)
    &
    (df["Salary"] > 60000)
]

Important:

Use:

& → AND

| → OR

with parentheses around conditions.

1.18.28OR Condition

  • Find people who are either:
  • Age > 35
  • OR
  • Salary > 90000
result = df[
    (df["Age"] > 35)
    |
    (df["Salary"] > 90000)
]

1.18.29isin()

Suppose:

names = ["Sreehari", "Ravi"]

Find those employees:

result = df[
    df["Name"].isin(names)
]

This is useful when filtering against a list of values.

1.18.30String Filtering

Find names beginning with S:

result = df[
    df["Name"].str.startswith("S")
]

Find names containing "avi":

result = df[
    df["Name"].str.contains("avi")
]
  • Convert names to uppercase:
  • df["Name"].str.upper()
  • Convert names to lowercase:
  • df["Name"].str.lower()
  • Remove whitespace:
  • df["Name"].str.strip()

These operations are very important for data cleaning.

1.18.31Sorting Data

Sort by salary:

sorted_df = df.sort_values(
    "Salary"
)
print(sorted_df)

Ascending is the default.

1.18.32Descending Sort

sorted_df = df.sort_values(
    "Salary",
    ascending=False
)

Highest salary appears first.

1.18.33Sorting by Multiple Columns

df.sort_values(

by=["Age", "Salary"],
ascending=[True, False]

)

  • Meaning:
  • Age → ascending
  • Salary → descending

1.18.34Adding a New Column

  • Suppose:
  • df
  • contains salary.
  • Add a bonus column:
  • df["Bonus"] = df["Salary"] * 0.10
  • Now:

Name Age Salary Bonus

0 Sreehari 37 100000 10000

1 Ravi 32 70000 7000

2 Kiran 28 50000 5000

1.18.35Creating a Calculated Column

  • df["Annual_Total"] = (
  • df["Salary"] +
  • df["Bonus"]

)

This is a vectorized operation.

You don't need a Python for loop.

1.18.36Renaming Columns

df = df.rename(
    columns={
        "Name": "Employee_Name",
        "Salary": "Annual_Salary"
    }
)
  • Now:
  • Employee_Name
  • Age
  • Annual_Salary

1.18.37Changing Column Names

You can replace all column names:

df.columns = [

"employee_name",

"age",

"salary"

]

This can be useful during ETL standardization.

1.18.38Removing Columns

Remove one column:

df = df.drop(
    columns=["Bonus"]
)

Multiple:

df = df.drop(
    columns=["Bonus", "Age"]
)

1.18.39Removing Rows

Remove row with index 1:

df = df.drop(index=1)

1.18.40Missing Values

Missing values are extremely common in real-world data.

Example:

data = {
    "Name": ["Sreehari", "Ravi", None],
    "Age": [37, None, 28],
    "Salary": [100000, 70000, None]
}
df = pd.DataFrame(data)

It may look like:

Name Age Salary

0 Sreehari 37.0 100000.0

1 Ravi NaN 70000.0

2 NaN 28.0 NaN

Pandas commonly represents missing numeric values as NaN.

1.18.41Detect Missing Values

print(df.isna())

You can also use:

df.isnull()

These are commonly used for checking missing values.

1.18.42Count Missing Values

print(df.isna().sum())

You might get:

Name 1

Age 1

Salary 1

This is an extremely useful data-quality check.

1.18.43Remove Missing Rows

df_clean = df.dropna()

This removes rows containing missing values.

Be careful: you may lose a lot of data.

1.18.44Fill Missing Values

  • For example:
  • df["Age"] = df["Age"].fillna(0)
  • Or use the mean:
  • df["Age"] = df["Age"].fillna(
  • df["Age"].mean()

)

This is common in data preprocessing, although the appropriate strategy depends on the business/data context.

1.18.45Forward Fill

df = df.ffill()

Missing values are filled using the previous available value.

This can be useful for certain time-series datasets.

1.18.46Backward Fill

df = df.bfill()

Missing values are filled from the next available value.

1.18.47Duplicates

  • Find duplicate rows:
  • df.duplicated()
  • Count them:
  • df.duplicated().sum()

1.18.48Remove Duplicates

df = df.drop_duplicates()

You can also consider duplicates based on selected columns:

df = df.drop_duplicates(
    subset=["Name"]
)

1.18.49GroupBy

groupby() is one of the most important Pandas operations.

Suppose:

data = {
    "Department": [
        "IT",
        "IT",
        "HR",
        "HR",
        "Sales"
    ],
    "Salary": [
        100000,
        80000,
        60000,
        70000,
        50000
    ]
}
df = pd.DataFrame(data)

Calculate average salary by department:

result = df.groupby(
    "Department"
)["Salary"].mean()
print(result)

Conceptually:

Department

HR 65000

IT 90000

Sales 50000

1.18.50GroupBy with Sum

df.groupby(

"Department"

)["Salary"].sum()

1.18.51GroupBy with Count

  • df.groupby(
  • "Department"
  • )["Salary"].count()

1.18.52Multiple Aggregations

result = df.groupby(
    "Department"
)["Salary"].agg(
    ["sum", "mean", "min", "max", "count"]
)

This gives a summary table.

1.18.53GroupBy Multiple Columns

  • Suppose you have:
  • Department
  • Location
  • Salary

You can group by:

  • df.groupby(
  • ["Department", "Location"]
  • )["Salary"].mean()

This is similar conceptually to SQL:

SELECT
  • Department,
  • Location,
  • AVG(Salary)
  • FROM employees
  • GROUP BY
  • Department,
  • Location;

This SQL-to-Pandas relationship is very important for you as a Data Engineer.

1.18.54Pandas and SQL

You'll find many similarities.

SQL

SELECT *
FROM employees;
  • Pandas
  • df
  • SQL
SELECT Name, Salary
  • FROM employees;
  • Pandas
  • df[
  • ["Name", "Salary"]

]

SQL

SELECT *
FROM employees
  • WHERE Salary > 60000;
  • Pandas
  • df[
  • df["Salary"] > 60000

]

SQL

SELECT Department, AVG(Salary)
  • FROM employees
  • GROUP BY Department;
  • Pandas
  • df.groupby(
  • "Department"
  • )["Salary"].mean()

This is why Pandas is particularly useful for your Data Engineering work.

1.18.55Reading CSV

One of the most important Pandas operations:

df = pd.read_csv(
    "employees.csv"
)

Then:

print(df.head())

1.18.56Reading Excel

df = pd.read_excel(
    "employees.xlsx"
)

Depending on the Excel file and environment, you may need an Excel engine such as openpyxl.

Install:

pip install openpyxl

1.18.57Writing CSV

df.to_csv(

"output.csv",

index=False

)

index=False prevents Pandas from writing the DataFrame index as an additional column.

1.18.58Writing Excel

df.to_excel(

"output.xlsx",

index=False

)

1.18.59Reading JSON

df = pd.read_json(
    "employees.json"
)

Pandas can also work with JSON structures returned by APIs, although nested JSON may require additional normalization.

1.18.60Reading SQL Data

Pandas can work with database queries through supported database connectors.

Conceptually:

df = pd.read_sql(
    "SELECT * FROM employees",
    connection
)

This is particularly useful in Data Engineering.

You can:

Database
SQL Query
Pandas DataFrame
Transform

CSV / Excel / Database / ML Model

1.18.61Combining DataFrames — concat()

Suppose:

df1 = pd.DataFrame({
    "Name": ["Sreehari", "Ravi"]
})
df2 = pd.DataFrame({
    "Name": ["Kiran", "Anil"]
})

Combine:

result = pd.concat(
    [df1, df2],
    ignore_index=True
)

Output:

Name

0 Sreehari

1 Ravi

2 Kiran

3 Anil

1.18.62Merge — Pandas Equivalent of JOIN

Suppose:

customers = pd.DataFrame({
    "Customer_ID": [1, 2, 3],
    "Name": ["Sreehari", "Ravi", "Kiran"]
})
orders = pd.DataFrame({
    "Customer_ID": [1, 2, 1],
    "Amount": [5000, 3000, 2000]
})

Merge:

result = pd.merge(
    customers,
    orders,
    on="Customer_ID",
    how="inner"
)

This is conceptually similar to:

SELECT *
FROM customers c

INNER JOIN orders o

ON c.Customer_ID = o.Customer_ID;

1.18.63Join Types

  • Pandas supports common join types:
  • inner
  • left
  • right
  • outer
  • cross

Example:

  • pd.merge(
  • df1,
  • df2,
on="Customer_ID",
how="left"

)

1.18.64Applying Functions

You can use .apply().

df["Salary_Lakh"] = df["Salary"].apply(

lambda x: x / 100000

)

However, prefer vectorized operations when practical.

  • For example:
  • df["Salary_Lakh"] = (
  • df["Salary"] / 100000

)

is usually simpler and more efficient.

1.18.65Mapping Values

Suppose:

department_codes = {
    "IT": 1,
    "HR": 2,
    "Sales": 3
}

You can map:

  • df["Department_Code"] = (
  • df["Department"].map(
  • department_codes

)

)

1.18.66Changing Data Types

Suppose:

df["Age"] = df["Age"].astype(int)

You can convert to:

  • float
  • int
  • string
  • For example:
  • df["Customer_ID"] = (
  • df["Customer_ID"].astype("string")

)

Be careful when missing values are present; Pandas' nullable dtypes can be useful in such cases.

1.18.67Date Columns

  • Suppose:
  • df["Order_Date"] = pd.to_datetime(
  • df["Order_Date"]

)

Now Pandas understands it as a datetime column.

You can extract:

  • df["Year"] = df["Order_Date"].dt.year
  • df["Month"] = df["Order_Date"].dt.month
  • df["Day"] = df["Order_Date"].dt.day

1.18.68Date Filtering

For example:

result = df[
    df["Order_Date"] >= "2026-01-01"
]

Or:

result = df[
    (df["Order_Date"] >= "2026-01-01")
    &
    (df["Order_Date"] < "2026-02-01")
]

This becomes extremely useful for data pipelines and reporting.

1.18.69Resetting Index

After filtering:

filtered = df[
    df["Salary"] > 60000
]
  • The original index may remain:
  • 0
  • 1
  • or perhaps:
  • 2
  • 5
  • 8

You can reset it:

filtered = filtered.reset_index(
    drop=True
)

1.18.70Setting an Index

You can make a column the index:

df = df.set_index(
    "Customer_ID"
)
  • Then:
  • df.loc[101]
  • can retrieve the row labeled 101.

1.18.71value_counts()

  • One of the most useful functions for analysis.
  • Suppose:
  • df["Department"]
  • contains:
  • IT
  • IT
  • HR
  • Sales
  • IT
  • HR
  • Run:
  • df["Department"].value_counts()
  • Output conceptually:

IT 3

HR 2

Sales 1

This is useful for understanding categorical distributions.

1.18.72unique()

Get unique values:

df["Department"].unique()

1.18.73nunique()

Count unique values:

df["Department"].nunique()

1.18.74query()

  • Pandas provides a SQL-like filtering syntax.
  • Instead of:
  • df[
  • df["Salary"] > 60000

]

you can write:

df.query(

"Salary > 60000"

)

  • For multiple conditions:
  • df.query(
  • "Age > 30 and Salary > 60000"

)

This can be convenient, though standard Boolean filtering remains very important to understand.

1.18.75sample()

  • Get random rows:
  • df.sample(5)
  • Useful for quickly inspecting data.

1.18.76sort_index()

Sort by index:

df.sort_index()

1.18.77Copying a DataFrame

Use:

new_df = df.copy()

This is preferable when you want an independent DataFrame.

1.18.78Important Pandas Workflow

A typical data-analysis workflow looks like:

Read Data
Inspect Data
Check Data Types
Check Missing Values
Remove Duplicates
Clean Data
Transform Data
Filter Data
Group / Aggregate
Analyze Data
Export Results

In code:

import pandas as pd
df = pd.read_csv("sales.csv")
print(df.head())
print(df.info())
print(df.describe())
print(df.isna().sum())
df = df.drop_duplicates()

df["Amount"] = df["Amount"].fillna(0)

result = (
    df.groupby("Category")["Amount"]
    .sum()
)

result.to_csv(

"category_sales.csv"

)

This is already a small ETL workflow.

1.18.79Complete Data Engineering Example

Let's create a small sales dataset:

import pandas as pd
data = {
    "Customer": [
        "Sreehari",
        "Ravi",
        "Kiran",
        "Anil",
        "Sreehari"
    ],
    "Product": [
        "Laptop",
        "Phone",
        "Laptop",
        "Tablet",
        "Phone"
    ],
    "Amount": [
        80000,
        30000,
        75000,
        25000,
        35000
    ]
}
df = pd.DataFrame(data)

Step 1 — Inspect

print(df.head())

Step 2 — Calculate total sales

total_sales = df["Amount"].sum()
print(total_sales)

Step 3 — Average sales

average_sales = df["Amount"].mean()

Step 4 — Find high-value transactions

high_value = df[
    df["Amount"] > 50000
]

Step 5 — Group by product

product_sales = (
    df.groupby("Product")["Amount"]
    .sum()
)

Step 6 — Sort

product_sales = product_sales.sort_values(
    ascending=False
)
  • Step 7 — Export
  • product_sales.to_csv(
  • "product_sales.csv"

)

You've just performed a basic:

Extract
Transform
Analyze
Load

workflow.

1.18.80Pandas vs NumPy

FeatureNumPyPandas
Main structurendarraySeries/DataFrame
Best forNumerical computingTabular data
LabelsLimitedStrong support
Missing dataBasicExtensive support
SQL-like operationsLimitedStrong
GroupByLimitedExcellent
CSV/ExcelNot primary focusExcellent
Data cleaningBasicExcellent
ML preparationFoundationVery common

Think:

NumPy
Fast numerical arrays
Pandas
Structured/tabular data

And internally, Pandas relies heavily on NumPy and other optimized components.

1.18.81Pandas vs SQL

Since you're learning SQL for Data Science, learn to translate between the two.

SQLPandas
SELECTColumn selection
WHEREBoolean filtering
GROUP BYgroupby()
ORDER BYsort_values()
JOINmerge()
DISTINCTdrop_duplicates() / unique()
COUNTcount() / size()
SUMsum()
AVGmean()
MINmin()
MAXmax()

For a Data Engineer, knowing both perspectives is extremely valuable.

1.18.82Essential Pandas Commands

  • You should become comfortable with these:
  • Create
  • pd.Series()
  • pd.DataFrame()
  • Inspect
  • df.head()
  • df.tail()
  • df.info()
  • df.describe()
  • df.shape
  • df.columns
  • df.dtypes
  • Select
  • df["column"]

df[["col1", "col2"]]

  • df.loc[]
  • df.iloc[]
  • Filter
  • df[df["column"] > value]

df["column"].isin(...)

  • df["column"].str.contains(...)
  • Clean
  • df.isna()
  • df.isna().sum()
  • df.dropna()
  • df.fillna()
  • df.drop_duplicates()
  • Transform
  • df["new_column"] = ...
  • df.rename()
  • df.astype()
  • df.map()
  • Analyze
  • df.groupby()
  • df.sum()
  • df.mean()
  • df.min()
  • df.max()
  • df.value_counts()
  • Combine
  • pd.concat()
  • pd.merge()
  • Files
  • pd.read_csv()
  • pd.read_excel()
  • pd.read_json()
  • df.to_csv()
  • df.to_excel()
  • df.to_json()

1.18.83Mini Project — Sales Analysis

Create the following dataset:

import pandas as pd
data = {
    "Order_ID": [101, 102, 103, 104, 105, 106],
    "Customer": [
        "Sreehari",
        "Ravi",
        "Kiran",
        "Anil",
        "Sreehari",
        "Ravi"
    ],
    "Product": [
        "Laptop",
        "Phone",
        "Laptop",
        "Tablet",
        "Phone",
        "Laptop"
    ],
    "Category": [
        "Electronics",
        "Electronics",
        "Electronics",
        "Electronics",
        "Electronics",
        "Electronics"
    ],
    "Amount": [
        80000,
        30000,
        75000,
        25000,
        35000,
        90000
    ]
}
df = pd.DataFrame(data)

Now perform these tasks.

  • Task 1
  • Display the first 5 records.
  • df.head()
  • Task 2
  • Find the number of rows and columns.
  • df.shape
  • Task 3
  • Find total sales.
  • df["Amount"].sum()
  • Task 4
  • Find average order value.
  • df["Amount"].mean()
  • Task 5
  • Find orders above ₹50,000.
  • df[
  • df["Amount"] > 50000

]

  • Task 6
  • Find total sales by product.
  • df.groupby(
  • "Product"

)["Amount"].sum()

  • Task 7
  • Find the highest-value order.
  • df.loc[
  • df["Amount"].idxmax()

]

  • Task 8
  • Sort orders by amount descending.
  • df.sort_values(
  • "Amount",
ascending=False

)

  • Task 9
  • Find sales by customer.
  • df.groupby(
  • "Customer"

)["Amount"].sum()

Task 10

Export the result:

customer_sales = (
    df.groupby("Customer")["Amount"]
    .sum()
    .reset_index()
)

customer_sales.to_csv(

"customer_sales.csv",

index=False

)

1.18.84Interview Questions

1. What is Pandas?

A Python library for data manipulation and analysis.

2. What are the two main Pandas data structures?

Series

DataFrame

3. What is a DataFrame?

A two-dimensional labeled tabular data structure.

4. What is a Series?

A one-dimensional labeled data structure.

5. Difference between loc and iloc?

loc → label-based

iloc → integer-position-based

6. How do you read a CSV?

pd.read_csv("file.csv")

7. How do you check missing values?

df.isna().sum()

8. How do you remove duplicates?

df.drop_duplicates()

9. How do you group data?

df.groupby("column")

10. How do you join two DataFrames?

pd.merge()

11. How do you sort a DataFrame?

df.sort_values()

12. How do you add a column?

df["NewColumn"] = ...

13. How do you remove a column?

df.drop(

columns=["ColumnName"]

)

14. How do you find unique values?

df["column"].unique()

15. How do you count occurrences of each value?

df["column"].value_counts()

1.18.85What You Should Master

  • For your Data Engineering + AI/ML course, don't try to memorize every Pandas function.
  • Master these first:
  • PANDAS

┌────────────┼────────────┐

↓ ↓ ↓

DataFrame Series Index

├── Read Data

│ ├── CSV

│ ├── Excel

│ └── JSON

├── Inspect

│ ├── head()

│ ├── info()

│ └── describe()

├── Select

│ ├── []

│ ├── loc

│ └── iloc

├── Filter

│ ├── conditions

│ ├── isin()

│ └── str methods

├── Clean

│ ├── isna()

│ ├── fillna()

│ ├── dropna()

│ └── duplicates

├── Transform

│ ├── columns

│ ├── map()

│ └── apply()

├── Analyze

│ ├── groupby()

│ ├── sum()

│ ├── mean()

│ └── value_counts()

└── Combine

├── concat()

└── merge()

Most important takeaway

If you remember only one workflow from this lesson, remember:

import pandas as pd
df = pd.read_csv("data.csv")
  • df.info()
  • df.head()
  • df.describe()
df = df.drop_duplicates()
print(df.isna().sum())
filtered = df[
    df["Amount"] > 50000
]
summary = (
    df.groupby("Category")["Amount"]
    .sum()
    .reset_index()
)

summary.to_csv(

"summary.csv",

index=False

)

That small workflow already covers a large part of what you will do with Pandas in real-world Data Engineering, ETL, analytics, and ML preprocessing.

Next in your syllabus: 1.19 Matplotlib Basics — creating line charts, bar charts, histograms, scatter plots, pie charts, labels, titles, legends, subplots, and visualizing DataFrame/NumPy data.

Module 1 · Lesson 1.19

Matplotlib Basics

Matplotlib is one of the most widely used Python libraries for data visualization.

It allows you to convert data into charts and graphs so that you can identify:

  • Trends
  • Patterns
  • Comparisons
  • Distributions
  • Relationships
  • Outliers

For your Data Engineering + AI/ML course, Matplotlib is especially important for:

  • Data Analysis
  • EDA
  • Statistics
  • Machine Learning
  • Model Evaluation
  • Time Series
  • Business Reporting
  • The standard import is:
import matplotlib.pyplot as plt

plt is the conventional alias.

1.19.1Installing Matplotlib

If it isn't installed:

pip install matplotlib

Check the installation:

import matplotlib
print(matplotlib.__version__)

1.19.2Your First Chart

Let's create a simple line chart.

import matplotlib.pyplot as plt
x = [1, 2, 3, 4, 5]
y = [10, 20, 30, 40, 50]

plt.plot(x, y)

  • plt.show()
  • This produces a line that rises from left to right.
  • The basic workflow is:
Data
plt.plot()
Customize
plt.show()

1.19.3Understanding x and y

Given:

x = [1, 2, 3, 4, 5]
y = [10, 20, 30, 40, 50]

Matplotlib creates points:

(1, 10)

(2, 20)

(3, 30)

(4, 40)

(5, 50)

Then connects them.

1.19.4Adding a Title

  • plt.plot(x, y)
  • plt.title("Sales Trend")
  • plt.show()

The title appears at the top of the chart.

1.19.5Adding X and Y Labels

  • plt.plot(x, y)
  • plt.title("Sales Trend")
  • plt.xlabel("Month")
  • plt.ylabel("Sales")
  • plt.show()

Now the chart clearly explains what each axis represents.

1.19.6Complete Basic Chart

import matplotlib.pyplot as plt
months = [
    "Jan",
    "Feb",
    "Mar",
    "Apr",
    "May"
]
sales = [
    100,
    150,
    130,
    180,
    220
]
  • plt.plot(months, sales)
  • plt.title("Monthly Sales")
  • plt.xlabel("Month")
  • plt.ylabel("Sales")
  • plt.show()

This is a basic business visualization.

1.19.7Line Chart

A line chart is useful for showing trends over time.

Example:

days = [1, 2, 3, 4, 5, 6, 7]
sales = [
    100,
    120,
    150,
    140,
    180,
    200,
    220
]
  • plt.plot(days, sales)
  • plt.title("Weekly Sales")
  • plt.xlabel("Day")
  • plt.ylabel("Sales")
  • plt.show()

Use line charts when you want to answer:

"How did this value change over time?"

1.19.8Markers

You can display individual data points.

plt.plot(

x,

y,

marker="o"

)

plt.show()

Common markers include:

o → circle

s → square

^ → triangle

* → star

x → x

1.19.9Line Styles

You can control the line style.

Examples:

plt.plot(

x,

y,

linestyle="--"

)

Common styles:

- solid

-- dashed

: dotted

-. dash-dot

1.19.10Multiple Lines

You can display multiple datasets.

months = ["Jan", "Feb", "Mar", "Apr"]
sales_2025 = [100, 150, 130, 180]
sales_2026 = [120, 170, 160, 210]
  • plt.plot(
  • months,
  • sales_2025,
label="2025"

)

  • plt.plot(
  • months,
  • sales_2026,
label="2026"

)

  • plt.title("Sales Comparison")
  • plt.xlabel("Month")
  • plt.ylabel("Sales")
  • plt.legend()
  • plt.show()

1.19.11Legend

When multiple datasets are displayed, use:

plt.legend()

The label parameter defines what appears in the legend:

  • plt.plot(
  • months,
  • sales_2025,
label="2025"

)

1.19.12Grid

  • A grid can make charts easier to read.
  • plt.plot(x, y)
  • plt.grid()
  • plt.show()

You can also specify which grid lines:

plt.grid(

axis="y"

)

1.19.13Figure Size

You can control chart size using:

plt.figure(

figsize=(10, 5)

)

Example:

plt.figure(

figsize=(10, 5)

)

  • plt.plot(x, y)
  • plt.show()
  • The dimensions are approximately:
width = 10 inches
height = 5 inches

1.19.14Bar Chart

A bar chart is useful for comparing categories.

products = [
    "Laptop",
    "Phone",
    "Tablet",
    "Monitor"
]
sales = [
    100,
    150,
    80,
    120
]
  • plt.bar(
  • products,
  • sales

)

  • plt.title("Product Sales")
  • plt.xlabel("Product")
  • plt.ylabel("Sales")
  • plt.show()

1.19.15Horizontal Bar Chart

  • Use:
  • plt.barh(
  • products,
  • sales

)

Example:

  • plt.barh(
  • products,
  • sales

)

plt.title("Product Sales")

plt.show()

Horizontal bars are useful when category names are long.

1.19.16Bar Chart with Data Labels

You can display values above bars.

products = [
    "Laptop",
    "Phone",
    "Tablet"
]
sales = [
    100,
    150,
    80
]
bars = plt.bar(
    products,
    sales
)

plt.bar_label(bars)

plt.show()

This makes the exact values easier to read.

1.19.17Histogram

A histogram shows the distribution of numerical data.

Example:

ages = [
    22, 25, 25, 27, 30,
    32, 32, 35, 37, 40,
    42, 45, 45, 50
]
  • plt.hist(ages)
  • plt.title("Age Distribution")
  • plt.xlabel("Age")
  • plt.ylabel("Frequency")
  • plt.show()
  • Histogram answers:
  • How are the values distributed?

1.19.18Histogram Bins

You can control the number of bins:

plt.hist(

ages,

bins=5

)

  • plt.show()
  • More bins:
  • plt.hist(
  • ages,
bins=10

)

The number of bins affects how detailed the distribution appears.

1.19.19Scatter Plot

A scatter plot shows the relationship between two numerical variables.

Suppose:

age = [
    25, 30, 35, 40, 45
]
salary = [
    40000,
    50000,
    65000,
    80000,
    100000
]
  • Create:
  • plt.scatter(
  • age,
  • salary

)

  • plt.title("Age vs Salary")
  • plt.xlabel("Age")
  • plt.ylabel("Salary")
  • plt.show()

This is extremely useful in Machine Learning and EDA.

1.19.20Understanding Scatter Plots

  • Each point represents an observation:
  • Age → X-axis
  • Salary → Y-axis

You can use scatter plots to investigate relationships such as:

  • Age vs Salary
  • Experience vs Salary
  • Advertising vs Sales
  • Height vs Weight
  • Study Hours vs Score

1.19.21Pie Chart

A pie chart shows proportions.

departments = [
    "IT",
    "HR",
    "Sales",
    "Finance"
]
employees = [
    50,
    20,
    20,
    10
]

plt.pie(

employees,

labels=departments

)

plt.title("Employees by Department")

plt.show()

1.19.22Percentages in Pie Chart

  • Use:
  • plt.pie(
  • employees,
labels=departments,
autopct="%1.1f%%"

)

This displays percentages.

1.19.23Exploding a Pie Slice

You can highlight one slice:

explode = [
    0.1,
    0,
    0,
    0
]

plt.pie(

employees,

labels=departments,
explode=explode,
autopct="%1.1f%%"

)

plt.show()

1.19.24Choosing the Right Chart

This is more important than memorizing syntax.

GoalRecommended Chart
Trend over timeLine
Compare categoriesBar
DistributionHistogram
RelationshipScatter
ProportionsPie
Multiple numerical relationshipsScatter
Time-series trendLine

For example:

Monthly sales
Line chart
Sales by product
Bar chart
Customer age distribution
Histogram
Age vs income
Scatter plot

1.19.25Subplots

You can display multiple charts in one figure.

fig, axes = plt.subplots(

2,

2

)

This creates:

+-----------+-----------+

| Chart 1 | Chart 2 |

+-----------+-----------+

| Chart 3 | Chart 4 |

+-----------+-----------+

1.19.26Example of Subplots

fig, axes = plt.subplots(

2,

2,

figsize=(10, 8)

)

axes[0, 0].plot(

[1, 2, 3],

[10, 20, 30]

)

axes[0, 0].set_title(

"Line Chart"

)

axes[0, 1].bar(

["A", "B", "C"],

[10, 20, 15]

)

axes[0, 1].set_title(

"Bar Chart"

)

axes[1, 0].scatter(

[1, 2, 3],

[10, 15, 30]

)

axes[1, 0].set_title(

"Scatter Plot"

)

axes[1, 1].hist(

[10, 12, 12, 15, 20, 20]

)

axes[1, 1].set_title(

"Histogram"

)

plt.tight_layout()

plt.show()

1.19.27plt vs ax

  • There are two common styles in Matplotlib.
  • Pyplot style
  • plt.plot(x, y)
  • plt.title("Sales")
  • plt.show()
  • Object-oriented style
  • fig, ax = plt.subplots()
  • ax.plot(x, y)
  • ax.set_title("Sales")
  • plt.show()
  • For simple charts, plt is easy.

For larger applications and multiple charts, the object-oriented style is usually preferable.

1.19.28Recommended Modern Pattern

  • Use:
  • fig, ax = plt.subplots()
  • ax.plot(

x,

y

)

ax.set_title(

"Sales Trend"

)

ax.set_xlabel(

"Month"

)

ax.set_ylabel(

"Sales"

)

plt.show()

Think:

Figure
Axes
Chart

1.19.29X-Axis Limits

You can control the x-axis:

plt.xlim(

0,

10

)

With the object-oriented API:

ax.set_xlim(

0,

10

)

1.19.30Y-Axis Limits

plt.ylim(

0,

100

)

Or:

ax.set_ylim(

0,

100

)

1.19.31Adding Annotations

You can highlight a particular point.

x = [1, 2, 3, 4, 5]
y = [10, 20, 30, 40, 50]
  • plt.plot(x, y)
  • plt.annotate(
  • "Highest",
xy=(5, 50),
xytext=(3.5, 40),
arrowprops={}

)

plt.show()

Annotations are useful in business reports when you want to highlight:

  • Maximum value
  • Minimum value
  • Important event
  • Outlier
  • Business milestone

1.19.32Saving a Chart

Instead of only displaying:

plt.show()

you can save:

plt.savefig(

"sales_chart.png"

)

Example:

  • plt.plot(x, y)
  • plt.title(
  • "Sales Trend"

)

plt.savefig(

"sales_trend.png",

dpi=300,
bbox_inches="tight"

)

plt.show()

1.19.33Common File Formats

  • Matplotlib can save charts as:
  • PNG
  • JPG
  • SVG
  • PDF

Example:

plt.savefig(

"chart.pdf"

)

1.19.34Plotting NumPy Data

Matplotlib works naturally with NumPy.

import numpy as np
import matplotlib.pyplot as plt
x = np.arange(
    0,
    10,
    0.1
)
y = x ** 2

plt.plot(

x,

y

)

plt.show()

This is one reason NumPy and Matplotlib are commonly used together.

1.19.35Plotting Pandas Data

Suppose:

import pandas as pd
df = pd.DataFrame({
    "Month": [
        "Jan",
        "Feb",
        "Mar",
        "Apr"
    ],
    "Sales": [
        100,
        150,
        130,
        200
    ]
})

You can use:

import matplotlib.pyplot as plt
  • plt.plot(
  • df["Month"],
  • df["Sales"]

)

plt.show()

1.19.36Pandas Built-in Plotting

Pandas can also use Matplotlib underneath:

df.plot(

x="Month",
y="Sales"

)

plt.show()

This is convenient for quick analysis.

1.19.37Business Example — Monthly Sales

import matplotlib.pyplot as plt
months = [
    "Jan",
    "Feb",
    "Mar",
    "Apr",
    "May",
    "Jun"
]
sales = [
    120000,
    150000,
    140000,
    180000,
    210000,
    250000
]

fig, ax = plt.subplots(

figsize=(10, 5)

)

  • ax.plot(
  • months,
  • sales,
marker="o"

)

ax.set_title(

"Monthly Sales Trend"

)

ax.set_xlabel(

"Month"

)

ax.set_ylabel(

"Sales"

)

ax.grid(

axis="y"

)

plt.tight_layout()

plt.show()

This is a realistic business visualization.

1.19.38Business Example — Product Sales

products = [
    "Laptop",
    "Phone",
    "Tablet",
    "Monitor"
]
sales = [
    800000,
    500000,
    250000,
    350000
]

fig, ax = plt.subplots()

bars = ax.bar(
    products,
    sales
)

ax.set_title(

"Sales by Product"

)

ax.set_xlabel(

"Product"

)

ax.set_ylabel(

"Sales"

)

ax.bar_label(

bars

)

plt.show()

1.19.39Business Example — Sales Distribution

Suppose:

sales = [
    10000,
    12000,
    15000,
    18000,
    20000,
    22000,
    25000,
    27000,
    30000,
    50000
]
  • Use:
  • plt.hist(
  • sales,
bins=5

)

plt.title(

"Sales Distribution"

)

plt.xlabel(

"Sales"

)

plt.ylabel(

"Frequency"

)

plt.show()

1.19.40Machine Learning Example

Suppose:

hours = [
    1,
    2,
    3,
    4,
    5,
    6
]
scores = [
    40,
    45,
    50,
    60,
    70,
    80
]
  • Plot:
  • plt.scatter(
  • hours,
  • scores

)

plt.title(

"Study Hours vs Exam Score"

)

plt.xlabel(

"Study Hours"

)

plt.ylabel(

"Exam Score"

)

plt.show()

You can visually investigate whether:

More study hours are associated with higher scores.

This is the type of relationship you'll later model using Linear Regression.

1.19.41Time Series Example

Suppose:

dates = [
    "2026-01-01",
    "2026-02-01",
    "2026-03-01",
    "2026-04-01"
]
revenue = [
    100000,
    120000,
    150000,
    170000
]

Convert dates:

import pandas as pd
dates = pd.to_datetime(
    dates
)
  • Then:
  • plt.plot(
  • dates,
  • revenue

)

plt.title(

"Revenue Trend"

)

plt.xlabel(

"Date"

)

plt.ylabel(

"Revenue"

)

plt.xticks(

rotation=45

)

plt.tight_layout()

plt.show()

1.19.42Working with Missing Values

Matplotlib generally handles missing numerical values by leaving gaps in line plots.

For example:

import numpy as np
x = [1, 2, 3, 4, 5]
y = [
    10,
    20,
    np.nan,
    40,
    50
]

plt.plot(

x,

y,

marker="o"

)

plt.show()

The missing value creates a break in the line.

This is useful when visualizing real-world incomplete data.

1.19.43Multiple Subplots — EDA

In exploratory data analysis, you might want to inspect:

  • Age distribution
  • Salary distribution
  • Age vs Salary
  • Department counts

You can create multiple charts in one figure:

fig, axes = plt.subplots(

2,

2,

figsize=(12, 8)

)

axes[0, 0].hist(

ages

)

axes[0, 0].set_title(

"Age Distribution"

)

axes[0, 1].hist(

salaries

)

axes[0, 1].set_title(

"Salary Distribution"

)

  • axes[1, 0].scatter(
  • ages,
  • salaries

)

axes[1, 0].set_title(

"Age vs Salary"

)

  • axes[1, 1].bar(
  • departments,
  • employee_counts

)

axes[1, 1].set_title(

"Employees by Department"

)

plt.tight_layout()

plt.show()

This type of visualization will become important in Module 5 — Statistics and Module 6 — Data Analysis & Visualization.

1.19.44Matplotlib and Pandas Workflow

A common real-world workflow is:

CSV
Pandas
Clean Data
Analyze Data
Matplotlib
Charts

Example:

import pandas as pd
import matplotlib.pyplot as plt
df = pd.read_csv(
    "sales.csv"
)
monthly_sales = (
    df.groupby("Month")["Amount"]
    .sum()
)

monthly_sales.plot(

kind="line"

)

plt.title(

"Monthly Sales"

)

plt.xlabel(

"Month"

)

plt.ylabel(

"Sales"

)

plt.show()

1.19.45Matplotlib and NumPy Workflow

Another common pattern:

NumPy
Generate/Transform Numerical Data
Matplotlib
Visualize

Example:

import numpy as np
import matplotlib.pyplot as plt
x = np.linspace(
    0,
    10,
    100
)
y = np.sin(x)

plt.plot(

x,

y

)

plt.title(

"Sine Wave"

)

plt.show()

1.19.46Common Plot Parameters

You should recognize these:

plt.plot(

x,

y,

marker="o",
linestyle="--",
label="Sales"

)

  • Important parameters:
  • marker
  • linestyle
  • label
  • linewidth
  • markersize
  • alpha
  • For example, alpha controls transparency:
  • plt.plot(

x,

y,

alpha=0.5

)

1.19.47Transparency

  • Transparency is controlled using alpha.
  • plt.scatter(
  • age,
  • salary,
alpha=0.6

)

  • Values are generally between:
  • 0 → transparent
  • 1 → fully opaque

This is especially useful for scatter plots with many overlapping points.

1.19.48Color

You can specify colors when needed:

plt.plot(

x,

y,

color="blue"

)

  • or:
  • plt.bar(
  • products,
  • sales,
color="green"

)

However, in professional visualization, focus first on clarity and correct chart selection, rather than decoration.

1.19.49Rotating Labels

  • Long x-axis labels can overlap.
  • Use:
  • plt.xticks(
rotation=45

)

Or with an axes object:

ax.tick_params(

axis="x",
rotation=45

)

1.19.50Tight Layout

  • Use:
  • plt.tight_layout()
  • This automatically adjusts spacing.
  • Especially useful when:
  • Labels are long
  • Multiple subplots exist
  • Titles overlap
  • Axis labels are rotated

1.19.51Common Mistakes

  • Mistake 1 — Forgetting plt.show()
  • plt.plot(x, y)
  • In scripts, use:
  • plt.show()
  • Mistake 2 — Different lengths

This is invalid:

x = [1, 2, 3]
y = [10, 20]
  • Both arrays need compatible lengths for a basic line plot.
  • Mistake 3 — Using the wrong chart
  • Don't use:
  • Pie chart → for detailed time trends
  • Line chart → for categorical comparisons
  • Histogram → for categorical names
  • Choose the chart based on the analytical question.

1.19.52Chart Selection Cheat Sheet

"What happened over time?"
LINE
"Which category is bigger?"
BAR
"How are values distributed?"
HISTOGRAM
"Are two variables related?"
SCATTER
"What percentage does each category represent?"
PIE

1.19.53Important Matplotlib Functions

  • Start by remembering these:
  • Basic
  • plt.plot()
  • plt.show()
  • plt.title()
  • plt.xlabel()
  • plt.ylabel()
  • plt.legend()
  • plt.grid()
  • Charts
  • plt.bar()
  • plt.barh()
  • plt.hist()
  • plt.scatter()
  • plt.pie()
  • Layout
  • plt.figure()
  • plt.subplots()
  • plt.tight_layout()
  • Axis
  • plt.xlim()
  • plt.ylim()
  • plt.xticks()
  • plt.yticks()

Output

plt.savefig()

1.19.54Mini Project — Sales Dashboard Visualization

Let's combine Pandas + Matplotlib.

import pandas as pd
import matplotlib.pyplot as plt
data = {
    "Month": [
        "Jan",
        "Feb",
        "Mar",
        "Apr",
        "May",
        "Jun"
    ],
    "Sales": [
        100000,
        120000,
        115000,
        150000,
        180000,
        210000
    ]
}
df = pd.DataFrame(data)

Chart 1 — Sales Trend

fig, ax = plt.subplots(

figsize=(10, 5)

)

  • ax.plot(
  • df["Month"],
  • df["Sales"],
marker="o"

)

ax.set_title(

"Monthly Sales Trend"

)

ax.set_xlabel(

"Month"

)

ax.set_ylabel(

"Sales"

)

ax.grid(

axis="y"

)

  • plt.tight_layout()
  • plt.show()
  • Chart 2 — Sales Bar Chart
  • fig, ax = plt.subplots(
figsize=(10, 5)

)

bars = ax.bar(
    df["Month"],
    df["Sales"]
)

ax.set_title(

"Monthly Sales"

)

ax.set_xlabel(

"Month"

)

ax.set_ylabel(

"Sales"

)

ax.bar_label(

bars

)

plt.tight_layout()

plt.show()

You have now created two different views of the same dataset.

1.19.55Practice Exercise 1 — Line Chart

Create:

months = [
    "Jan",
    "Feb",
    "Mar",
    "Apr",
    "May"
]
revenue = [
    100,
    120,
    150,
    140,
    180
]
  • Create a line chart with:
  • Title
  • X-axis label
  • Y-axis label
  • Markers
  • Grid

1.19.56Practice Exercise 2 — Bar Chart

Given:

products = [
    "Laptop",
    "Phone",
    "Tablet",
    "Monitor"
]
sales = [
    800,
    500,
    300,
    400
]

Create a bar chart.

Also display the values on the bars.

1.19.57Practice Exercise 3 — Histogram

Given:

ages = [
    22, 24, 25, 25, 27,
    28, 30, 30, 32, 35,
    36, 38, 40, 42, 45,
    50, 52
]
  • Create a histogram with 5 bins.
  • Add:
  • Title
  • X-axis label
  • Y-axis label

1.19.58Practice Exercise 4 — Scatter Plot

Given:

experience = [
    1, 2, 3, 4, 5, 6, 7
]
salary = [
    30000,
    35000,
    42000,
    50000,
    60000,
    70000,
    85000
]

Create a scatter plot showing:

Experience vs Salary

1.19.59Practice Exercise 5 — Multiple Lines

Create:

months = [
    "Jan",
    "Feb",
    "Mar",
    "Apr"
]
sales_2025 = [
    100,
    120,
    140,
    160
]
sales_2026 = [
    110,
    130,
    155,
    190
]
  • Plot both lines on the same chart.
  • Include:
  • Title
  • Legend
  • X-axis label
  • Y-axis label
  • Grid

1.19.60Practice Exercise 6 — NumPy + Matplotlib

Create:

import numpy as np

Generate 100 values between 0 and 2π:

x = np.linspace(
    0,
    2 * np.pi,
    100
)

Calculate:

y = np.sin(x)

Plot the sine wave.

This is a great exercise for understanding how NumPy + Matplotlib work together.

1.19.61Practice Exercise 7 — Pandas + Matplotlib

Create:

import pandas as pd
df = pd.DataFrame({
    "Department": [
        "IT",
        "HR",
        "Sales",
        "Finance"
    ],
    "Employees": [
        50,
        20,
        30,
        15
    ]
})

Create a bar chart showing employee count by department.

1.19.62Interview Questions

1. What is Matplotlib?

A Python library used for creating static, animated, and interactive visualizations.

2. What is the standard import?

import matplotlib.pyplot as plt

3. How do you create a line chart?

plt.plot(x, y)

4. How do you create a bar chart?

plt.bar(x, y)

5. How do you create a histogram?

plt.hist(data)

6. How do you create a scatter plot?

plt.scatter(x, y)

7. How do you add a title?

plt.title("My Chart")

8. How do you add axis labels?

plt.xlabel("X")

plt.ylabel("Y")

9. How do you display a legend?

plt.legend()

10. How do you display a chart?

plt.show()

11. How do you save a chart?

plt.savefig("chart.png")

12. What is a subplot?

A way to display multiple axes/charts within one figure.

13. What is the difference between a figure and an axes?

Figure
Overall container
Axes
Individual plotting area

14. What is a histogram used for?

To visualize the distribution of numerical data.

15. What is a scatter plot used for?

To visualize the relationship between two numerical variables.

1.19.63Matplotlib in Your AI/ML Journey

Your course is now progressing through the core Python data stack:

Python
NumPy
Pandas
Matplotlib
Statistics
EDA
Machine Learning
Deep Learning
AI / GenAI

The roles are different:

NumPy
Numerical computation
Pandas
Data manipulation
Matplotlib
Data visualization

A very common real-world workflow is:

CSV / Database / API
Pandas
Data Cleaning
NumPy / Pandas
Statistics
Matplotlib
EDA
Machine Learning

What you should master from this lesson

Don't try to memorize every Matplotlib parameter. Become comfortable with these:

  • plt.plot()
  • plt.bar()
  • plt.hist()
  • plt.scatter()
  • plt.title()
  • plt.xlabel()
  • plt.ylabel()
  • plt.legend()
  • plt.grid()
  • plt.subplots()
  • plt.tight_layout()
  • plt.savefig()
  • plt.show()

And, most importantly, understand when to use each chart.

Next in your syllabus: 1.20 Working with CSV & Excel — reading/writing files, handling headers, selecting sheets, data types, missing values, Excel workbooks, and building practical CSV/Excel ETL workflows with Pandas.

Module 1 · Lesson 1.20

Working with CSV & Excel

1.20 Working with CSV & Excel

Working with CSV and Excel files is one of the most practical skills in Python, especially for Data Engineering, Data Analysis, ETL, and Machine Learning.

A very common real-world flow is:

CSV / Excel
Python
Pandas
Read Data
Clean Data
Transform Data
Analyze Data
Export Data

Since you're learning Data Engineering, this topic is particularly important because many business systems still exchange data through CSV and Excel files.

1.20.1What is CSV?

CSV means:

Comma-Separated Values

A CSV file stores tabular data as plain text.

Example:

  • Employee_ID,Name,Department,Salary
  • 101,Sreehari,IT,100000
  • 102,Ravi,HR,70000
  • 103,Kiran,Sales,60000
  • Conceptually:
  • CSV

┌──────┬──────┬──────┐

│ ID │ Name │ Dept │

├──────┼──────┼──────┤

│ 101 │ ... │ ... │

│ 102 │ ... │ ... │

└──────┴──────┴──────┘

1.20.2What is Excel?

Excel files usually have extensions such as:

  • .xlsx
  • .xls
  • An Excel workbook can contain:
Workbook
├── Sheet1

├── Sheet2

└── Sheet3

Unlike CSV, Excel supports features such as:

  • Multiple worksheets
  • Formulas
  • Formatting
  • Charts
  • Cell types
  • Tables
  • Filters

1.20.3CSV vs Excel

FeatureCSVExcel
Extension.csv.xlsx
Multiple sheets
Formatting
Formulas
LightweightLess lightweight
Easy to process
Pandas supportExcellentExcellent
Database/ETL usageVery commonCommon
Human-friendlyModerateExcellent

For automated data pipelines, CSV is often simpler.

For business users, Excel is extremely common.

1.20.4Required Libraries

Install Pandas:

pip install pandas

For Excel .xlsx files, install openpyxl:

pip install openpyxl

Then:

import pandas as pd

1.20.5Creating a Sample DataFrame

Let's start with data in Python:

import pandas as pd
data = {
    "Employee_ID": [101, 102, 103, 104],
    "Name": [
        "Sreehari",
        "Ravi",
        "Kiran",
        "Anil"
    ],
    "Department": [
        "IT",
        "HR",
        "Sales",
        "IT"
    ],
    "Salary": [
        100000,
        70000,
        60000,
        90000
    ]
}
df = pd.DataFrame(data)
print(df)

Output:

Employee_ID Name Department Salary

0 101 Sreehari IT 100000

1 102 Ravi HR 70000

2 103 Kiran Sales 60000

3 104 Anil IT 90000

1.20.6Writing a CSV File

  • Use:
  • df.to_csv(
  • "employees.csv",
index=False

)

  • This creates:
  • employees.csv
  • The file will contain:
  • Employee_ID,Name,Department,Salary
  • 101,Sreehari,IT,100000
  • 102,Ravi,HR,70000
  • 103,Kiran,Sales,60000
  • 104,Anil,IT,90000

1.20.7Why index=False?

  • Pandas has its own index:
  • 0
  • 1
  • 2
  • 3
  • If you write:
  • df.to_csv(
  • "employees.csv"

)

the index may become an extra column:

  • ,Employee_ID,Name,Department,Salary
  • 0,101,Sreehari,IT,100000
  • 1,102,Ravi,HR,70000
  • Usually you don't want that.

Therefore:

index=False

is commonly used when exporting data.

1.20.8Reading a CSV File

Use:

df = pd.read_csv(
    "employees.csv"
)

Then:

print(df)

1.20.9Inspecting the CSV

  • After loading:
  • df.head()
  • Check structure:
  • df.info()
  • Check statistics:
  • df.describe()
  • Check columns:
print(df.columns)

Check dimensions:

print(df.shape)

A good habit is:

df = pd.read_csv("employees.csv")
print(df.head())
print(df.shape)

df.info()

1.20.10Reading a CSV from a Different Folder

You can specify a path.

Windows:

df = pd.read_csv(
    r"C:\Data\employees.csv"
)

Or:

df = pd.read_csv(
    "C:/Data/employees.csv"
)

Using r before a Windows path creates a raw string:

r"C:\Data\employees.csv"

This avoids problems with backslashes.

1.20.11Reading CSV with Different Delimiters

  • CSV doesn't always use commas.
  • For example:
  • Employee_ID;Name;Department;Salary
  • 101;Sreehari;IT;100000
  • 102;Ravi;HR;70000
  • Use:
df = pd.read_csv(
    "employees.csv",
    sep=";"
)

Another common delimiter is tab:

df = pd.read_csv(
    "employees.txt",
    sep="\t"
)

1.20.12Handling Headers

Normally Pandas assumes the first row contains column names.

Example:

  • ID,Name,Salary
  • 101,Sreehari,100000
  • So:
df = pd.read_csv(
    "employees.csv"
)
  • automatically uses:
  • ID
  • Name
  • Salary
  • as column names.

1.20.13CSV Without a Header

  • Suppose the file contains:
  • 101,Sreehari,100000
  • 102,Ravi,70000
  • 103,Kiran,60000
  • Read it using:
df = pd.read_csv(
    "employees.csv",
    header=None
)
  • Pandas will create:
  • 0
  • 1
  • 2
  • as column names.

1.20.14Providing Column Names

You can specify them:

df = pd.read_csv(
    "employees.csv",
    header=None,
    names=[
        "Employee_ID",
        "Name",
        "Salary"
    ]
)

1.20.15Selecting Specific Columns While Reading

  • Suppose your CSV has:
  • Employee_ID
  • Name
  • Department
  • Salary
  • Location
  • Joining_Date
  • You only need:
  • Name
  • Salary

You can load only those:

df = pd.read_csv(
    "employees.csv",
    usecols=[
        "Name",
        "Salary"
    ]
)

This can reduce memory usage when working with large files.

1.20.16Reading a Large CSV in Chunks

Suppose your CSV contains millions of records.

Instead of loading everything:

df = pd.read_csv(
    "large_file.csv"
)

you can process it in chunks:

for chunk in pd.read_csv(
    "large_file.csv",
    chunksize=100000
):
    print(chunk.shape)

Conceptually:

Large CSV
100,000 rows
Process
100,000 rows
Process
100,000 rows

...

This is an important concept for Data Engineering.

1.20.17Reading Only the First N Rows

df = pd.read_csv(
    "employees.csv",
    nrows=100
)

This loads only the first 100 rows.

Useful for testing a large file.

1.20.18Skipping Rows

You can skip rows:

df = pd.read_csv(
    "employees.csv",
    skiprows=2
)

This can be useful when files contain metadata or unwanted header rows.

1.20.19Handling Missing Values in CSV

  • Suppose:
  • ID,Name,Salary
  • 101,Sreehari,100000
  • 102,Ravi,
  • 103,Kiran,60000

Pandas may interpret the missing salary as NaN.

Check:

print(df.isna().sum())
  • Fill missing salary:
  • df["Salary"] = df["Salary"].fillna(0)
  • Or drop the rows:
df = df.dropna(
    subset=["Salary"]
)

1.20.20Handling Different Missing Value Representations

  • Some files use:
  • NULL
  • NA
  • N/A

-

unknown

You can specify them:

df = pd.read_csv(
    "employees.csv",
    na_values=[
        "NULL",
        "NA",
        "N/A",
        "-"
    ]
)

Pandas will treat those values as missing.

1.20.21Handling Encoding

Sometimes CSV files contain special characters.

You might see encoding errors such as:

UnicodeDecodeError

Try:

df = pd.read_csv(
    "employees.csv",
    encoding="utf-8"
)

For files produced by some Windows applications, you may encounter:

df = pd.read_csv(
    "employees.csv",
    encoding="cp1252"
)

The correct encoding depends on how the source file was created.

1.20.22Reading Excel Files

For .xlsx:

df = pd.read_excel(
    "employees.xlsx"
)

Then:

print(df.head())

1.20.23Creating an Excel File

You can write a DataFrame:

df.to_excel(

"employees.xlsx",

index=False

)

This creates:

employees.xlsx

1.20.24Excel Worksheets

  • Excel workbooks can contain multiple sheets.
  • For example:
  • employees.xlsx
  • ├── Employees
  • ├── Departments
  • └── Salaries

You can specify a sheet:

df = pd.read_excel(
    "employees.xlsx",
    sheet_name="Employees"
)

1.20.25Reading a Sheet by Number

df = pd.read_excel(
    "employees.xlsx",
    sheet_name=0
)

This reads the first sheet.

Second sheet:

df = pd.read_excel(
    "employees.xlsx",
    sheet_name=1
)
  • Remember:
  • 0 → first sheet
  • 1 → second sheet
  • 2 → third sheet

1.20.26Reading Multiple Excel Sheets

You can load multiple sheets:

sheets = pd.read_excel(
    "employees.xlsx",
    sheet_name=None
)

This returns a dictionary-like object.

Conceptually:

{

  • "Employees": DataFrame,
  • "Departments": DataFrame,
  • "Salaries": DataFrame

}

You can access:

employees = sheets["Employees"]

1.20.27Getting Excel Sheet Names

You can inspect the workbook:

excel_file = pd.ExcelFile(
    "employees.xlsx"
)
print(
    excel_file.sheet_names
)

Example:

['Employees', 'Departments', 'Salaries']

This is useful when you don't know the sheet names beforehand.

1.20.28Reading Specific Columns from Excel

df = pd.read_excel(
    "employees.xlsx",
    usecols=[
        "Name",
        "Salary"
    ]
)

Again, this can reduce unnecessary processing.

1.20.29Excel Header Control

If the Excel file has no header:

df = pd.read_excel(
    "employees.xlsx",
    header=None
)

Or specify column names:

df = pd.read_excel(
    "employees.xlsx",
    header=None,
    names=[
        "Employee_ID",
        "Name",
        "Salary"
    ]
)

1.20.30Excel Multiple Sheets Example

Let's create two DataFrames:

employees = pd.DataFrame({
    "Employee_ID": [101, 102, 103],
    "Name": [
        "Sreehari",
        "Ravi",
        "Kiran"
    ],
    "Department_ID": [10, 20, 10]
})
departments = pd.DataFrame({
    "Department_ID": [10, 20],
    "Department": [
        "IT",
        "HR"
    ]
})

Write them into one Excel workbook:

with pd.ExcelWriter(
    "company.xlsx",
    engine="openpyxl"
) as writer:
    employees.to_excel(
        writer,
        sheet_name="Employees",
        index=False
    )

departments.to_excel(

writer,

sheet_name="Departments",
index=False

)

Result:

company.xlsx

├── Employees
└── Departments

This is very useful for generating business Excel reports.

1.20.31Reading Those Sheets

employees = pd.read_excel(
    "company.xlsx",
    sheet_name="Employees"
)
departments = pd.read_excel(
    "company.xlsx",
    sheet_name="Departments"
)

Then merge them:

result = pd.merge(
    employees,
    departments,
    on="Department_ID",
    how="left"
)

Now you have:

  • Employee_ID
  • Name
  • Department_ID
  • Department

1.20.32CSV Data Cleaning Workflow

A very common workflow:

import pandas as pd
df = pd.read_csv(
    "employees.csv"
)

# Inspect

print(df.head())

df.info()

# Remove duplicate records

df = df.drop_duplicates()
  • # Remove leading/trailing spaces
  • df["Name"] = df["Name"].str.strip()
  • # Handle missing salary
  • df["Salary"] = df["Salary"].fillna(0)
  • # Convert salary
  • df["Salary"] = df["Salary"].astype(float)
  • # Export cleaned data
  • df.to_csv(
  • "employees_clean.csv",
index=False

)

This is a basic ETL pipeline.

1.20.33Excel Data Cleaning Workflow

import pandas as pd
df = pd.read_excel(
    "employees.xlsx"
)
print(df.head())
df = df.drop_duplicates()

df["Name"] = (

df["Name"]

.str.strip()

)

  • df["Salary"] = (
  • df["Salary"]
  • .fillna(0)

)

df.to_excel(

"employees_clean.xlsx",

index=False

)

1.20.34CSV → Data Transformation → Excel

This is a very common business requirement.

CSV
Read with Pandas
Clean
Transform
Generate Excel

Example:

import pandas as pd
df = pd.read_csv(
    "sales.csv"
)
df = df.drop_duplicates()

df["Amount"] = (

df["Amount"].fillna(0)

)

summary = (
    df.groupby("Product")["Amount"]
    .sum()
    .reset_index()
)

summary.to_excel(

"sales_summary.xlsx",

index=False

)

1.20.35Excel → CSV

You can convert Excel to CSV:

import pandas as pd
df = pd.read_excel(
    "input.xlsx"
)

df.to_csv(

"output.csv",

index=False

)

1.20.36CSV → Excel

And the reverse:

import pandas as pd
df = pd.read_csv(
    "input.csv"
)

df.to_excel(

"output.xlsx",

index=False

)

This is a simple file conversion utility.

1.20.37Filtering Before Export

Suppose we want only IT employees:

it_employees = df[
    df["Department"] == "IT"
]
  • Export:
  • it_employees.to_excel(
  • "it_employees.xlsx",
index=False

)

1.20.38Sorting Before Export

df = df.sort_values(
    "Salary",
    ascending=False
)

df.to_excel(

"employees_sorted.xlsx",

index=False

)

1.20.39Grouping Before Export

Suppose:

summary = (
    df.groupby("Department")
    ["Salary"]
    .agg([
        "count",
        "mean",
        "sum",
        "min",
        "max"
    ])
    .reset_index()
)
  • Export:
  • summary.to_excel(
  • "department_summary.xlsx",
index=False

)

This produces a useful management report.

1.20.40Working with Dates

CSV and Excel frequently contain dates.

Example:

  • Employee_ID,Name,Joining_Date
  • 101,Sreehari,2020-05-10
  • 102,Ravi,2021-07-15
  • Read:
df = pd.read_csv(
    "employees.csv"
)
  • Convert:
  • df["Joining_Date"] = pd.to_datetime(
  • df["Joining_Date"]

)

Now you can extract:

df["Joining_Year"] = (

df["Joining_Date"].dt.year

)

  • Month:
  • df["Joining_Month"] = (
  • df["Joining_Date"].dt.month

)

1.20.41Date Format During Reading

  • You may sometimes encounter:
  • 10/05/2020
  • Pandas can parse many common formats:
  • df["Joining_Date"] = pd.to_datetime(
  • df["Joining_Date"],
dayfirst=True

)

Always validate ambiguous date formats rather than assuming the intended interpretation.

1.20.42Handling Numeric Columns

Sometimes numbers are stored as strings.

Example:

  • Salary
  • ₹100,000
  • ₹70,000
  • ₹60,000

These aren't directly numeric.

  • You might clean them:
  • df["Salary"] = (
  • df["Salary"]
  • .str.replace("₹", "", regex=False)
  • .str.replace(",", "", regex=False)

)

  • Then:
  • df["Salary"] = pd.to_numeric(
  • df["Salary"],
errors="coerce"

)

Now Pandas can treat salary as numeric data.

1.20.43pd.to_numeric()

  • Useful when numeric values arrive as strings.
  • df["Amount"] = pd.to_numeric(
  • df["Amount"],
errors="coerce"

)

errors="coerce" converts invalid values to missing values rather than stopping the entire operation.

For example:

"1000" → 1000

"2000" → 2000

"ABC" → NaN

1.20.44Data Validation

Before loading processed data, check:

print(df.shape)
print(df.isna().sum())
print(df.duplicated().sum())
print(df.dtypes)

You might also check business rules:

invalid_salary = df[
    df["Salary"] < 0
]

Or:

invalid_age = df[
    (df["Age"] < 18) |
    (df["Age"] > 100)
]

This is a very important concept in ETL.

1.20.45CSV/Excel ETL Architecture

  • Imagine a company sends you:
  • daily_sales.xlsx
  • Your Python pipeline could be:
daily_sales.xlsx
Read Excel
Pandas DataFrame
Validate Data
Remove Duplicates
Handle NULLs
Convert Data Types
Apply Business Rules
Aggregate / Transform

┌──────┴──────┐

↓ ↓

CSV Output Database

This is much closer to real Data Engineering than simply reading a file.

1.20.46Practical ETL Example

  • Suppose you receive:
  • sales.csv
  • containing:
  • Order_ID,Customer,Product,Amount,Order_Date
  • 1001,Sreehari,Laptop,80000,2026-01-10
  • 1002,Ravi,Phone,30000,2026-01-11
  • 1003,Kiran,Laptop,75000,2026-01-12
  • 1004,Anil,Tablet,,2026-01-13
  • 1005,Sreehari,Phone,35000,2026-01-14
  • Load:
import pandas as pd
df = pd.read_csv(
    "sales.csv"
)

Step 1 — Inspect

print(df.head())
print(df.shape)

df.info()

Step 2 — Check missing values

print(
    df.isna().sum()
)

You might see:

Order_ID 0

Customer 0

Product 0

Amount 1

Order_Date 0

  • Step 3 — Fill missing amount
  • df["Amount"] = (
  • df["Amount"]
  • .fillna(0)

)

  • Step 4 — Convert date
  • df["Order_Date"] = pd.to_datetime(
  • df["Order_Date"]

)

Step 5 — Remove duplicates

df = df.drop_duplicates(
    subset=["Order_ID"]
)

Step 6 — Create month

df["Month"] = (

  • df["Order_Date"]
  • .dt.to_period("M")
  • .astype(str)

)

Step 7 — Calculate sales summary

monthly_sales = (
    df.groupby("Month")["Amount"]
    .sum()
    .reset_index()
)
  • Step 8 — Export
  • monthly_sales.to_excel(
  • "monthly_sales.xlsx",
index=False

)

Your pipeline is:

sales.csv
Read
Validate
Clean
Transform
Aggregate
monthly_sales.xlsx

This is a real-world mini ETL pipeline.

1.20.47CSV Quoting Problems

  • CSV can become complicated when values contain commas.
  • For example:
  • 101,"Sreehari, Mekala",IT,100000
  • Pandas normally handles quoted fields correctly:
df = pd.read_csv(
    "employees.csv"
)

This is one reason it's better to use a proper CSV parser rather than manually splitting lines with:

line.split(",")

1.20.48CSV Newline Issues

  • If you're writing CSV files for compatibility with certain Windows applications, you may occasionally encounter newline-related behavior.
  • Pandas handles normal CSV writing:
  • df.to_csv(
  • "output.csv",
index=False

)

For most cases, you don't need to manually manage newline characters.

1.20.49Excel File Engine

  • Pandas uses an Excel engine to work with Excel files.
  • For modern .xlsx files, openpyxl is commonly used.
  • Install:
pip install openpyxl

Then:

df = pd.read_excel(
    "employees.xlsx",
    engine="openpyxl"
)

Often Pandas can infer the appropriate engine, so specifying it isn't always necessary.

1.20.50Excel Formatting

Pandas is primarily a data processing tool, not a full Excel formatting library.

You can create worksheets:

df.to_excel(

"report.xlsx",

index=False

)

For advanced formatting, you can use libraries such as:

openpyxl

XlsxWriter

For example, advanced Excel requirements may include:

  • Bold headers
  • Column widths
  • Colors
  • Borders
  • Conditional formatting
  • Charts
  • Formulas
  • Multiple worksheets

Those are beyond basic Pandas and can be handled using Excel-specific libraries.

1.20.51Writing Multiple DataFrames to Excel

This is extremely useful for reporting.

with pd.ExcelWriter(
    "sales_report.xlsx",
    engine="openpyxl"
) as writer:
    df.to_excel(
        writer,
        sheet_name="Raw_Data",
        index=False
    )

monthly_sales.to_excel(

writer,

sheet_name="Monthly_Sales",
index=False

)

product_sales.to_excel(

writer,

sheet_name="Product_Sales",
index=False

)

  • The final workbook:
  • sales_report.xlsx
  • ├── Raw_Data
  • ├── Monthly_Sales
  • └── Product_Sales

This is a very common reporting pattern.

1.20.52Useful CSV Parameters

  • You should recognize these:
  • pd.read_csv(
  • "file.csv",
sep=",",
header=0,
usecols=None,
nrows=None,
skiprows=None,
encoding="utf-8",
na_values=None,
chunksize=None

)

Don't try to memorize every parameter. Understand what each one does.

1.20.53Useful Excel Parameters

  • Common ones:
  • pd.read_excel(
  • "file.xlsx",
sheet_name=0,
header=0,
usecols=None,
nrows=None

)

  • Most important:
  • sheet_name
  • header
  • usecols
  • nrows

1.20.54CSV vs Database

As a Data Engineer, you'll frequently encounter this situation:

CSV
Python
Pandas
Transform
SQL Database

For example:

import pandas as pd
df = pd.read_csv(
    "customers.csv"
)

Then after cleaning, the DataFrame could be loaded into a database using an appropriate database connector.

This concept becomes important when you combine:

  • Python
  • Pandas
  • SQL
  • ETL
  • APIs
  • Cloud platforms

1.20.55Important Commands to Remember

  • CSV
  • Read:
  • pd.read_csv()
  • Write:
  • df.to_csv()
  • Excel
  • Read:
  • pd.read_excel()
  • Write:
  • df.to_excel()
  • Multiple Excel sheets
  • pd.ExcelFile()
  • pd.ExcelWriter()
  • Inspection
  • df.head()
  • df.info()
  • df.describe()
  • df.shape
  • Cleaning
  • df.drop_duplicates()
  • df.dropna()
  • df.fillna()
  • Transformation
  • df.astype()
  • pd.to_numeric()
  • pd.to_datetime()

1.20.56Interview Questions

1. What is CSV?

A plain-text tabular file format where values are typically separated by commas.

2. How do you read a CSV using Pandas?

pd.read_csv("file.csv")

3. How do you write a CSV?

df.to_csv(

"file.csv",

index=False

)

4. How do you read Excel?

pd.read_excel(

"file.xlsx"

)

5. How do you write Excel?

df.to_excel(

"file.xlsx",

index=False

)

6. How do you read a specific Excel sheet?

pd.read_excel(

"file.xlsx",

sheet_name="Employees"

)

7. How do you read multiple Excel sheets?

pd.read_excel(

"file.xlsx",

sheet_name=None

)

8. Why use index=False?

To prevent the DataFrame index from being written as an additional column.

9. How do you process a large CSV?

  • Use:
  • pd.read_csv(
  • "large.csv",
chunksize=100000

)

10. How do you handle missing values?

  • df.fillna()
  • or:
  • df.dropna()

11. How do you convert a column to numeric?

pd.to_numeric(

df["Amount"],

errors="coerce"

)

12. How do you convert a column to datetime?

pd.to_datetime(

df["Order_Date"]

)

13. How do you combine multiple DataFrames into different Excel sheets?

Use:

pd.ExcelWriter()

14. What library is commonly used by Pandas for .xlsx files?

openpyxl is commonly used.

1.20.57Mini Project — CSV to Excel Sales ETL

This is the exercise I recommend you actually code.

Input

Create:

sales.csv

with:
    Order_ID,Customer,Product,Amount,Order_Date
  • 1001,Sreehari,Laptop,80000,2026-01-10
  • 1002,Ravi,Phone,30000,2026-01-11
  • 1003,Kiran,Laptop,75000,2026-01-12
  • 1004,Anil,Tablet,25000,2026-01-13
  • 1005,Sreehari,Phone,35000,2026-01-14
  • 1006,Ravi,Laptop,90000,2026-01-15
  • Requirements
  • Build a Python program that:

1. Read CSV

2. Display basic information

3. Check duplicates

4. Check missing values

5. Convert Order_Date

6. Create Month column

7. Calculate total sales

8. Calculate product-wise sales

9. Calculate customer-wise sales

10. Export everything to Excel

  • Your final Excel workbook should contain:
  • sales_report.xlsx
  • ├── Raw_Data
  • ├── Product_Sales
  • ├── Customer_Sales
  • └── Monthly_Sales

This single exercise will reinforce Pandas + CSV + Excel + GroupBy + DateTime + ETL, which are much more valuable than simply memorizing read_csv() and read_excel().

What you should remember from 1.20

The core pattern is:

import pandas as pd

# READ

df = pd.read_csv("input.csv")

# INSPECT

print(df.head())

df.info()

# CLEAN

df = df.drop_duplicates()
df = df.dropna()
  • # TRANSFORM
  • df["Order_Date"] = pd.to_datetime(
  • df["Order_Date"]

)

# ANALYZE

summary = (
    df.groupby("Product")["Amount"]
    .sum()
    .reset_index()
)
  • # WRITE
  • summary.to_excel(
  • "output.xlsx",
index=False

)

Think of Pandas + CSV/Excel as your first practical ETL toolkit:

DATA ENGINEERING

┌───────────┼───────────┐

↓ ↓ ↓

CSV Excel Database

│ │ │

└───────────┼───────────┘

Pandas
Clean / Transform
Analyze / Validate
CSV / Excel / Database

Next topic: 1.21 JSON Handling — JSON structure, objects, arrays, nested JSON, reading/writing JSON with Python, converting JSON to Pandas DataFrames, handling API responses, and flattening nested JSON.

Module 1 · Lesson 1.21

JSON Handling

JavaScript Object Notation

Imagine you build an Employee Management System.

You need to exchange employee data between:

  • Python backend
  • React frontend
  • Mobile application
  • Database
  • Cloud services
  • A common format is required.
Module 1 · Lesson 1.22

APIs in Python

API stands for:

  • Application Programming Interface
  • An API is a set of rules that allows two software applications to communicate.
  • Think of it as a messenger.
Application A
API
Application B
  • Python sends a request.
  • The API processes it.
  • Python receives a response.
Module 1 · Lesson 1.23

Logging

1.23 Logging in Python

Logging is the process of recording what an application is doing while it runs.

For small programs, you might use:

print("Pipeline started")

But in real-world applications, especially Data Engineering, ETL, APIs, and production systems, print() is not enough.

Instead, we use Python's built-in logging module.

import logging

1.23.1Why Do We Need Logging?

Imagine a data pipeline:

Read CSV
Validate Data
Transform Data
Load Database
Pipeline Completed

If something fails in production, you need to know:

  • When did it fail?
  • Which step failed?
  • What was the error?
  • Which file was being processed?
  • How many records were processed?
  • Was the failure a warning or a critical problem?

Logging gives you this information.

For example:

2026-08-22 10:15:01 INFO Pipeline started

2026-08-22 10:15:02 INFO Reading sales.csv

2026-08-22 10:15:03 INFO 10000 records loaded

2026-08-22 10:15:04 WARNING 15 records contain missing values

2026-08-22 10:15:05 ERROR Database connection failed

1.23.2Logging vs Print

print()
print("Pipeline started")
  • Good for:
  • Simple scripts
  • Temporary debugging
  • Learning
  • But it doesn't provide built-in:
  • Log levels
  • Timestamps
  • Log files
  • Structured configuration
  • Filtering
  • Production management
  • Logging
  • logging.info("Pipeline started")
  • Better for:
  • Production applications
  • ETL pipelines
  • APIs
  • Data Engineering
  • Machine Learning systems
  • Troubleshooting

1.23.3Importing Logging

Python already provides the logging module.

import logging

No installation is required.

1.23.4Your First Log Message

import logging
  • logging.warning("Something may be wrong")
  • You may see:
  • WARNING:root:Something may be wrong

By default, Python's logging system doesn't display every log level.

1.23.5Basic Log Levels

  • Python provides several standard levels:
  • DEBUG
  • INFO

WARNING

  • ERROR
  • CRITICAL
  • Think of them as increasing severity:
DEBUG
INFO
WARNING
ERROR
CRITICAL

1.23.6DEBUG

  • DEBUG provides detailed information useful during development.
  • logging.debug(
  • "Reading configuration file"

)

Example:

  • DEBUG: Reading configuration file
  • Typical use:
  • Variable values
  • Processing steps
  • Detailed execution information

1.23.7INFO

  • INFO tells you that something normal happened.
  • logging.info(
  • "Pipeline started"

)

  • Examples:
  • logging.info(
  • "Reading sales.csv"

)

logging.info(

"10000 records loaded"

)

logging.info(

"Pipeline completed successfully"

)

1.23.8WARNING

  • A warning indicates something unexpected but not necessarily fatal.
  • logging.warning(
  • "10 records contain missing values"

)

  • Examples:
  • Missing values
  • Deprecated functionality
  • Unexpected but recoverable conditions

1.23.9ERROR

  • An error means an operation failed.
  • logging.error(
  • "Database connection failed"

)

The application may still continue running depending on how the error is handled.

1.23.10CRITICAL

  • A critical error indicates a serious failure.
  • logging.critical(
  • "Application cannot start"

)

  • Examples:
  • Database unavailable
  • Configuration missing
  • System initialization failure

1.23.11Example of All Levels

import logging

logging.debug(

"Debug message"

)

logging.info(

"Information message"

)

logging.warning(

"Warning message"

)

logging.error(

"Error message"

)

logging.critical(

"Critical message"

)

By default, you'll generally see:

WARNING

ERROR

CRITICAL

because the default logging level is WARNING.

1.23.12Setting the Logging Level

You can configure logging:

import logging

logging.basicConfig(

level=logging.DEBUG

)

  • Now:
  • logging.debug(
  • "Debug message"

)

logging.info(

"Info message"

)

logging.warning(

"Warning message"

)

logging.error(

"Error message"

)

will all be displayed.

1.23.13Understanding Logging Levels

If you set:

level=logging.INFO

you get:

INFO

WARNING

  • ERROR
  • CRITICAL
  • but not:
  • DEBUG
  • If you set:
level=logging.ERROR
  • you get:
  • ERROR
  • CRITICAL
  • So:

DEBUG ← most detailed

INFO

WARNING

ERROR

CRITICAL ← most severe

1.23.14Adding Timestamps

A production log should usually contain a timestamp.

import logging

logging.basicConfig(

level=logging.INFO,
format="%(asctime)s - %(levelname)s - %(message)s"

)

logging.info(

"Pipeline started"

)

Example:

2026-08-22 22:15:30,123 - INFO - Pipeline started

1.23.15Understanding the Format

  • This:
  • "%(asctime)s - %(levelname)s - %(message)s"
  • contains:
  • asctime
  • Time when the log was generated.
  • levelname
  • Log level:
  • INFO

WARNING

  • ERROR
  • message
  • Your actual message.

1.23.16Including the Module Name

You can use:

logging.basicConfig(

level=logging.INFO,
format=(
    "%(asctime)s - "
    "%(name)s - "
    "%(levelname)s - "
    "%(message)s"
)

)

Example:

2026-08-22 22:20:00 - root - INFO - Pipeline started

1.23.17Writing Logs to a File

This is extremely important for production applications.

import logging

logging.basicConfig(

filename="application.log",
level=logging.INFO,
format=(
    "%(asctime)s - "
    "%(levelname)s - "
    "%(message)s"
)

)

logging.info(

"Application started"

)

Now the log is written to:

application.log

Instead of only appearing on the console.

1.23.18Example Log File

  • Your file might contain:
  • 2026-08-22 22:30:01,101 - INFO - Application started
  • 2026-08-22 22:30:02,201 - INFO - Reading input file
  • 2026-08-22 22:30:03,301 - INFO - Data loaded successfully
  • 2026-08-22 22:30:04,401 - WARNING - Missing values detected
  • 2026-08-22 22:30:05,501 - ERROR - Database connection failed
  • This becomes very useful when troubleshooting production jobs.

1.23.19Logging to Both Console and File

In real applications, you often want:

Console

+

Log File

For example:

import logging
logger = logging.getLogger(
    "my_app"
)

logger.setLevel(

logging.INFO

)

formatter = logging.Formatter(
    "%(asctime)s - %(levelname)s - %(message)s"
)
console_handler = logging.StreamHandler()
file_handler = logging.FileHandler(
    "application.log"
)

console_handler.setFormatter(

formatter

)

file_handler.setFormatter(

formatter

)

logger.addHandler(

console_handler

)

logger.addHandler(

file_handler

)

logger.info(

"Application started"

)

Now the message goes to both:

Terminal

application.log

1.23.20What is a Logger?

  • A logger is the object through which your application creates log messages.
  • Instead of:
  • logging.info(
  • "Pipeline started"

)

you can create:

logger = logging.getLogger(
    __name__
)
  • Then:
  • logger.info(
  • "Pipeline started"

)

This is the recommended approach for larger applications.

1.23.21__name__

You will frequently see:

logger = logging.getLogger(
    __name__
)
  • Suppose your file is:
  • pipeline.py
  • Then:

__name__

might represent:

pipeline

This helps identify which module generated a log.

1.23.22Recommended Basic Pattern

For most Python applications:

import logging

logging.basicConfig(

level=logging.INFO,
format=(
    "%(asctime)s - "
    "%(name)s - "
    "%(levelname)s - "
    "%(message)s"
)

)

logger = logging.getLogger(
    __name__
)

logger.info(

"Application started"

)

Then throughout your program:

  • logger.debug("Debug details")
  • logger.info("Process started")
  • logger.warning("Potential issue")
  • logger.error("Operation failed")
  • logger.critical("Critical failure")

1.23.23Logging Exceptions

This is one of the most important features.

Suppose:

try:
    result = 10 / 0
except Exception as e:
    logger.error(
        f"Error occurred: {e}"
    )
  • You might see:
  • ERROR - Error occurred: division by zero
  • But there's an even better approach.

1.23.24logger.exception()

Inside an exception handler:

try:
    result = 10 / 0
except Exception:
    logger.exception(
        "Failed to calculate result"
    )

This logs the error and traceback.

Example:

ERROR - Failed to calculate result

Traceback (most recent call last):

...

ZeroDivisionError: division by zero

This is extremely useful for debugging production failures.

1.23.25logger.error() vs logger.exception()

logger.error()

try:

...

except Exception as e:
    logger.error(
        f"Error: {e}"
    )

Logs the error message.

logger.exception()

try:

...

except Exception:
    logger.exception(
        "Operation failed"
    )

Logs:

Error message

+

Stack trace

For exception handling, logger.exception() is often the better choice.

1.23.26Logging Variables

You can include variables:

records = 15000

logger.info(

f"Processed {records} records"

)

Output:

INFO - Processed 15000 records

1.23.27Preferred Logging Formatting

  • Instead of:
  • logger.info(
  • f"Processed {records} records"

)

  • the logging module also supports:
  • logger.info(
  • "Processed %d records",
  • records

)

This is preferred in many logging-heavy applications because formatting can be deferred until needed.

1.23.28Logging a Data Pipeline

Imagine this pipeline:

CSV
Validate
Transform
Database
  • A good logging strategy might be:
  • logger.info(
  • "Pipeline started"

)

logger.info(

"Reading input file: sales.csv"

)

logger.info(

"Input file loaded successfully"

)

logger.info(

"Validating records"

)

logger.warning(

"15 records contain missing values"

)

logger.info(

"Transforming data"

)

logger.info(

"Loading data into database"

)

logger.info(

"Pipeline completed successfully"

)

1.23.29Practical ETL Logging Example

import logging
import pandas as pd

logging.basicConfig(

level=logging.INFO,
format=(
    "%(asctime)s - "
    "%(levelname)s - "
    "%(message)s"
),
filename="etl.log"

)

logger = logging.getLogger(
    __name__
)

logger.info(

"ETL pipeline started"

)

try:
    logger.info(
        "Reading sales.csv"
    )
df = pd.read_csv(
    "sales.csv"
)

logger.info(

"Loaded %d records",

len(df)

)

logger.info(

"Removing duplicate records"

)

df = df.drop_duplicates()

logger.info(

"Records after deduplication: %d",

len(df)

)

logger.info(

"Exporting cleaned data"

)

df.to_csv(

"sales_clean.csv",

index=False

)

logger.info(

"ETL pipeline completed successfully"

)

except Exception:
    logger.exception(
        "ETL pipeline failed"
    )

This is already much closer to production-quality Python.

1.23.30Logging API Calls

  • Logging is very useful when working with APIs.
  • logger.info(
  • "Calling customer API"

)

response = requests.get(
    url
)
  • logger.info(
  • "API returned status code %s",
  • response.status_code

)

If the API fails:

if response.status_code != 200:
    logger.error(
        "Customer API failed with status %s",
        response.status_code
    )

1.23.31Logging Database Operations

  • For Data Engineering:
  • logger.info(
  • "Connecting to database"

)

logger.info(

"Executing customer extraction query"

)

logger.info(

"Retrieved %d records",

len(df)

)

logger.info(

"Loading records into target table"

)

logger.info(

"Database load completed"

)

This makes pipeline monitoring much easier.

1.23.32Logging File Processing

Suppose you're processing many files:

files = [
    "sales_jan.csv",
    "sales_feb.csv",
    "sales_mar.csv"
]
for file in files:
    logger.info(
        "Processing file: %s",
        file
    )
try:
    df = pd.read_csv(file)

logger.info(

"Loaded %d records from %s",

len(df),
file

)

except Exception:
    logger.exception(
        "Failed to process %s",
        file
    )

This is very common in batch processing.

1.23.33Logger Hierarchy

Python logging supports hierarchical loggers.

For example:

application
├── application.database
  • ├── application.pipeline
  • ├── application.api
  • └── application.validation

You can create:

logger = logging.getLogger(
    "application.pipeline"
)

And another:

logger = logging.getLogger(
    "application.database"
)

This becomes useful in large applications.

1.23.34Handlers

A handler determines where logs go.

Common handlers:

StreamHandler
Console
FileHandler
File
RotatingFileHandler
Rotating log files
TimedRotatingFileHandler
Logs rotated based on time

1.23.35Formatter

A formatter controls the appearance of log messages.

Example:

formatter = logging.Formatter(
    "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
)

Output:

2026-08-22 22:45:00 - pipeline - INFO - Pipeline started

1.23.36Handler + Formatter + Logger

Think of logging like:

Application
Logger
Handler
Formatter
Console / File

For example:

logger
FileHandler
Formatter
application.log

1.23.37Rotating Log Files

  • A production application shouldn't necessarily allow:
  • application.log
  • to grow forever.
  • Use:
from logging.handlers import RotatingFileHandler

Example:

handler = RotatingFileHandler(
    "application.log",
    maxBytes=5_000_000,
    backupCount=5
)

This means the log file is rotated when it reaches approximately 5 MB.

You can keep several backup files.

  • Conceptually:
  • application.log
  • application.log.1
  • application.log.2
  • application.log.3

...

1.23.38Time-Based Log Rotation

You can also rotate based on time.

from logging.handlers import TimedRotatingFileHandler

Example:

handler = TimedRotatingFileHandler(
    "application.log",
    when="midnight",
    backupCount=7
)
  • This can maintain daily log files.
  • For example:
  • application.log
  • application.log.2026-08-20
  • application.log.2026-08-21
  • application.log.2026-08-22

1.23.39Logging Configuration

  • For larger applications, logging configuration is often separated from application code.
  • Instead of hardcoding everything, you can use configuration files or Python configuration dictionaries.
  • For example:
  • config
  • ├── application settings
  • └── logging settings

This becomes useful when deploying the same application to:

  • Development
  • Testing
  • Production

1.23.40Development vs Production Logging

  • You might want:
  • Development
  • DEBUG
  • INFO

WARNING

  • ERROR
  • Production
  • INFO

WARNING

ERROR

CRITICAL

You generally don't want huge volumes of DEBUG logs in production unless temporarily enabled for troubleshooting.

1.23.41What Should You Log?

  • Good things to log:
  • Application start/stop
  • Pipeline start/end
  • File names
  • Record counts
  • Database operations
  • API status
  • Processing duration
  • Warnings
  • Exceptions
  • Important business events

Example:

  • logger.info(
  • "Loaded %d records from %s",
  • record_count,
  • file_name

)

1.23.42What Should You NOT Log?

  • Be careful with sensitive information.
  • Don't log:
  • Passwords
  • API keys
  • Access tokens
  • Connection strings containing secrets
  • Credit card numbers
  • Personal secrets
  • For example, avoid:
  • logger.info(
  • "Password: %s",
  • password

)

  • Instead:
  • logger.info(
  • "User authentication successful"

)

Security is an important part of production logging.

1.23.43Logging Performance

Logging every single row can be a bad idea.

Avoid:

for row in df.itertuples():
    logger.info(
        "Processing row %s",
        row
    )
  • For millions of records, this could create enormous logs.
  • Prefer:
  • logger.info(
  • "Processing %d records",
len(df)

)

  • Or periodic progress:
  • Processed 100,000 records
  • Processed 200,000 records
  • Processed 300,000 records

1.23.44Logging Processing Time

You can measure execution time.

import time
start = time.time()

logger.info(

"Processing started"

)

# Processing

end = time.time()
  • logger.info(
  • "Processing completed in %.2f seconds",
  • end - start

)

Example:

INFO - Processing completed in 12.45 seconds

This is useful for ETL performance monitoring.

1.23.45Better Timing with time.perf_counter()

For measuring execution duration:

import time
start = time.perf_counter()

# Processing

duration = (
    time.perf_counter() - start
)
  • logger.info(
  • "Processing completed in %.2f seconds",
  • duration

)

1.23.46A Production-Style ETL Logger

Here's a useful pattern to understand:

import logging
import time
import pandas as pd

logging.basicConfig(

filename="etl.log",
level=logging.INFO,
format=(
    "%(asctime)s | "
    "%(levelname)s | "
    "%(name)s | "
    "%(message)s"
)

)

logger = logging.getLogger(
    __name__
)
def run_pipeline():
    start = time.perf_counter()

logger.info(

"ETL pipeline started"

)

try:
    logger.info(
        "Reading input file"
    )
df = pd.read_csv(
    "sales.csv"
)

logger.info(

"Loaded %d records",

len(df)

)

logger.info(

"Removing duplicates"

)

df = df.drop_duplicates()

logger.info(

"Remaining records: %d",

len(df)

)

logger.info(

"Writing output file"

)

df.to_csv(

"sales_clean.csv",

index=False

)

duration = (
    time.perf_counter() - start
)
  • logger.info(
  • "Pipeline completed in %.2f seconds",
  • duration

)

except Exception:
    logger.exception(
        "Pipeline failed"
    )

run_pipeline()

This gives you a strong foundation for production ETL logging.

1.23.47Logging Architecture for Data Engineering

As you progress toward Data Engineering, think of logging like this:

ETL PIPELINE

┌───────────┴───────────┐

↓ ↓

Pipeline Logger

│ │

│ ┌───────┴────────┐

│ ↓ ↓

│ Console File

┌─────┼─────┬─────────┐

↓ ↓ ↓ ↓

Extract Transform Load Validate

│ │ │ │

└─────┴─────┴─────────┘

  • Logs
  • Later, in cloud environments, those logs can be collected by monitoring platforms.
  • For example:
Python Application
Application Logs
Log Collection
Central Monitoring
Alerts / Dashboards

That concept will become very useful when you work with Azure, Databricks, ADF, Synapse, APIs, and MLOps.

1.23.48Complete Example — CSV ETL + Logging

Let's combine several topics you've learned.

import logging
import time
import pandas as pd

# -----------------------------------

# Logging Configuration

# -----------------------------------

logging.basicConfig(

filename="sales_etl.log",
level=logging.INFO,
format=(
    "%(asctime)s | "
    "%(levelname)s | "
    "%(message)s"
)

)

logger = logging.getLogger(
    __name__
)

# -----------------------------------

# Pipeline

# -----------------------------------

def process_sales():
    start = time.perf_counter()

logger.info(

"Sales ETL pipeline started"

)

try:
    # Extract

logger.info(

"Reading sales.csv"

)

df = pd.read_csv(
    "sales.csv"
)

logger.info(

"Loaded %d records",

len(df)

)

# Validate

missing = (
    df.isna()
    .sum()
    .sum()
)
if missing > 0:
    logger.warning(
        "Found %d missing values",
        missing
    )
  • # Transform
  • logger.info(
  • "Removing duplicate records"

)

before = len(df)
df = df.drop_duplicates()
after = len(df)
  • logger.info(
  • "Removed %d duplicates",
  • before - after

)

  • # Load
  • logger.info(
  • "Writing cleaned data"

)

df.to_csv(

"sales_clean.csv",

index=False

)

duration = (
    time.perf_counter() - start
)
  • logger.info(
  • "Pipeline completed successfully "
  • "in %.2f seconds",
  • duration

)

except Exception:
    logger.exception(
        "Sales ETL pipeline failed"
    )

raise

if __name__ == "__main__":
    process_sales()

This combines:

Python

+

Pandas

+

CSV

+

Exception Handling

+

Logging

+

ETL

That's exactly the direction you want to move toward as a Data Engineer.

1.23.49Practice Exercise 1 — Basic Logging

  • Create a program that logs:
  • Application started
  • Reading data
  • Data processing started
  • Data processing completed
  • Application finished
  • Use:
  • INFO

1.23.50Practice Exercise 2 — Error Logging

Write:

try:
    number = 10 / 0
except Exception:

...

Log the exception using:

logger.exception()

1.23.51Practice Exercise 3 — CSV Pipeline

Create:

employees.csv

Then build a pipeline:

Read CSV
Log record count
Check missing values
Log warning if missing values exist
Remove duplicates
Export cleaned CSV
Log completion

1.23.52Practice Exercise 4 — Processing Multiple Files

Create:

files = [
    "sales_jan.csv",
    "sales_feb.csv",
    "sales_mar.csv"
]
  • Process each file and log:
  • Processing sales_jan.csv
  • Loaded 1000 records
  • Completed sales_jan.csv
  • Processing sales_feb.csv
  • Loaded 1200 records
  • Completed sales_feb.csv
  • If a file fails:
  • ERROR - Failed to process sales_feb.csv
  • Use logger.exception().

1.23.53Practice Exercise 5 — Performance Logging

Create a program that:

Starts timer
Processes data
Stops timer
Logs execution time

Example:

INFO - Processing completed in 3.25 seconds

1.23.54Interview Questions

  • What is logging?
  • Recording application events and execution information for monitoring and troubleshooting.
  • Which module provides logging in Python?
  • logging
  • What are the standard log levels?
  • DEBUG
  • INFO

WARNING

  • ERROR
  • CRITICAL
  • What is the default logging level?
  • Typically:

WARNING

How do you create a logger?

logger = logging.getLogger(__name__)
  • How do you log information?
  • logger.info("Message")
  • How do you log an error?
  • logger.error("Something failed")
  • How do you log an exception with traceback?
  • logger.exception("Operation failed")
  • How do you write logs to a file?
  • logging.basicConfig(
filename="application.log"

)

  • What is a handler?
  • A handler determines where log messages are sent, such as a console or file.
  • What is a formatter?
  • A formatter determines the structure and appearance of a log message.
  • Why shouldn't you log passwords?
  • Because logs can expose sensitive credentials and create security risks.
  • Why is logging better than print()?
  • Logging provides levels, timestamps, destinations, filtering, exception tracebacks, and production-oriented configuration.

1.23.55Key Concepts to Remember

You don't need to memorize the entire logging API yet.

Master these first:

import logging

logging.basicConfig(...)

logger = logging.getLogger(__name__)
  • Then:
  • logger.debug(...)
  • logger.info(...)
  • logger.warning(...)
  • logger.error(...)
  • logger.critical(...)
  • And especially:
try:

...

except Exception:
    logger.exception("Operation failed")

For production applications:

Logger
Handler
Formatter
Console / File / Monitoring System

The bigger picture

You've now covered several important building blocks:

Python Basics
Variables & Data Types
Operators
Conditions
Loops
Functions
File Handling
Exception Handling
OOP
Iterators & Generators
Decorators
List Comprehensions
NumPy
Pandas
Matplotlib
CSV & Excel
Logging

You are moving from basic Python programming toward practical data-processing and production Python.

Next in your syllabus is 1.24 Best Practices, where the focus shifts from "How do I make Python code work?" to "How do I write Python code that is clean, maintainable, secure, testable, and production-ready?"

Module 1 · Lesson 1.24

Best Practices

1.24 Best Practices in Python

Python is easy to learn, but writing good Python code is different from simply writing code that works.

As you move toward Data Engineering, AI/ML, APIs, ETL, and production systems, you should develop good coding practices early.

A useful principle is:

Write code for humans first and computers second.

1.24.1What Are Coding Best Practices?

Coding best practices are guidelines that help make code:

  • Readable
  • Maintainable
  • Reusable
  • Testable
  • Secure
  • Efficient
  • Consistent
  • Easy to debug
  • Compare:
  • Poor code
x=100
y=200
z=x+y
print(z)

It works, but isn't very descriptive.

Better code

first_number = 100
second_number = 200
total = first_number + second_number
print(total)

The second version is easier to understand.

1.24.2Follow PEP 8

  • PEP 8 is Python's style guide.
  • It provides recommendations for writing consistent Python code.
  • For example:
def calculate_total(price, quantity):
    return price * quantity

Instead of:

def calculate_total(price,quantity): return price*quantity
  • PEP 8 covers things like:
  • Indentation
  • Naming
  • Spacing
  • Line length
  • Imports
  • Comments
  • Blank lines
  • Code organization

You don't need to memorize every PEP 8 rule, but you should follow its general principles.

1.24.3Use 4 Spaces for Indentation

Preferred:

if age >= 18:
    print("Adult")

Avoid mixing tabs and spaces.

Don't write:

if age >= 18:
    print("Adult")

Consistent indentation is fundamental to Python.

1.24.4Use Meaningful Variable Names

Bad:

x = 100
y = 20
z = x * y

Better:

price = 100
quantity = 20
total_cost = price * quantity

For Data Engineering:

Bad:

df1 = ...
df2 = ...
df3 = ...

Better:

customers_df = ...
orders_df = ...
sales_summary_df = ...

Names should communicate what the data represents.

1.24.5Use snake_case

Python commonly uses snake_case for variables and functions.

Good:

customer_name = "Sreehari"
total_amount = 5000
order_date = "2026-08-22"

Functions:

def calculate_total():
    pass
  • Avoid:
  • customerName
  • totalAmount
  • CalculateTotal()

For normal Python variables and functions, prefer:

snake_case

1.24.6Constants

Constants are usually written in uppercase.

MAX_RETRIES = 3
DATABASE_TIMEOUT = 30
DEFAULT_PAGE_SIZE = 100

This communicates:

"This value is intended to remain constant."

1.24.7Class Names

Classes generally use PascalCase.

class Customer:
    pass

Another example:

class SalesProcessor:
    pass

Not:

class sales_processor:
    pass

1.24.8Function Names Should Describe Actions

Good:

def calculate_salary():
    pass
def load_customer_data():
    pass
def validate_orders():
    pass

Poor:

def data():
    pass

A function name should tell you what the function does.

1.24.9Keep Functions Small

Avoid giant functions.

Poor:

def process_everything():
    # Read file
  • # Validate
  • # Clean
  • # Transform
  • # Connect DB
  • # Load DB
  • # Send email
  • # Generate report

# ...

pass

Better:

def read_data():
    pass
def validate_data():
    pass
def transform_data():
    pass
def load_data():
    pass
def generate_report():
    pass

Then:

def run_pipeline():
    data = read_data()

validate_data(data)

data = transform_data(data)

load_data(data)

generate_report(data)

This is much easier to maintain.

1.24.10Don't Repeat Yourself — DRY

  • DRY means:
  • Don't Repeat Yourself.
  • Suppose you repeatedly write:
total = price * quantity

in multiple places.

Instead, create a function:

def calculate_total(price, quantity):
    return price * quantity

Then:

total1 = calculate_total(100, 5)
total2 = calculate_total(200, 3)

Benefits:

Less duplicate code
Easier maintenance
Fewer bugs

1.24.11Avoid Copy-Paste Programming

Poor:

total1 = price1 * quantity1
total2 = price2 * quantity2
total3 = price3 * quantity3
total4 = price4 * quantity4

Better:

def calculate_total(price, quantity):
    return price * quantity

Then:

total1 = calculate_total(price1, quantity1)
total2 = calculate_total(price2, quantity2)
total3 = calculate_total(price3, quantity3)
total4 = calculate_total(price4, quantity4)

1.24.12Use Comments Wisely

  • Comments should explain why, not simply repeat what the code does.
  • Poor:
  • # Add two numbers
total = a + b

The code already tells us that.

Better:

# Apply the business discount before calculating tax.

discounted_price = price * 0.90

The comment explains the business reasoning.

1.24.13Use Docstrings

Functions should have docstrings when their behavior isn't obvious.

def calculate_total(price, quantity):
    """

Calculate the total cost based on price

and quantity.

"""

return price * quantity

For more complex functions:

def calculate_discount(price, discount_percent):
    """
  • Calculate the discounted price.
  • Parameters:
  • price: Original product price.
  • discount_percent: Discount percentage.
  • Returns:
  • Price after applying the discount.

"""

return price * (
    1 - discount_percent / 100
)

Docstrings are particularly useful in reusable libraries and production projects.

1.24.14Don't Use Magic Numbers

Avoid:

if age > 18:

...

when 18 has business meaning.

Better:

MINIMUM_ADULT_AGE = 18
if age >= MINIMUM_ADULT_AGE:

...

Another example:

Poor:

if salary > 100000:
    bonus = salary * 0.10

Better:

BONUS_THRESHOLD = 100000
BONUS_RATE = 0.10
if salary > BONUS_THRESHOLD:
    bonus = salary * BONUS_RATE

This makes business rules easier to change.

1.24.15Use with for Resources

When working with files, use:

with open("data.txt", "r") as file:
    content = file.read()

Instead of manually:

file = open("data.txt", "r")
content = file.read()

file.close()

The with statement ensures the resource is properly managed.

1.24.16Handle Exceptions Properly

Avoid:

try:

...

except:
    pass

This hides errors.

Bad:

try:
    result = 10 / 0
except:
    pass

You won't know that something went wrong.

Better:

try:
    result = 10 / 0
except ZeroDivisionError:
    print("Cannot divide by zero")

Catch the specific exception whenever practical.

1.24.17Don't Overuse try/except

Avoid wrapping your entire program:

try:
    # 500 lines of code

...

except Exception:
    print("Something went wrong")

This makes debugging difficult.

Prefer:

def load_data():
    try:

...

except FileNotFoundError:

...

Keep exception handling close to the operation where you can actually handle the problem.

1.24.18Use Logging Instead of print() in Production

You learned this in the previous topic.

Development:

print("Processing started")
  • Production:
  • logger.info(
  • "Processing started"

)

  • For errors:
  • logger.exception(
  • "Data processing failed"

)

  • Logging provides:
  • Timestamp
  • Level
  • Module
  • Message
  • Traceback

1.24.19Don't Hardcode Credentials

Never do this:

username = "admin"
password = "MyPassword123"

Or:

connection_string = (
    "Server=myserver;"
    "User=admin;"
    "Password=secret"
)

This is a serious security problem.

  • Instead, use:
  • Environment Variables
  • Secret Managers
  • Configuration Services
  • Managed Identity
  • For example:
import os
username = os.getenv(
    "DB_USERNAME"
)
password = os.getenv(
    "DB_PASSWORD"
)

1.24.20Use Environment Variables

  • Suppose:
  • DB_HOST
  • DB_NAME
  • DB_USERNAME
  • DB_PASSWORD

Then Python:

import os
db_host = os.getenv(
    "DB_HOST"
)
db_name = os.getenv(
    "DB_NAME"
)

This allows the same application to work in:

  • Development
  • Testing
  • Production
  • without changing the source code.

1.24.21Use .env Carefully

During local development, you may use:

.env

Example:

DB_HOST=localhost
DB_NAME=sales
DB_USERNAME=admin
DB_PASSWORD=secret

Python:

from dotenv import load_dotenv
import os

load_dotenv()

db_host = os.getenv(
    "DB_HOST"
)

Install:

pip install python-dotenv
  • Important: never commit .env containing secrets to Git.
  • Add it to:
  • .gitignore

1.24.22Use Git

For real projects, use version control.

Typical workflow:

Project
Git
GitHub / GitLab / Azure Repos
  • Useful commands:
  • git init
  • git status
  • git add .
  • git commit -m "Initial commit"
  • git push
  • Git allows you to:
  • Track changes
  • Restore previous versions
  • Collaborate
  • Review code
  • Create branches

1.24.23Use a Good Project Structure

  • Instead of putting everything into:
  • main.py
  • Use a structure such as:
  • sales_pipeline/

├── src/

│ ├── __init__.py

│ ├── extract.py

│ ├── transform.py

│ ├── load.py

│ └── validation.py

├── tests/

│ ├── test_extract.py

│ ├── test_transform.py

│ └── test_validation.py

├── config/

│ └── settings.py

├── data/

├── logs/

├── .env

  • ├── .gitignore
  • ├── requirements.txt
  • └── main.py

This becomes increasingly important as your projects grow.

1.24.24Separate Extract, Transform, and Load

For Data Engineering, this is particularly important.

Instead of:

def process():
    # read
  • # clean
  • # transform
  • # load
  • pass
  • Separate:
def extract():
    pass
def transform(data):
    pass
def load(data):
    pass

Then:

data = extract()
transformed_data = transform(
    data
)

load(

transformed_data

)

Architecture:

Extract
Transform
Load

This is the foundation of ETL.

1.24.25Validate Input Data

Never blindly trust input data.

For example:

if df.empty:
    raise ValueError(
        "Input data is empty"
    )

Check required columns:

required_columns = {
    "Customer_ID",
    "Amount",
    "Order_Date"
}
missing_columns = (
    required_columns
    - set(df.columns)
)
if missing_columns:
    raise ValueError(
        f"Missing columns: {missing_columns}"
    )

This is excellent practice for data pipelines.

1.24.26Validate Data Types

  • Suppose Amount should be numeric.
  • df["Amount"] = pd.to_numeric(
  • df["Amount"],
errors="coerce"

)

Then:

invalid_count = (
    df["Amount"].isna().sum()
)

You can log or reject invalid records.

1.24.27Avoid Unnecessary Loops with Pandas

Poor:

for index, row in df.iterrows():
    df.loc[index, "Total"] = (
        row["Price"] * row["Quantity"]
    )

Usually better:

df["Total"] = (

df["Price"]

* df["Quantity"]

)

This is called vectorization.

Pandas/NumPy are designed to operate on entire arrays or columns efficiently.

1.24.28Avoid iterrows() When Possible

Instead of:

for _, row in df.iterrows():

...

consider:

df["Total"] = (

df["Price"] *

df["Quantity"]

)

Or use:

df.apply(...)

when vectorization isn't practical.

But don't automatically replace every loop with apply()—vectorized operations are generally preferable when available.

1.24.29Choose the Right Data Structure

  • Python provides:
  • list
  • tuple
  • set
  • dictionary
  • Use them appropriately.
  • List
  • Ordered collection:
customers = [
    "Sreehari",
    "Ravi",
    "Kiran"
]

Tuple

Immutable collection:

coordinates = (
    17.4,
    78.5
)

Set

Unique values:

departments = {
    "IT",
    "HR",
    "Sales"
}

Dictionary

Key-value data:

employee = {
    "name": "Sreehari",
    "department": "IT",
    "salary": 100000
}

Choosing the appropriate data structure improves clarity and sometimes performance.

1.24.30Use List Comprehensions Carefully

You learned list comprehensions earlier.

Good:

squares = [
    x ** 2
    for x in range(10)
]

But avoid overly complicated comprehensions.

Poor:

result = [
    x * 2
    for x in data
    if x > 10
    if x % 2 == 0
    if x < 100
]

Sometimes a normal loop is easier to understand.

Readability is more important than cleverness.

1.24.31Don't Write Clever Code

Python allows compact code.

But:

result = [x for x in data if x > 10]
  • is good.
  • Whereas extremely complicated one-line expressions can be difficult to maintain.
  • A useful principle:
  • Simple and readable code beats clever code.

1.24.32Use Type Hints

Type hints make code easier to understand.

Without:

def calculate_total(price, quantity):
    return price * quantity

With:

def calculate_total(
    price: float,
    quantity: int
) -> float:
    return price * quantity

Now it is clear that:

  • price → float
  • quantity → integer
  • result → float

1.24.33Type Hints with Data Engineering

Example:

import pandas as pd
def clean_sales_data(
    df: pd.DataFrame
) -> pd.DataFrame:
    df = df.drop_duplicates()
return df

This communicates that the function expects and returns a Pandas DataFrame.

1.24.34Use Optional Carefully

For functions where a value may be absent:

from typing import Optional
def find_customer(
    customer_id: int
) -> Optional[str]:

...

This means the result may be:

  • str
  • or:
  • None
  • Modern Python versions also support:
def find_customer(
    customer_id: int
) -> str | None:

...

1.24.35Avoid Global Variables

Poor:

total_sales = 0
def calculate():
    global total_sales
  • total_sales += 100
  • Global state can make programs difficult to understand and test.
  • Better:
def calculate(total_sales):
    return total_sales + 100

Then:

total_sales = calculate(
    total_sales
)

1.24.36Keep Configuration Separate

Avoid:

def connect():
    host = "localhost"
port = 5432
database = "sales"

Better:

DB_HOST = "localhost"
DB_PORT = 5432
DB_NAME = "sales"

Or preferably retrieve them from configuration/environment variables.

1.24.37Use Virtual Environments

Each project should ideally have its own environment.

Create:

python -m venv .venv
  • Activate on Windows:
  • .venv\Scripts\activate
  • Then:
pip install pandas
pip install numpy
  • This prevents dependency conflicts.
  • For example:
  • Project A
  • └── pandas 2.x
  • Project B
  • └── pandas 3.x

Each project can have its own environment.

1.24.38Use requirements.txt

Create:

pip freeze > requirements.txt

Then another developer can install:

pip install -r requirements.txt

Example:

pandas==...
numpy==...
matplotlib==...
openpyxl==...

This makes environments reproducible.

1.24.39Use __name__ == "__main__"

Instead of:

run_pipeline()

at the bottom of every module, use:

if __name__ == "__main__":
    run_pipeline()

Example:

def run_pipeline():
    print("Pipeline started")
if __name__ == "__main__":
    run_pipeline()

This means the pipeline runs when the file is executed directly, but not automatically when imported as a module.

1.24.40Separate Business Logic from Execution

Poor:

if __name__ == "__main__":
    # 300 lines of code

Better:

def run_pipeline():

...

if __name__ == "__main__":
    run_pipeline()

This makes your code easier to test and reuse.

1.24.41Write Unit Tests

Testing is a critical best practice.

Example function:

def calculate_total(
    price,
    quantity
):
    return price * quantity

Test:

def test_calculate_total():
    result = calculate_total(
        100,
        5
    )

assert result == 500

Using pytest:

pip install pytest

Run:

pytest

1.24.42Test Edge Cases

Don't only test normal cases.

For:

def calculate_total(price, quantity):
    return price * quantity
  • Test:
  • Normal quantity
  • Zero quantity
  • Negative quantity
  • Large quantity
  • Invalid input
  • None
  • Good software anticipates unusual input.

1.24.43Validate Before Processing

For example:

def calculate_total(price, quantity):
    if price < 0:
        raise ValueError(
            "Price cannot be negative"
        )
if quantity < 0:
    raise ValueError(
        "Quantity cannot be negative"
    )
return price * quantity

This is safer than silently producing incorrect results.

1.24.44Use Assertions Carefully

  • Assertions are useful for developer assumptions.
  • assert len(df) > 0
  • Or:
  • assert "Customer_ID" in df.columns

However, assertions should not be your only mechanism for validating untrusted production input, because Python can run with assertions disabled.

For production validation, explicit checks and exceptions are preferable.

1.24.45Avoid Bare except

Avoid:

try:

...

except:

...

Prefer:

try:

...

except ValueError:

...

Or, when you genuinely need a catch-all:

try:

...

except Exception:
    logger.exception(
        "Unexpected error"
    )

1.24.46Don't Ignore Exceptions

Bad:

try:
    process_data()
except Exception:
    pass

The program silently hides the problem.

Better:

try:
    process_data()
except Exception:
    logger.exception(
        "Data processing failed"
    )

raise

The raise allows the failure to propagate after logging it.

1.24.47Avoid Deep Nesting

Poor:

if customer:
    if customer.active:
        if customer.balance > 0:
            if customer.region == "IN":
                process(customer)

Better:

if not customer:
    return
if not customer.active:
    return
if customer.balance <= 0:
    return
if customer.region != "IN":
    return

process(customer)

This is often called using guard clauses.

1.24.48Keep Code Modular

A good application might look like:

sales_pipeline/
├── extract.py
  • ├── transform.py
  • ├── validate.py
  • ├── load.py
  • ├── config.py
  • ├── logger.py
  • └── main.py
  • Each module has a clear responsibility.

1.24.49Single Responsibility Principle

A function/class should ideally have one primary responsibility.

Poor:

class Employee:
    # Reads CSV
  • # Cleans data
  • # Connects database
  • # Sends email
  • # Generates Excel

# ...

  • Better:
  • CSVReader
  • DataValidator
  • DataTransformer
  • DatabaseLoader
  • ReportGenerator
  • EmailService
  • Each component has a clear purpose.

1.24.50Use Meaningful Error Messages

  • Poor:
  • raise ValueError(
  • "Invalid data"

)

  • Better:
  • raise ValueError(
  • "Amount must be greater than or equal to zero"

)

Even better:

raise ValueError(

f"Invalid amount for Order_ID {order_id}: {amount}"

)

But make sure error messages don't expose sensitive information.

1.24.51Use Logging for Production Diagnostics

A strong production pattern is:

try:
    result = process_data()
except Exception:
    logger.exception(
        "Failed to process customer data"
    )
  • raise
  • This gives you:
  • Error message

+

Timestamp

+

Log level

+

Traceback

1.24.52Don't Commit Secrets to Git

  • Never commit:
  • .env
  • passwords
  • API keys
  • private keys
  • database credentials
  • access tokens
  • Use .gitignore:
  • .env
  • .venv/
  • __pycache__/
  • *.pyc
  • logs/

1.24.53Use Linters

  • A linter analyzes your code and identifies potential problems.
  • Popular Python tools include:
  • Ruff
  • Pylint
  • Flake8
  • For example, Ruff can identify:
  • Unused imports
  • Formatting problems
  • Potential bugs
  • Style issues

This is especially useful in team environments.

1.24.54Use Code Formatting Tools

  • Instead of manually formatting everything, use a formatter.
  • A popular choice is:
  • Black
  • Another modern option is:
  • Ruff formatter

Example:

black .

or use your IDE/editor formatting support.

1.24.55IDE Best Practices

  • Whether you're using:
  • VS Code
  • PyCharm
  • Visual Studio
  • Jupyter
  • enable:
  • Syntax highlighting
  • Linting
  • Formatting
  • Type checking
  • Code completion
  • Git integration
  • These tools catch problems early.

1.24.56Avoid Unused Imports

Poor:

import pandas as pd
import numpy as np
import os
import json
import math
if only Pandas is used.

Better:

import pandas as pd

Unused imports make code noisy and can sometimes increase startup time or introduce unnecessary dependencies.

1.24.57Organize Imports

Generally:

# Standard library

import os
import logging
from pathlib import Path

# Third-party

import pandas as pd
import numpy as np

# Local

from src.transform import clean_data

This makes dependencies easy to understand.

1.24.58Use pathlib for File Paths

Instead of:

file_path = (
    "C:\\Data\\Sales\\sales.csv"
)

you can use:

from pathlib import Path
file_path = (
    Path("Data")
    / "Sales"
    / "sales.csv"
)

Then:

df = pd.read_csv(
    file_path
)

pathlib is generally cleaner and more portable.

1.24.59Avoid Hardcoded File Paths

  • Poor:
  • pd.read_csv(
  • r"C:\Users\Sreehari\Desktop\sales.csv"

)

Better:

from pathlib import Path
DATA_DIR = Path("data")
file_path = (
    DATA_DIR / "sales.csv"
)

This makes the project portable.

1.24.60Use Configuration

Instead of scattering values throughout the application:

MAX_RETRIES = 3
INPUT_FILE = "sales.csv"
OUTPUT_FILE = "sales_clean.csv"

centralize configuration.

Example:

from pathlib import Path
DATA_DIR = Path("data")
INPUT_FILE = (
    DATA_DIR / "sales.csv"
)
OUTPUT_FILE = (
    DATA_DIR / "sales_clean.csv"
)
MAX_RETRIES = 3

1.24.61Use Efficient Data Types

  • For large Pandas datasets, data types matter.
  • Check:
  • df.info()
  • For example:
  • object
  • int64
  • float64
  • datetime64

You can sometimes optimize:

  • df["Department"] = (
  • df["Department"]
  • .astype("category")

)

This can reduce memory usage when a column contains many repeated values.

1.24.62Don't Load More Data Than Necessary

Instead of:

df = pd.read_csv(
    "huge_file.csv"
)

when you only need a few columns:

df = pd.read_csv(
    "huge_file.csv",
    usecols=[
        "Customer_ID",
        "Amount",
        "Order_Date"
    ]
)

This can significantly reduce memory consumption.

1.24.63Don't Load Huge Files Blindly

For large files:

for chunk in pd.read_csv(
    "huge_file.csv",
    chunksize=100000
):
    process_chunk(chunk)

This is a much more scalable approach.

1.24.64Security Best Practices

  • Always consider:
  • Input validation
  • Secret management
  • Authentication
  • Authorization
  • Encryption
  • Dependency updates
  • Logging security
  • File permissions

For example, never trust uploaded filenames blindly.

Validate:

allowed_extensions = {
    ".csv",
    ".xlsx"
}

1.24.65Dependency Management

Don't install everything globally:

pip install pandas
pip install numpy
pip install ...

Instead:

Project
Virtual Environment
Dependencies

Then record:

pip freeze > requirements.txt

For larger modern projects, tools such as:

  • uv
  • Poetry
  • pip-tools
  • can also help manage dependencies.

1.24.66Don't Optimize Too Early

A common mistake is trying to make code extremely fast before knowing whether performance is actually a problem.

First:

Correct
Readable
Tested
Measured
Optimized

Use profiling to find actual bottlenecks.

Don't assume something is slow without measuring it.

1.24.67Measure Before Optimizing

For simple timing:

import time
start = time.perf_counter()

process_data()

duration = (
    time.perf_counter() - start
)
print(
    f"Duration: {duration:.2f}s"
)

Then optimize the actual slow part.

1.24.68Documentation

  • A good project should explain:
  • What the project does
  • How to install it
  • How to configure it
  • How to run it
  • How to test it
  • How to deploy it
  • A typical:
  • README.md
  • might contain:
  • # Sales ETL Pipeline
  • ## Installation
  • ## Configuration
  • ## Usage
  • ## Input Data
  • ## Output Data
  • ## Testing
  • ## Troubleshooting

1.24.69Keep README Updated

Example:

  • # Sales ETL Pipeline
  • ## Installation
  • ```bash
pip install -r requirements.txt

Run

python main.py

Input

Place sales.csv in the data/ directory.

Output

Cleaned data is written to output/.

Documentation saves significant time for future developers.

---

# 1.24.70 A Good Production Project

A realistic Python ETL project could look like:

```text

sales_etl/

├── src/

│ ├── __init__.py

│ ├── extract.py

│ ├── transform.py

│ ├── validate.py

│ ├── load.py

│ └── logger.py

├── tests/

│ ├── test_extract.py

│ ├── test_transform.py

│ └── test_validate.py

├── data/

│ ├── input/

│ └── output/

├── logs/

├── config/

├── .env

  • ├── .gitignore
  • ├── requirements.txt
  • ├── README.md
  • └── main.py

This is a much better structure than:

project/

└── everything.py

1.24.71Putting Everything Together

Consider this simple ETL program.

import logging
from pathlib import Path
import pandas as pd
DATA_DIR = Path("data")
INPUT_FILE = (
    DATA_DIR / "sales.csv"
)
OUTPUT_FILE = (
    DATA_DIR / "sales_clean.csv"
)

logging.basicConfig(

level=logging.INFO,
format=(
    "%(asctime)s | "
    "%(levelname)s | "
    "%(message)s"
)

)

logger = logging.getLogger(
    __name__
)
def extract(
    file_path: Path
) -> pd.DataFrame:
    """Read sales data from CSV."""
  • logger.info(
  • "Reading %s",
  • file_path

)

df = pd.read_csv(
    file_path
)

logger.info(

"Loaded %d records",

len(df)

)

return df
def validate(
    df: pd.DataFrame
) -> None:
    """Validate required columns."""
required_columns = {
    "Order_ID",
    "Amount"
}
missing_columns = (
    required_columns
    - set(df.columns)
)
if missing_columns:
    raise ValueError(
        f"Missing columns: "
        f"{missing_columns}"
    )
def transform(
    df: pd.DataFrame
) -> pd.DataFrame:
    """Clean and transform sales data."""
df = df.drop_duplicates()

df["Amount"] = pd.to_numeric(

df["Amount"],

errors="coerce"

)

df["Amount"] = (

df["Amount"].fillna(0)

)

return df
def load(
    df: pd.DataFrame,
    file_path: Path
) -> None:
    """Write cleaned data to CSV."""

df.to_csv(

file_path,

index=False

)

  • logger.info(
  • "Output written to %s",
  • file_path

)

def run_pipeline():
    """Run the complete ETL pipeline."""

logger.info(

"Pipeline started"

)

df = extract(
    INPUT_FILE
)

validate(

df

)

df = transform(
    df
)
  • load(
  • df,
  • OUTPUT_FILE

)

logger.info(

"Pipeline completed"

)

if __name__ == "__main__":
    run_pipeline()

Notice how many best practices are being applied:

  • ✓ Meaningful names
  • ✓ Functions
  • ✓ Type hints
  • ✓ Docstrings
  • ✓ Logging
  • ✓ pathlib
  • ✓ Constants
  • ✓ Input validation
  • ✓ Exception-ready architecture
  • ✓ Separate ETL stages
  • ✓ No hardcoded credentials
  • ✓ __main__ guard
  • ✓ Pandas vectorization

1.24.72The Golden Rules

If you're starting your career as a Data Engineer, remember these rules:

  • Rule 1 — Make it readable
  • Readable > Clever
  • Rule 2 — Don't repeat code
  • DRY
  • Rule 3 — Keep functions focused
  • One function → One main responsibility
  • Rule 4 — Validate input
  • Never blindly trust data
  • Rule 5 — Handle errors properly
  • Don't silently ignore exceptions
  • Rule 6 — Log production events
  • Logging > print()
  • Rule 7 — Never expose secrets
  • Secrets → Environment / Secret Manager
  • Rule 8 — Use version control
  • Git
  • Rule 9 — Test your code
  • Unit Tests
  • Integration Tests
  • Rule 10 — Keep dependencies isolated
  • Virtual Environment
  • Rule 11 — Document your project
  • README.md
  • Docstrings
  • Rule 12 — Measure before optimizing
  • Correct → Measure → Optimize

1.24.73Best Practices Cheat Sheet

AreaBest Practice
StyleFollow PEP 8
VariablesMeaningful snake_case
ConstantsUPPER_CASE
ClassesPascalCase
FunctionsSmall and focused
Code reuseDRY
CommentsExplain why
DocumentationUse docstrings
ErrorsSpecific exceptions
LoggingUse logging
SecretsEnvironment/secret manager
DependenciesVirtual environment
Packagesrequirements.txt
Version controlGit
Testingpytest/unit tests
Pathspathlib
DataFramesVectorized operations
Large filesChunk processing
ValidationCheck input data
PerformanceMeasure before optimizing
DocumentationREADME
StructureModular project

1.24.74What Matters Most for You

Because your course is moving toward Data Science, Machine Learning, AI, and Data Engineering, don't treat best practices as just Python syntax.

You should start thinking like this:

Python Application

┌─────────────┼─────────────┐

↓ ↓ ↓

Config Logging Errors

│ │ │

└─────────────┼─────────────┘

Extract
Validate
Transform
Test
Load
Monitor

And your code should be:

Readable

+

Reusable

+

Testable

+

Secure

+

Observable

+

  • Maintainable
  • That's the difference between Python code that works and production-quality Python code.
  • Practice Project — Production-Style CSV ETL
  • For this lesson, I recommend building one small project instead of doing many disconnected exercises.
  • Build:
  • Sales ETL Pipeline
  • Requirements:
sales.csv
Python
Extract
Validate
Clean
Transform
Log everything
Handle exceptions
Generate sales_clean.csv
Generate sales_report.xlsx
  • Include:
  • main.py
  • extract.py
  • transform.py
  • validate.py
  • load.py
  • logger.py
  • tests/
  • requirements.txt
  • .gitignore
  • README.md

If you can build this cleanly, you will have combined almost everything you've learned in Module 1 so far.

Your Module 1 progression

1.1 Python Basics

1.2 Variables & Data Types

1.3 Operators

1.4 Conditional Statements

1.5 Loops

1.6 Functions

1.7 Lambda Functions

1.8 Recursion

1.9 Modules & Packages

1.10 File Handling

1.11 Exception Handling

1.12 OOP

1.13 Iterators & Generators

1.14 Decorators

1.15 List Comprehensions

1.16 Virtual Environments

1.17 NumPy Basics

1.18 Pandas Basics

1.19 Matplotlib Basics

1.20 CSV & Excel

1.21 JSON Handling

1.22 APIs in Python

1.23 Logging

Module 1 · Lesson 1.25

Mini Project

You are now one topic away from completing Module 1: 1.25 Mini Project. The mini project should bring together Python + Pandas + CSV/Excel + JSON/API concepts + logging + exception handling + OOP + best practices into one practical application.

1.25 Mini Project — Sales Data ETL & Reporting System

For the final topic of Module 1 – Python Programming, let's build a practical project that combines the major concepts you've learned.

The project will simulate a real-world Data Engineering ETL pipeline.

1.25.1Project Overview

  • Project Name
  • Sales Data ETL & Reporting System
  • Objective
  • Build a Python application that:
  • Reads sales data from CSV
  • Validates the input
  • Cleans the data
  • Removes duplicates
  • Handles missing values
  • Calculates sales metrics
  • Generates summary reports
  • Saves results to Excel
  • Produces JSON output
  • Logs the entire process
  • Handles errors gracefully
  • Overall:
  • SALES.CSV

┌───────────┐

│ EXTRACT │

└─────┬─────┘

┌───────────┐

│ VALIDATE │

└─────┬─────┘

┌───────────┐

│ CLEAN │

└─────┬─────┘

┌───────────┐

│ TRANSFORM │

└─────┬─────┘

┌─────────┼─────────┐

▼ ▼ ▼

CSV Excel JSON

│ │ │

└─────────┼─────────┘

LOG FILE

1.25.2What You Will Practice

This project combines:

TopicUsage
Python BasicsProgram structure
VariablesStore configuration
ConditionsValidation
LoopsProcessing
FunctionsModular design
ModulesSeparate files
File HandlingRead/write files
Exception HandlingError management
OOPPipeline class
List ComprehensionsData manipulation
Virtual EnvironmentDependency isolation
NumPyNumerical calculations
PandasData processing
MatplotlibVisualization
CSVInput
ExcelReporting
JSONAPI/report output
LoggingMonitoring
Best PracticesProduction-style code

1.25.3Sample Input Data

  • Create:
  • data/sales.csv
  • Use:
  • Order_ID,Order_Date,Customer,Product,Category,Quantity,Unit_Price,Region
  • 1001,2026-01-05,Ravi,Laptop,Electronics,2,75000,South
  • 1002,2026-01-07,Suresh,Mouse,Electronics,5,1200,South
  • 1003,2026-01-10,Anita,Keyboard,Electronics,3,2500,North
  • 1004,2026-01-12,John,Monitor,Electronics,2,18000,West
  • 1005,2026-01-15,Ravi,Chair,Furniture,4,8500,South
  • 1006,2026-01-20,Meena,Desk,Furniture,2,15000,East
  • 1007,2026-02-01,Suresh,Laptop,Electronics,1,75000,South
  • 1008,2026-02-05,Anita,Chair,Furniture,3,8500,North
  • 1009,2026-02-10,John,Monitor,Electronics,1,18000,West
  • 1010,2026-02-15,Ravi,Desk,Furniture,2,15000,South

1.25.4Project Structure

Create this structure:

sales_etl/

├── data/

│ ├── sales.csv

│ └── output/

├── logs/

├── reports/

├── src/

│ ├── __init__.py

│ ├── config.py

│ ├── logger.py

│ ├── extract.py

│ ├── validate.py

│ ├── transform.py

│ ├── report.py

│ └── pipeline.py

├── tests/

│ └── test_transform.py

  • ├── main.py
  • ├── requirements.txt
  • ├── .gitignore
  • └── README.md

This structure itself demonstrates several best practices.

1.25.5Step 1 — Create Virtual Environment

Open your terminal:

python -m venv .venv
  • Activate it on Windows:
  • .venv\Scripts\activate
  • You should see something similar to:
  • (.venv)
  • in your terminal.

1.25.6Step 2 — Install Libraries

Install:

pip install pandas numpy matplotlib openpyxl

Then generate:

pip freeze > requirements.txt

Your project now has its own isolated environment.

1.25.7Step 3 — Configuration

  • Create:
  • src/config.py
  • Code:
from pathlib import Path
BASE_DIR = Path(__file__).resolve().parent.parent
DATA_DIR = BASE_DIR / "data"
OUTPUT_DIR = DATA_DIR / "output"
LOG_DIR = BASE_DIR / "logs"
REPORT_DIR = BASE_DIR / "reports"
INPUT_FILE = DATA_DIR / "sales.csv"
CLEANED_FILE = (
    OUTPUT_DIR / "sales_clean.csv"
)
EXCEL_REPORT = (
    REPORT_DIR / "sales_report.xlsx"
)
JSON_REPORT = (
    REPORT_DIR / "sales_summary.json"
)
  • Notice we're using:
  • Path
  • instead of hardcoded Windows paths.

1.25.8Step 4 — Logging

Create:

src/logger.py

import logging
from src.config import LOG_DIR

LOG_DIR.mkdir(

parents=True,
exist_ok=True

)

LOG_FILE = LOG_DIR / "sales_etl.log"

logging.basicConfig(

filename=LOG_FILE,
level=logging.INFO,
format=(
    "%(asctime)s | "
    "%(levelname)s | "
    "%(name)s | "
    "%(message)s"
)

)

logger = logging.getLogger(
    "sales_etl"
)

Now every module can use:

from src.logger import logger

1.25.9Step 5 — Extract Data

Create:

src/extract.py

import pandas as pd
from src.logger import logger
def extract_sales_data(
    file_path
) -> pd.DataFrame:
    """

Read sales data from CSV.

"""

  • logger.info(
  • "Reading input file: %s",
  • file_path

)

try:
    df = pd.read_csv(
        file_path
    )

logger.info(

"Loaded %d records",

len(df)

)

return df
except FileNotFoundError:
    logger.exception(
        "Input file not found: %s",
        file_path
    )

raise

except Exception:
    logger.exception(
        "Failed to read input file"
    )
  • raise
  • We are using:
  • Functions
  • Pandas
  • Logging
  • Exception handling
  • Docstrings

1.25.10Step 6 — Validate Data

Create:

src/validate.py

from src.logger import logger
REQUIRED_COLUMNS = {
    "Order_ID",
    "Order_Date",
    "Customer",
    "Product",
    "Category",
    "Quantity",
    "Unit_Price",
    "Region"
}
def validate_columns(df):
    """

Validate required columns.

"""

logger.info(

"Validating input columns"

)

missing_columns = (
    REQUIRED_COLUMNS
    - set(df.columns)
)
if missing_columns:
    logger.error(
        "Missing columns: %s",
        missing_columns
    )
  • raise ValueError(
  • f"Missing columns: "
  • f"{missing_columns}"

)

logger.info(

"Column validation successful"

)

1.25.11Validate Data Values

Add another function:

def validate_values(df):
    """

Validate important data values.

"""

logger.info(

"Validating data values"

)

if (df["Quantity"] < 0).any():
    raise ValueError(
        "Quantity cannot be negative"
    )
if (df["Unit_Price"] < 0).any():
    raise ValueError(
        "Unit price cannot be negative"
    )

logger.info(

"Value validation successful"

)

1.25.12Step 7 — Transform Data

Create:

src/transform.py

import pandas as pd
from src.logger import logger
def clean_data(
    df: pd.DataFrame
) -> pd.DataFrame:
    """

Clean and transform sales data.

"""

logger.info(

"Starting data transformation"

)

df = df.copy()
  • # Convert date
  • df["Order_Date"] = pd.to_datetime(
  • df["Order_Date"],
errors="coerce"

)

# Remove duplicates

before = len(df)
df = df.drop_duplicates()
removed = (
    before - len(df)
)
  • logger.info(
  • "Removed %d duplicate records",
  • removed

)

  • # Convert numeric columns
  • df["Quantity"] = pd.to_numeric(
  • df["Quantity"],
errors="coerce"

)

df["Unit_Price"] = pd.to_numeric(

df["Unit_Price"],

errors="coerce"

)

  • # Calculate sales amount
  • df["Sales_Amount"] = (
  • df["Quantity"]
  • * df["Unit_Price"]

)

# Handle missing values

missing_values = (
    df.isna().sum().sum()
)
if missing_values > 0:
    logger.warning(
        "Found %d missing values",
        missing_values
    )
df = df.dropna()

logger.info(

"Transformation completed"

)

return df

1.25.13What Happened Here?

  • Our original data had:
  • Quantity
  • Unit_Price
  • We created:
  • Sales_Amount
  • using:
  • df["Sales_Amount"] = (
  • df["Quantity"]
  • * df["Unit_Price"]

)

For example:

Laptop

Quantity = 2
  • Unit Price = 75,000
  • Sales Amount =
  • 2 × 75,000
  • = ₹150,000

1.25.14Step 8 — Create Reports

Create:

src/report.py

import json
import pandas as pd
from src.logger import logger
def create_summary(
    df: pd.DataFrame
) -> dict:
    """

Create sales summary metrics.

"""

total_sales = (
    df["Sales_Amount"].sum()
)
total_orders = (
    df["Order_ID"].nunique()
)
total_quantity = (
    df["Quantity"].sum()
)
average_order_value = (
    total_sales / total_orders
    if total_orders > 0
    else 0
)
summary = {
    "total_sales": float(
        total_sales
    ),
    "total_orders": int(
        total_orders
    ),
    "total_quantity": int(
        total_quantity
    ),
    "average_order_value": float(
        average_order_value
    )
}
return summary

1.25.15Regional Sales Report

Add:

def regional_sales(
    df: pd.DataFrame
) -> pd.DataFrame:
    """

Generate regional sales summary.

"""

return (
    df.groupby("Region")
    ["Sales_Amount"]
    .sum()
    .reset_index()
    .sort_values(
        "Sales_Amount",
        ascending=False
    )
)

Result:

Region Sales_Amount

South ...

North ...

West ...

East ...

1.25.16Product Sales Report

Add:

def product_sales(
    df: pd.DataFrame
) -> pd.DataFrame:
    """

Generate product sales summary.

"""

return (
    df.groupby("Product")
    ["Sales_Amount"]
    .sum()
    .reset_index()
    .sort_values(
        "Sales_Amount",
        ascending=False
    )
)

1.25.17Generate Excel Report

Add:

def create_excel_report(
    df,
    regional_df,
    product_df,
    output_file
):
    logger.info(
        "Creating Excel report"
    )
with pd.ExcelWriter(
    output_file,
    engine="openpyxl"
) as writer:
    df.to_excel(
        writer,
        sheet_name="Sales Data",
        index=False
    )

regional_df.to_excel(

writer,

sheet_name="Regional Sales",
index=False

)

product_df.to_excel(

writer,

sheet_name="Product Sales",
index=False

)

  • logger.info(
  • "Excel report created: %s",
  • output_file

)

Your Excel file will contain:

sales_report.xlsx
├── Sales Data

├── Regional Sales

└── Product Sales

1.25.18Generate JSON Report

Add:

def create_json_report(
    summary,
    output_file
):
    logger.info(
        "Creating JSON report"
    )
with open(
    output_file,
    "w",
    encoding="utf-8"
) as file:
    json.dump(
        summary,
        file,
        indent=4
    )
  • logger.info(
  • "JSON report created: %s",
  • output_file

)

Example JSON:

{

  • "total_sales": 500000,
  • "total_orders": 10,
  • "total_quantity": 25,
  • "average_order_value": 50000

}

1.25.19Step 9 — Pipeline Class

Now we'll use OOP, which you've already learned.

Create:

src/pipeline.py

from src.config import (
    INPUT_FILE,
    OUTPUT_DIR,
    CLEANED_FILE,
    EXCEL_REPORT,
    JSON_REPORT
)
from src.extract import (
    extract_sales_data
)
from src.validate import (
    validate_columns,
    validate_values
)
from src.transform import (
    clean_data
)
from src.report import (
    create_summary,
    regional_sales,
    product_sales,
    create_excel_report,
    create_json_report
)
from src.logger import logger
class SalesETLPipeline:
    """

End-to-end sales ETL pipeline.

"""

def __init__(self):
    OUTPUT_DIR.mkdir(
        parents=True,
        exist_ok=True
    )

EXCEL_REPORT.parent.mkdir(

parents=True,
exist_ok=True

)

self.df = None

def run(self):
    logger.info(
        "Sales ETL pipeline started"
    )
try:
    # Extract
  • self.df = (
  • extract_sales_data(
  • INPUT_FILE

)

)

  • # Validate
  • validate_columns(
  • self.df

)

validate_values(

self.df

)

  • # Transform
  • self.df = clean_data(
  • self.df

)

  • # Save cleaned data
  • self.df.to_csv(
  • CLEANED_FILE,
index=False

)

logger.info(

"Cleaned data saved"

)

# Reports

summary = create_summary(
    self.df
)
regional_df = (
    regional_sales(
        self.df
    )
)
product_df = (
    product_sales(
        self.df
    )
)
  • create_excel_report(
  • self.df,
  • regional_df,
  • product_df,
  • EXCEL_REPORT

)

  • create_json_report(
  • summary,
  • JSON_REPORT

)

logger.info(

"Sales ETL pipeline completed successfully"

)

except Exception:
    logger.exception(
        "Sales ETL pipeline failed"
    )

raise

1.25.20Step 10 — Main Program

Create:

main.py

from src.pipeline import (
    SalesETLPipeline
)
def main():
    pipeline = (
        SalesETLPipeline()
    )

pipeline.run()

if __name__ == "__main__":
    main()

This is our application entry point.

1.25.21Run the Project

From the project root:

python main.py

You should see the program execute.

The output files should be created:

data/
├── sales.csv

└── output/

└── sales_clean.csv

reports/
├── sales_report.xlsx
  • └── sales_summary.json
  • logs/
  • └── sales_etl.log

1.25.22Example Log

  • Your log might look like:
  • 2026-08-22 23:10:01 | INFO | sales_etl | Sales ETL pipeline started
  • 2026-08-22 23:10:01 | INFO | sales_etl | Reading input file
  • 2026-08-22 23:10:01 | INFO | sales_etl | Loaded 10 records
  • 2026-08-22 23:10:01 | INFO | sales_etl | Validating input columns
  • 2026-08-22 23:10:01 | INFO | sales_etl | Column validation successful
  • 2026-08-22 23:10:01 | INFO | sales_etl | Starting data transformation
  • 2026-08-22 23:10:01 | INFO | sales_etl | Removed 0 duplicate records
  • 2026-08-22 23:10:01 | INFO | sales_etl | Transformation completed
  • 2026-08-22 23:10:02 | INFO | sales_etl | Excel report created
  • 2026-08-22 23:10:02 | INFO | sales_etl | JSON report created
  • 2026-08-22 23:10:02 | INFO | sales_etl | Sales ETL pipeline completed successfully

1.25.23Add Visualization

Now let's use Matplotlib.

Create:

src/visualization.py

import matplotlib.pyplot as plt
from src.logger import logger
def create_regional_chart(
    regional_df,
    output_file
):
    logger.info(
        "Creating regional sales chart"
    )

plt.figure(

figsize=(8, 5)

)

  • plt.bar(
  • regional_df["Region"],
  • regional_df["Sales_Amount"]

)

plt.title(

"Sales by Region"

)

plt.xlabel(

"Region"

)

plt.ylabel(

"Sales Amount"

)

  • plt.tight_layout()
  • plt.savefig(
  • output_file

)

  • plt.close()
  • logger.info(
  • "Regional sales chart created"

)

Then you can generate:

reports/

└── regional_sales.png

1.25.24Add Monthly Sales

This is a useful Data Analytics feature.

def monthly_sales(
    df
):
    monthly = (
        df.groupby(
            df["Order_Date"]
            .dt.to_period("M")
        )["Sales_Amount"]
        .sum()
        .reset_index()
    )
  • monthly["Order_Date"] = (
  • monthly["Order_Date"]
  • .astype(str)

)

return monthly

Output:

Month Sales

2026-01 250000

2026-02 300000

1.25.25Business KPIs

Our project can calculate:

Total Sales

total_sales = df["Sales_Amount"].sum()

Total Orders

total_orders = df["Order_ID"].nunique()

Total Quantity

total_quantity = df["Quantity"].sum()

Average Order Value

average_order_value = (
    total_sales / total_orders
)

Top Product

top_product = (
    df.groupby("Product")[
        "Sales_Amount"
    ]
    .sum()
    .idxmax()
)

1.25.26Final Business Report

The project should ultimately answer questions such as:

How much did we sell?

Which region performed best?

Which product generated the most revenue?

How many orders were processed?

What is the average order value?

What are the monthly sales trends?

This is where programming starts becoming data analysis.

1.25.27Add Data Quality Checks

  • A professional ETL pipeline should check:
  • ✓ File exists
  • ✓ File isn't empty
  • ✓ Required columns exist
  • ✓ Data types are valid
  • ✓ Quantity isn't negative
  • ✓ Price isn't negative
  • ✓ Dates are valid
  • ✓ Duplicate records
  • ✓ Missing values

You can calculate:

quality_report = {
    "row_count": len(df),
    "duplicate_count": df.duplicated().sum(),
    "missing_values": df.isna().sum().sum()
}

1.25.28Add Unit Tests

Create:

tests/test_transform.py

Example:

import pandas as pd
from src.transform import clean_data
def test_sales_amount():
    df = pd.DataFrame({
        "Order_Date": [
            "2026-01-01"
        ],
        "Quantity": [
            2
        ],
        "Unit_Price": [
            100
        ]
    })
result = clean_data(df)
  • assert (
  • result["Sales_Amount"].iloc[0]
  • == 200

)

  • Run:
  • pytest
  • If everything is correct:
  • 1 passed

1.25.29Add .gitignore

  • Create:
  • .gitignore
  • Add:
  • .venv/
  • __pycache__/
  • *.pyc
  • .env
  • logs/
  • *.log
  • data/output/
  • reports/

You generally don't want generated files, local environments, or secrets committed to Git.

1.25.30README

Create:

README.md

Example:

  • # Sales ETL & Reporting System
  • ## Overview
  • A Python-based ETL pipeline for processing
  • sales data and generating analytical reports.
  • ## Features
  • - CSV extraction
  • - Data validation
  • - Data cleaning
  • - Duplicate removal
  • - Sales calculations
  • - Excel reporting
  • - JSON reporting
  • - Visualization
  • - Logging
  • - Exception handling
  • - Unit testing
  • ## Installation
  • ```bash
python -m venv .venv

Activate the environment and install:

pip install -r requirements.txt

Run

python main.py

Test

pytest

---

  • # 1.25.31 Final Architecture
  • Your completed project will look like:
  • ```text
  • SALES.CSV

┌───────────────┐

│ EXTRACT │

│ Pandas │

└───────┬───────┘

┌───────────────┐

│ VALIDATE │

│ Data Quality │

└───────┬───────┘

┌───────────────┐

│ TRANSFORM │

│ Clean + KPIs │

└───────┬───────┘

┌───────────┼───────────┐

▼ ▼ ▼

Clean CSV Excel JSON

Visualization

Reports

───────────────────────

Logging

sales_etl.log

1.25.32What You Have Built

This isn't just a Python exercise.

You've built a small production-style ETL application containing:

Python
├── Functions

├── OOP

├── Modules

├── Exception Handling

├── Logging
├── Pandas

├── NumPy

├── Matplotlib

├── CSV

├── Excel

├── JSON

├── Data Validation

├── Data Cleaning

├── ETL

├── Reporting
├── Unit Testing

├── Virtual Environment

├── Git

└── Best Practices

1.25.33Module 1 — Final Learning Outcome

After completing this project, you should be comfortable with:

  • Python
  • Variables
  • Conditions
  • Loops
  • Functions
  • OOP
  • Modules
  • Exceptions
  • Decorators
  • Generators
  • Comprehensions
  • Data
  • CSV
  • Excel
  • JSON
  • Pandas
  • NumPy
  • Visualization
  • Matplotlib
  • Production Concepts
  • Logging
  • Validation
  • Testing
  • Configuration
  • Virtual environments
  • Git
  • Project structure
  • Error handling
  • Security
  • Data Engineering
  • Most importantly:
  • SOURCE

EXTRACT

VALIDATE

TRANSFORM

LOAD

REPORT

  • MONITOR
  • That ETL pattern will appear repeatedly throughout your career as a Data Engineer.
  • 🎓 Module 1 Complete
  • Your first module is now:
  • MODULE 1 – PYTHON PROGRAMMING

═══════════════════════════════