Exercises¶

1. Using the demo data, write flag_and_clip(df, col) that returns a copy with is_outlier and col_clipped.

2. Experiment with different IQR multipliers (1.5 vs 3.0) and report how many points are flagged.

3. (Challenge) Implement a winsorize function that replaces values outside bounds with the nearest bound and test it on the demo.

Hints:
- Use `df[col].quantile(0.25)` and `quantile(0.75)` for IQR
- Use `df.clip(lower=..., upper=...)` to winsorize

Handling Missing Data & Outliers¶

Practical, business-minded patterns for imputing, flagging, and handling extreme values. Includes a Pyodide-safe demo that runs on generated data when files aren't available.

Learning goals

• Distinguish when to impute, flag, or remove missing values.

• Detect outliers with IQR and visualize their impact before taking action.

• Apply safe, reproducible cleaning steps useful for business reports and model pipelines.

Handling Missing Data & Outliers¶

Managing Incomplete Records and Extreme Values Responsibly¶

Notebook Guide¶

This notebook preserves the original explanations and examples while adding a study scaffold.

Central questions¶

• when should missing values be imputed, flagged, or left untouched

• when is an outlier an error versus a meaningful event

• how do these decisions affect business interpretation and model behavior

Real-world business quote: “Garbage in, garbage out. But 80% of the battle in analytics is making sure the garbage never gets in.” — Former Chief Data Officer, Walmart

🎯 Why Every Business Student MUST Master This (Before Your First Internship)¶

Bottom line: 70–80% of a data analyst’s time is spent cleaning data. Master this → you become the most valuable intern in the room.¶

Step 1: Load Data & First Crime-Scene Investigation¶

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

df = pd.read_csv("sales_data.csv")
print(df.head())
print(df.info())

Business student tip: Always check dtypes — if sales_amount is object → someone typed “₹50,000” instead of 50000 → disaster!

Step 2: Missing Values → The Ghost Customers¶

missing = pd.DataFrame({
    'Missing Count': df.isnull().sum(),
    '% Missing': round(df.isnull().sum() / len(df) * 100, 2)
})
print(missing)

Real Business Rules (Add these comments in your code!)¶

# Business Rule 1: If <5% missing → safe to impute
# Business Rule 2: If >30% missing in a column → consider dropping entire column
# Business Rule 3: Never ever use mean() for revenue/salary → always median()

Smart Imputation Cheat-Sheet for Business Data¶

Pro move for interviews:

# Segment-wise median — shows you think like a business analyst
df['sales_amount'] = df.groupby('region')['sales_amount'].transform(
    lambda x: x.fillna(x.median())
)

Step 3: Outliers → The ₹9,99,99,999 “Typo” That Crashed Your Model¶

Visualization First (Always!)¶

plt.figure(figsize=(12,4))
plt.subplot(1,2,1)
sns.boxplot(x=df['sales_amount'])
plt.title("Boxplot - Spot the Billionaire Typo")

plt.subplot(1,2,2)
sns.histplot(df['sales_amount'], bins=50, kde=True)
plt.title("Histogram - Is that a second peak at 1 Billion?")
plt.tight_layout()
plt.show()

IQR Method (Standard in 90% of companies)¶

def detect_outliers_iqr(data, column):
    Q1 = data[column].quantile(0.25)
    Q3 = data[column].quantile(0.75)
    IQR = Q3 - Q1
    lower = Q1 - 1.5 * IQR
    upper = Q3 + 1.5 * IQR

    outliers = data[(data[column] < lower) | (data[column] > upper)]
    print(f"⚠️ Found {len(outliers)} outliers in {column}")
    return outliers, lower, upper

outliers, lower_bound, upper_bound = detect_outliers_iqr(df, 'sales_amount')

Business Decision Matrix (Paste this in your Jupyter notebook!)¶

# Winsorization (used by HDFC Bank risk models)
df['sales_amount'] = df['sales_amount'].clip(lower=lower_bound, upper=upper_bound)

# Or create a flag (Goldman Sachs style)
df['is_outlier'] = (df['sales_amount'] < lower_bound) | (df['sales_amount'] > upper_bound)

Step 4: Before vs After — The Money Shot for Your Resume¶

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15,5))

ax1.hist(df_raw['sales_amount'], bins=50, alpha=0.7, label='Before', color='red')
ax1.set_title("Before Cleaning - Chaos")
ax1.legend()

ax2.hist(df['sales_amount'], bins=50, alpha=0.7, label='After', color='green')
ax2.set_title("After Cleaning - Beautiful Normal Distribution")
ax2.legend()

plt.suptitle("Impact of Proper Data Cleaning - From Garbage to Gold")
plt.show()

Sample insight to impress your interviewer:

“After cleaning, average ticket size dropped from ₹1,42,000 to ₹68,000 — revealing that 3 typo entries were inflating AOV by 108%. This prevents ₹42 crore over-forecast in annual revenue.”

The 3 Golden Rules for Business Analytics Roles¶

1. Never trust raw data — always assume it’s lying

2. Document every cleaning decision — your future self (or auditor) will thank you

3. Always validate with business stakeholder — “Is ₹5 crore order possible?” → call sales head!

Analyzing Distribution¶

• Description: Quantifying the shape of a distribution by evaluating its asymmetry (skewness) and the heaviness of its tails (kurtosis). These measures provide insights beyond central tendency and dispersion, revealing important characteristics of the data’s distribution.

• Related Concepts:

  • Skewness: A measure of the asymmetry of the probability distribution of a real-valued random variable about its mean. A positive skew indicates a long tail extending towards more positive values (right-skewed), while a negative skew indicates a long tail extending towards more negative values (left-skewed). The formula for sample skewness is:

    $$\text{Skew} = \frac{\frac{1}{n} \sum_{i=1}^n (x_i - \mu)^3}{\sigma^3}$$

    (1)

    where $x_i$ are the data points, $\mu$ is the sample mean, $\sigma$ is the sample standard deviation, and $n$ is the number of data points.

  • Kurtosis: A measure of the “tailedness” of the probability distribution. High kurtosis indicates that the distribution has heavier tails and a sharper peak around the mean compared to a normal distribution, suggesting more frequent extreme values. The formula for sample kurtosis (often adjusted to excess kurtosis by subtracting 3, where a normal distribution has a kurtosis of 3) is:

    $$\text{Kurt} = \frac{\frac{1}{n} \sum_{i=1}^n (x_i - \mu)^4}{\sigma^4}$$

    (2)

    Excess kurtosis is then $\text{Kurt}_{excess} = \text{Kurt} - 3$.

• Example: In financial modeling, checking the skewness of stock returns is vital. A significant negative skew might indicate a higher probability of large negative returns (crashes) compared to large positive returns. Similarly, examining kurtosis helps assess the risk associated with extreme price movements.

Distribution analysis provides crucial insights into the shape of time series data, helping to determine if the returns or values are normally distributed, skewed, or exhibit heavy tails.

Tools for Assessing Distribution Shape:¶

• Histogram: Visual inspection can reveal asymmetry and the relative frequency of values in the tails.

• KDE Plot: Provides a smoothed visualization of the distribution, making it easier to observe skewness and the overall shape.

• Skewness Coefficient: A numerical measure of asymmetry.

  $$\text{Skewness} > 0 \implies \text{Right-tailed (positive skew)}$$

  (3)
  $$\text{Skewness} < 0 \implies \text{Left-tailed (negative skew)}$$

  (4)
  $$\text{Skewness} \approx 0 \implies \text{Symmetric}$$

  (5)

• Kurtosis Coefficient: A numerical measure of the tailedness of the distribution.

  $$\text{Kurtosis} > 3 \implies \text{Leptokurtic (heavy tails, sharper peak)}$$

  (6)
  $$\text{Kurtosis} < 3 \implies \text{Platykurtic (thin tails, flatter peak)}$$

  (7)
  $$\text{Kurtosis} \approx 3 \implies \text{Mesokurtic (similar to normal distribution)}$$

  (8)

🔍 Detecting and Handling Missing Data in Pandas¶

Reindexing¶

Reindexing allows you to change/add/delete the index on a specified axis:

Dropping Missing Data¶

Use .dropna() to remove rows/columns with missing values:

Filling Missing Data¶

Use .fillna() to fill missing values with a specified value:

Detecting Missing Values¶

Use isna() or isnull() to get a boolean mask:

Summary¶

The important habit here is not blindly removing unusual values. Use the original notebook examples to practice asking whether a value is impossible, rare, or business-significant.

8. Interactive Code¶

[10, 12, 11, 90]

[90]

9. Guided Practice¶