Real-world business quote: “Garbage in, garbage out. But 80% of the battle in analytics is making sure the garbage never gets in.” — Former Chief Data Officer, Walmart

🎯 Why Every Business Student MUST Master This (Before Your First Internship)¶

Bottom line: 70–80% of a data analyst’s time is spent cleaning data. Master this → you become the most valuable intern in the room.¶

Step 1: Load Data & First Crime-Scene Investigation¶

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

df = pd.read_csv("sales_data.csv")
print(df.head())
print(df.info())

Business student tip: Always check dtypes — if sales_amount is object → someone typed “₹50,000” instead of 50000 → disaster!

Step 2: Missing Values → The Ghost Customers¶

missing = pd.DataFrame({
    'Missing Count': df.isnull().sum(),
    '% Missing': round(df.isnull().sum() / len(df) * 100, 2)
})
print(missing)

Real Business Rules (Add these comments in your code!)¶

# Business Rule 1: If <5% missing → safe to impute
# Business Rule 2: If >30% missing in a column → consider dropping entire column
# Business Rule 3: Never ever use mean() for revenue/salary → always median()

Smart Imputation Cheat-Sheet for Business Data¶

Pro move for interviews:

# Segment-wise median — shows you think like a business analyst
df['sales_amount'] = df.groupby('region')['sales_amount'].transform(
    lambda x: x.fillna(x.median())
)

Step 3: Outliers → The ₹9,99,99,999 “Typo” That Crashed Your Model¶

Visualization First (Always!)¶

plt.figure(figsize=(12,4))
plt.subplot(1,2,1)
sns.boxplot(x=df['sales_amount'])
plt.title("Boxplot - Spot the Billionaire Typo")

plt.subplot(1,2,2)
sns.histplot(df['sales_amount'], bins=50, kde=True)
plt.title("Histogram - Is that a second peak at 1 Billion?")
plt.tight_layout()
plt.show()

IQR Method (Standard in 90% of companies)¶

def detect_outliers_iqr(data, column):
    Q1 = data[column].quantile(0.25)
    Q3 = data[column].quantile(0.75)
    IQR = Q3 - Q1
    lower = Q1 - 1.5 * IQR
    upper = Q3 + 1.5 * IQR

    outliers = data[(data[column] < lower) | (data[column] > upper)]
    print(f"⚠️ Found {len(outliers)} outliers in {column}")
    return outliers, lower, upper

outliers, lower_bound, upper_bound = detect_outliers_iqr(df, 'sales_amount')

Business Decision Matrix (Paste this in your Jupyter notebook!)¶

# Winsorization (used by HDFC Bank risk models)
df['sales_amount'] = df['sales_amount'].clip(lower=lower_bound, upper=upper_bound)

# Or create a flag (Goldman Sachs style)
df['is_outlier'] = (df['sales_amount'] < lower_bound) | (df['sales_amount'] > upper_bound)

Step 4: Before vs After — The Money Shot for Your Resume¶

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15,5))

ax1.hist(df_raw['sales_amount'], bins=50, alpha=0.7, label='Before', color='red')
ax1.set_title("Before Cleaning - Chaos")
ax1.legend()

ax2.hist(df['sales_amount'], bins=50, alpha=0.7, label='After', color='green')
ax2.set_title("After Cleaning - Beautiful Normal Distribution")
ax2.legend()

plt.suptitle("Impact of Proper Data Cleaning - From Garbage to Gold")
plt.show()

Sample insight to impress your interviewer:

“After cleaning, average ticket size dropped from ₹1,42,000 to ₹68,000 — revealing that 3 typo entries were inflating AOV by 108%. This prevents ₹42 crore over-forecast in annual revenue.”

The 3 Golden Rules for Business Analytics Roles¶

1. Never trust raw data — always assume it’s lying

2. Document every cleaning decision — your future self (or auditor) will thank you

3. Always validate with business stakeholder — “Is ₹5 crore order possible?” → call sales head!

Analyzing Distribution¶

• Description: Quantifying the shape of a distribution by evaluating its asymmetry (skewness) and the heaviness of its tails (kurtosis). These measures provide insights beyond central tendency and dispersion, revealing important characteristics of the data’s distribution.

• Related Concepts:

  • Skewness: A measure of the asymmetry of the probability distribution of a real-valued random variable about its mean. A positive skew indicates a long tail extending towards more positive values (right-skewed), while a negative skew indicates a long tail extending towards more negative values (left-skewed). The formula for sample skewness is:

    $$\text{Skew} = \frac{\frac{1}{n} \sum_{i=1}^n (x_i - \mu)^3}{\sigma^3}$$

    (1)

    where $x_i$ are the data points, $\mu$ is the sample mean, $\sigma$ is the sample standard deviation, and $n$ is the number of data points.

  • Kurtosis: A measure of the “tailedness” of the probability distribution. High kurtosis indicates that the distribution has heavier tails and a sharper peak around the mean compared to a normal distribution, suggesting more frequent extreme values. The formula for sample kurtosis (often adjusted to excess kurtosis by subtracting 3, where a normal distribution has a kurtosis of 3) is:

    $$\text{Kurt} = \frac{\frac{1}{n} \sum_{i=1}^n (x_i - \mu)^4}{\sigma^4}$$

    (2)

    Excess kurtosis is then $\text{Kurt}_{excess} = \text{Kurt} - 3$.

• Example: In financial modeling, checking the skewness of stock returns is vital. A significant negative skew might indicate a higher probability of large negative returns (crashes) compared to large positive returns. Similarly, examining kurtosis helps assess the risk associated with extreme price movements.

Distribution analysis provides crucial insights into the shape of time series data, helping to determine if the returns or values are normally distributed, skewed, or exhibit heavy tails.

Tools for Assessing Distribution Shape:¶

• Histogram: Visual inspection can reveal asymmetry and the relative frequency of values in the tails.

• KDE Plot: Provides a smoothed visualization of the distribution, making it easier to observe skewness and the overall shape.

• Skewness Coefficient: A numerical measure of asymmetry.

  $$\text{Skewness} > 0 \implies \text{Right-tailed (positive skew)}$$

  (3)
  $$\text{Skewness} < 0 \implies \text{Left-tailed (negative skew)}$$

  (4)
  $$\text{Skewness} \approx 0 \implies \text{Symmetric}$$

  (5)

• Kurtosis Coefficient: A numerical measure of the tailedness of the distribution.

  $$\text{Kurtosis} > 3 \implies \text{Leptokurtic (heavy tails, sharper peak)}$$

  (6)
  $$\text{Kurtosis} < 3 \implies \text{Platykurtic (thin tails, flatter peak)}$$

  (7)
  $$\text{Kurtosis} \approx 3 \implies \text{Mesokurtic (similar to normal distribution)}$$

  (8)

🔍 Detecting and Handling Missing Data in Pandas¶

Reindexing¶

Reindexing allows you to change/add/delete the index on a specified axis:

Dropping Missing Data¶

Use .dropna() to remove rows/columns with missing values:

Filling Missing Data¶

Use .fillna() to fill missing values with a specified value:

Detecting Missing Values¶

Use isna() or isnull() to get a boolean mask: