Numerical Stability & Vectorization

🔄 Restart Kernel ⏳ Loading Pyodide…

Have you ever seen your training loss suddenly turn into NaN (Not-a-Number)? That’s your model screaming:

“Help! I just divided by zero or overflowed to infinity!” 😭

Welcome to the world of Numerical Stability — where numbers behave badly and we must keep them under control.

---

💣 What Can Go Wrong?

Let’s meet the villains of numerical chaos 👇

Villain	Description	Example Disaster
Overflow	Numbers get too big to store	exp(1000) → 💥
Underflow	Numbers get too small to notice	exp(-1000) → 0.0
Division by Zero	The forbidden math act 😬	1/0 → Infinity
Loss Explosion	Training loss goes from 0.3 → 300,000 overnight	Classic deep learning drama 🎭

---

🧊 Cool Tricks to Stay Stable

Here’s how the pros keep their models from catching fire 🔥:

🧯 1. Log-Sum-Exp Trick

When dealing with probabilities: [ \text{softmax}(x_i) = \frac{e^{x_i}}{\sum_j e^{x_j}} ] can overflow if (x_i) is huge. Use: [ \text{softmax}(x_i) = \frac{e^{x_i - \max(x)}}{\sum_j e^{x_j - \max(x)}} ] ✅ Subtracting max(x) keeps numbers in check.

---

🧮 2. Add Epsilon (ϵ)

When dividing or taking logs, add a tiny constant to stay safe:

▶ Run Clear

`

It’s like putting bubble wrap around your math 📦.

---

🧤 3. Gradient Clipping

During backpropagation, gradients sometimes go berserk 🤯. Clamp them down:

▶ Run Clear

Think of it as telling your model, “Calm down, champ.” 🧘

---

⚡ 4. Normalize Inputs

Feeding unscaled data to a model is like giving a toddler 10 espressos ☕. Normalize features: [ x’ = \frac{x - \mu}{\sigma} ] so everything stays balanced.

---

🧠 Vectorization: The Zen of Speed and Stability

Loops in Python are like waiting in a grocery line with one cashier 🐢. Vectorized operations (NumPy, PyTorch, etc.) open 20 registers at once 🏎️.

Example:

▶ Run Clear

Result: Same output, but the vectorized version runs ~100x faster and is more numerically stable due to optimized C backend.

---

🧪 Practice: Find the Stability Hero

▶ Run Clear

Check the difference — the first one will blow up 💥, the second one will calmly return [0.5, 0.5]. 😌

---

🎯 TL;DR — Stability Survival Kit

Problem	Fix
Overflow	Use log-sum-exp
Division by zero	Add epsilon (1e-8)
Exploding gradients	Clip them
Unstable features	Normalize or standardize
Slow math	Vectorize everything

---

💬 “Numerical stability: because one NaN can ruin your entire day.” 😅

---

🔗 Next Up: Optimization Lab – Comparing GD Variants Time to put all your optimizer powers to the test in one grand experiment ⚔️.

# Your code here