Teaching Machines the Art of Regret

“To err is human. To minimize that error — that’s machine learning.” 🤖

🎯 What Is MSE?¶

Imagine your ML model as a junior analyst making predictions about sales. Every time it’s wrong, you slap it with a gentle mathematical frown 😠 — that’s the error.

Now, instead of yelling at it for each mistake, you:

1. Measure how far off it was (difference between prediction & truth),

2. Square it (because negative errors also hurt),

3. Average them all.

That’s Mean Squared Error (MSE) — the average squared disappointment of your model.

[ \text{MSE} = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y_i})^2 ]

Where:

• ( y_i ) = actual value

• ( \hat{y_i} ) = predicted value

• ( n ) = number of observations

🧮 Example: Predicting Sales¶

[ \text{MSE} = \frac{100 + 100 + 900}{3} = 366.67 ]

💬 “On average, the model is about 19 units off — but because errors are squared, big mistakes really sting.” 🥲

📊 Why Square the Errors?¶

Because math loves drama. Squaring does two things:

1. Eliminates negatives (so overestimates and underestimates don’t cancel out),

2. Punishes big errors — the way your boss remembers that one bad forecast from last year.

💼 Business Analogy¶

Let’s say you’re a coffee chain manager predicting daily sales.

• Predicting 480 cups when you sell 500 → okay, close enough. ☕

• Predicting 50 cups when you sell 500 → catastrophic caffeine shortage! 😱

Squaring ensures the big misses matter a lot more.

That’s why MSE = “Measure of how badly your model can mess up, weighted by embarrassment.”

⚙️ In Python¶

from sklearn.metrics import mean_squared_error

y_true = [100, 200, 300]
y_pred = [110, 190, 270]

mse = mean_squared_error(y_true, y_pred)
print(f"MSE: {mse:.2f}")
print(f"RMSE: {mse**0.5:.2f}")  # Root Mean Squared Error

Output:

MSE: 366.67
RMSE: 19.15

💡 RMSE (Root Mean Squared Error) brings the units back — so it’s easier to explain in meetings. “Our model’s off by about 19 sales per day on average.” 📉

📈 Visual Intuition¶

import matplotlib.pyplot as plt
import numpy as np

y_actual = np.array([100, 200, 300])
y_pred = np.array([110, 190, 270])

plt.scatter(y_actual, y_pred, color='blue', label='Predictions')
plt.plot([0, 300], [0, 300], color='red', linestyle='--', label='Perfect Prediction Line')
for i in range(len(y_actual)):
    plt.plot([y_actual[i], y_actual[i]], [y_actual[i], y_pred[i]], 'gray', linestyle=':')
plt.xlabel('Actual')
plt.ylabel('Predicted')
plt.title('Error Distances (MSE Visualization)')
plt.legend()
plt.show()

Each gray line = a “mistake.” The longer the line, the bigger the pain (and the bigger the MSE). 💔

🧠 When to Use MSE¶

💬 Pro tip: If your business hates big mistakes (like loan defaults or stock forecasts), go with MSE/RMSE. If you want to treat all errors equally, use MAE.

🤹 Practice Corner: “Forecast Like a Boss”¶

Try this small challenge 👇

You’re predicting next month’s sales revenue for 4 stores:

1. Compute MSE and RMSE.

2. Which store has the largest squared error?

3. In business terms — what does this tell you?

💡 Hint: One bad forecast can ruin your bonus report 😅

🧩 Common Pitfalls¶

🧭 Recap¶

💬 Final Thought¶

“MSE is like guilt — it teaches your model to reflect on its mistakes, but too much of it and nobody learns anything.” 😅

🔜 Next Up¶

👉 Head to Gradients & Partial Derivatives — where your model learns how to apologize properly by adjusting itself step by step. 🧘‍♂️

“Because knowing you’re wrong is one thing — learning from it is where the magic happens.” 🌟

An Objective: Mean Squared Error¶

We pick $\theta$ to minimize the mean squared error (MSE). Slight variants of this objective are also known as the residual sum of squares (RSS) or the sum of squared residuals (SSR).

In other words, we are looking for the best compromise in $\theta$ over all the data points.

Mean Squared Error: Partial Derivatives¶

Let’s work out the derivatives for $\frac{1}{2} \left( f_\theta(x^{(i)}) - y^{(i)} \right)^2,$ the MSE of a linear model $f_\theta$ for one training example $(x^{(i)}, y^{(i)})$, which we denote $J^{(i)}(\theta)$.

Mean Squared Error: The Gradient¶

We can use this derivation to obtain an expression for the gradient of the MSE for a linear model

Note that the MSE over the entire dataset is $J(\theta) = \frac{1}{n}\sum_{i=1}^n J^{(i)}(\theta)$. Therefore:

Ordinary Least Squares¶

The gradient $\nabla_\theta f$ further extends the derivative to multivariate functions $f : \mathbb{R}^d \to \mathbb{R}$, and is defined at a point $\theta_0$ as

In other words, the $j$-th entry of the vector $\nabla_\theta f (\theta_0)$ is the partial derivative $\frac{\partial f(\theta_0)}{\partial \theta_j}$ of $f$ with respect to the $j$-th component of $\theta$.

Notation: Design Matrix¶

Machine learning algorithms are most easily defined in the language of linear algebra. Therefore, it will be useful to represent the entire dataset as one matrix $X \in \mathbb{R}^{n \times d}$, of the form:

Notation: Target Vector¶

Similarly, we can vectorize the target variables into a vector $y \in \mathbb{R}^n$ of the form

Squared Error in Matrix Form¶

Recall that we may fit a linear model by choosing $\theta$ that minimizes the squared error:

We can write this sum in matrix-vector form as:

where $X$ is the design matrix and $\|\cdot\|$ denotes the Euclidean norm.

The Gradient of the Squared Error¶

We can compute the gradient of the mean squared error as follows.

We used the facts that $a^\top b = b^\top a$ (line 3), that $\nabla_x b^\top x = b$ (line 4), and that $\nabla_x x^\top A x = 2 A x$ for a symmetric matrix $A$ (line 4).

Normal Equations¶

Setting the above derivative to zero, we obtain the normal equations:

Hence, the value $\theta^*$ that minimizes this objective is given by:

Note that we assumed that the matrix $(X^\top X)$ is invertible; we will soon see a simple way of dealing with non-invertible matrices.

Why square the difference?¶

1. The error will be positive

2. If you take the absolute function (to cover point 1), the absolute function isn’t differentiable at the origin. Hence, we square the error.

 Why $\frac{1}{2m}$ instead of $\frac{1}{m}$ ?¶

As we will see later, when we differentiate the squared error, the $\frac{1}{2}$ will cancel out. If we don’t do that we’ll be stuck with a 2 in the equation which is useless.

Minimising the cost function¶

Our objective function is

$$\displaystyle \operatorname*{argmin}_\theta J(\theta)$$

(13)

¶

Which simply means, find the value of $\theta$ that minimises the error function $J(\theta)$. In order to do that, we will differentiate our cost function. When we differentiate it, it will give us gradient, which is the direction in which the error will be reduced. Upon having the gradient, we will simply update our $\theta$ values to reflect that step (a step in the direction of lower error)

So, the update rule is the following equation

$$\theta = \theta - \alpha \frac{\partial}{\partial \theta} J(\theta)$$

(14)

¶

Where,

$\alpha$ = learning rate. Which is the rate at which we will travel to the direction of the lower error.

This process is nothing but Gradient Descent. There are few version of gradient descent, few of them are:

1. Batch Gradient Descent: Go through all your input samples, compute the gradient once, and then update $\theta$s.

2. Stochastic Gradient Descent: Go through a single sample, compute gradient, update $\theta$s, repeat $m$ times

3. Mini Batch Gradient Descent: Go through a batch of $k$ samples, compute gradient, update $\theta$s, repeat $\frac{m}{k}$ times.

Differentiating the loss function:¶

In the update rule:

$$\theta = \theta - \alpha \frac{\partial}{\partial \theta} J(\theta)$$

(15)

¶

The important part is calculating the derivative. Since we have two variables, we will have two derivatives, one for $\theta_0$ and another for $\theta_1$.

So the first equation is:

$

$

Similarly, for $\theta_1$¶

$

$

We will implement Batch Gradient Descent i.e. we’ll update the gradients after 1 pass through the entire dataset. Our Algorithm hence becomes:

Repeat till convergence:¶

1.       $\theta_0= \theta_0 - \frac{1}{m}\sum_{i=0}^m{(h_\theta(x_i) - y_i)}$¶

2.       $\theta_1= \theta_1 - \frac{1}{m}\sum_{i=0}^m{(h_\theta(x_i) - y_i)}(x)$¶

theta, theta_prev = random_initialization()
while norm(theta - theta_prev) > convergence_threshold:
    theta_prev = theta
    theta = theta_prev - step_size * gradient(theta_prev)

In the above algorithm, we stop when $||\theta_i - \theta_{i-1}||$ is small.