Supervised Regression – Linear Models¶

Turning numbers into predictions — the foundation of quantitative business analysis

You already know how to work with vectors, matrices, derivatives, and expected values from the Math Foundations block. Now those tools become a real model: one that learns a relationship from data and makes predictions you can explain to a business stakeholder in a single sentence.

Why Regression Matters in Business¶

Regression answers the question every business leader asks: "If this input changes, what happens to the outcome?" It is interpretable, fast to train, and often accurate enough for a first production model.

Business question	Regression answer
How much revenue does one extra dollar of ad spend generate?	Coefficient on `ad_spend`
What will next quarter’s sales be?	Predicted $\hat{y}$ from a trained model
Which product features drive price?	Coefficients ranked by magnitude
How risky is this loan application?	Predicted expected loss
Will this marketing channel improve ROI?	Coefficient sign and confidence interval

The Core Equation¶

A linear model combines features using a weighted sum:

\hat{y} \;=\; \color{#7c3aed}{\beta_0} + \color{#1d4ed8}{\beta_1}\,\color{#047857}{x_1} + \color{#1d4ed8}{\beta_2}\,\color{#047857}{x_2} + \cdots + \color{#1d4ed8}{\beta_n}\,\color{#047857}{x_n}

(1)

Color legend: $\color{#047857}{\text{green}} =$ input features $\;|\;$ $\color{#1d4ed8}{\text{blue}} =$ learned coefficients $\;|\;$ $\color{#7c3aed}{\text{purple}} =$ intercept

Symbol	Name	Plain meaning
$\hat{y}$	Prediction	The number the model outputs
$x_i$	Feature	One measurable input
$\beta_i$	Coefficient	How much $\hat{y}$ changes per unit change in $x_i$
$\beta_0$	Intercept	Baseline prediction when all features are zero

In matrix form the same equation compresses to:

\hat{\mathbf{y}} \;=\; \mathbf{X}\,\boldsymbol{\beta}

(2)

where $\mathbf{X}$ is the feature matrix (one row per observation, one column per feature plus a bias column of ones) and $\boldsymbol{\beta}$ is the coefficient vector.

Visual Intuition — How a Linear Model Works¶

The diagram below traces the lifecycle of a regression model from raw data to a business decision.

Alt: Data enters as a matrix, the model minimises squared errors to learn coefficients, then those coefficients produce predictions that drive decisions.

The cost the model minimises is Mean Squared Error (MSE):

\text{MSE} \;=\; \frac{1}{n}\sum_{i=1}^{n}\bigl(y_i - \hat{y}_i\bigr)^2

(3)

A smaller MSE means predictions stay closer to the actual values. The MSE notebook covers this in depth.

Worked Example — Ad Spend vs Sales¶

A marketing team collected weekly advertising spend and sales revenue. The question: does spending more on ads reliably lift sales, and by how much?

Ad_Spend: feature (£) Sales: target (£ thousands) model.coef_: rate of change model.intercept_: baseline

import pandas as pd
from sklearn.linear_model import LinearRegression
import matplotlib.pyplot as plt

data = pd.DataFrame({
    'Ad_Spend': [100, 200, 300, 400, 500, 600, 700],
    'Sales':    [12,  22,  26,  36,  44,  52,  60]
})

X = data[['Ad_Spend']]
y = data['Sales']

model = LinearRegression().fit(X, y)
print(f"Coefficient (slope):  {model.coef_[0]:.4f}  → each £1 extra spend → £{model.coef_[0]:.4f}k extra sales")
print(f"Intercept (baseline): {model.intercept_:.2f}")
print(f"R² on training data:  {model.score(X, y):.4f}")

# Plot
fig, ax = plt.subplots(figsize=(7, 4))
ax.scatter(data['Ad_Spend'], data['Sales'], color='#1d4ed8', s=60, label='Actual data', zorder=3)
ax.plot(data['Ad_Spend'], model.predict(X), color='#b91c1c', lw=2, label='Fitted line')
ax.set_xlabel('Advertising Spend (£)')
ax.set_ylabel('Sales (£ thousands)')
ax.set_title('Linear Regression: Ad Spend → Sales')
ax.legend()
plt.tight_layout()
plt.show()

Coefficient (slope):  0.0793  → each £1 extra spend → £0.0793k extra sales
Intercept (baseline): 4.29
R² on training data:  0.9956

Key Assumptions¶

Linear regression relies on four main assumptions. Violating them does not always break the model, but it does limit what you can safely conclude.

Assumption	If violated	Practical check
Linearity	Model underfits curved patterns	Residual vs fitted plot — look for curves
Independence	Standard errors are wrong	Time or group structure in residuals
Homoscedasticity	Confidence intervals are unreliable	Residual spread widens with fitted values
Low multicollinearity	Coefficients become unstable	Variance Inflation Factor (VIF) > 10

Interactive — Explore Slope and Intercept¶

Use the Pyodide cell below to change the coefficient (slope) and intercept values and observe how the predicted line shifts. This builds intuition before you move to fitting from real data.

Chapter Map¶

This chapter is a parent section. Each child notebook covers one focused topic. Use the map below to navigate.

Suggested reading order: Linear Model Family → MSE → Metrics → Gradients → OLS → Polynomial → Regularization → Bias–Variance → Lab.

Guided Practice¶

What kind of output does a regression model produce?¶

A continuous numerical valueCorrect. Regression predicts quantities such as revenue, price, demand, or loss — not category labels.

A class label like "churn" or "no churn"That is a classification task. Regression returns a number on a continuous scale.

A probability between 0 and 1 onlyLinear regression is not bounded to [0, 1]. Logistic regression uses that range for classification.

A ranked list of itemsRanking is a different task. Regression returns a single scalar prediction per input.

A coefficient of 0.09 on an "ad_spend" feature means:¶

The model has 9 % accuracyAccuracy is a classification metric. Coefficients describe feature relationships, not model performance.

Each extra unit of ad spend is associated with a 0.09 unit increase in the predictionCorrect. A coefficient represents the estimated change in the target for a one-unit increase in that feature, all else equal.

The model uses 9 % of the data for trainingData splits are a separate concept. Coefficients describe the learned relationship.

Ad spend is the ninth-most important featureFeature importance requires comparing standardised coefficients or permutation importance — the raw coefficient value alone is not an importance rank.

Which assumption is most likely violated when a residual plot shows a funnel shape (small spread on the left, large spread on the right)?¶

LinearityLinearity violations look like curves in the residual plot, not a widening cone.

IndependenceIndependence violations appear as patterns over time or groups, not a funnel shape.

Homoscedasticity (constant variance)Correct. A funnel shape means error variance grows with the fitted value — the definition of heteroscedasticity.

Low multicollinearityMulticollinearity affects coefficient stability, not the pattern of residual spread across fitted values.

Exercises¶

Exercise 1 — Interpret a coefficient

A model trained on house price data returns:

Intercept: 50,000
Coefficient on floor_area_sqm: 2,500
Coefficient on distance_to_station_km: -8,000

Answer in plain language: (a) What does the model predict for a 100 sqm house 1 km from a station? (b) What does the negative coefficient on distance mean for a business selling property near commuter hubs?

Exercise 2 — Fit and plot from data

Use the dataset below. Fit a linear regression, print the coefficient and intercept, plot the fitted line, and calculate $R^2$ .

import pandas as pd
from sklearn.linear_model import LinearRegression

df = pd.DataFrame({
    'store_size_sqm': [200, 350, 500, 650, 800, 950, 1100],
    'weekly_revenue': [8000, 13500, 18000, 23500, 28000, 33000, 38500]
})

# TODO: Fit LinearRegression, print coefficient, intercept, R²
# TODO: Plot scatter + fitted line

import matplotlib.pyplot as plt

X = df[['store_size_sqm']]
y = df['weekly_revenue']
model = LinearRegression().fit(X, y)

print(f"Coefficient: {model.coef_[0]:.2f}")
print(f"Intercept:   {model.intercept_:.2f}")
print(f"R²:          {model.score(X, y):.4f}")

fig, ax = plt.subplots(figsize=(6, 4))
ax.scatter(df['store_size_sqm'], df['weekly_revenue'], color='#1d4ed8', label='Actual')
ax.plot(df['store_size_sqm'], model.predict(X), color='#b91c1c', lw=2, label='Fitted')
ax.set_xlabel('Store Size (sqm)')
ax.set_ylabel('Weekly Revenue (£)')
ax.legend()
plt.tight_layout()
plt.show()

Interpretation: A coefficient near 35 means each extra square metre is associated with roughly £35 of additional weekly revenue. $R^2$ close to 1.0 on this synthetic dataset confirms the linear fit is excellent.

Summary¶

Next Steps¶

The next notebook — Linear Model Family — maps out the full family of linear models: simple regression with one feature, multiple regression with many features, interaction terms, and generalised linear models. Start there to build a complete picture before diving into the MSE objective, gradient derivation, or regularization.

Up next: Linear Model Family — simple, multiple, and generalised linear models side by side.