Welcome to Logistic Regression, the model that takes the seriousness of math and says:
“Relax. I just want to predict whether they’ll buy the product or ghost you.” 👻
It’s simple, interpretable, and — despite the name — it’s not regression in the traditional sense. It’s classification in disguise!
🎯 Intuition¶
Imagine you’re a business analyst trying to predict:
Will a customer churn (1) or stay (0)?
A linear regression might predict something like 1.4 or -0.2. But you can’t tell your boss, “There’s a -0.2 chance the customer will leave.” 😬
So we need something that outputs probabilities between 0 and 1 — Enter Logistic Regression, our smooth-talking probability machine 🧠.
⚙️ The Sigmoid Function — Our Secret Sauce¶
The logistic model converts linear outputs into probabilities using the sigmoid function:
[ P(y=1|x) = \frac{1}{1 + e^{-(w^T x + b)}} ]
This squashes any real number into the 0–1 range:
Large positive → close to 1 (very likely to churn)
Large negative → close to 0 (probably staying)
Let’s plot it:
import numpy as np
import matplotlib.pyplot as plt
x = np.linspace(-6, 6, 200)
sigmoid = 1 / (1 + np.exp(-x))
plt.plot(x, sigmoid)
plt.title("Sigmoid Function – Turning Scores into Probabilities")
plt.xlabel("Score (w·x + b)")
plt.ylabel("P(y=1)")
plt.grid(True)
plt.show()That “S” shape? That’s the curve of business prediction love ❤️.
🧮 Training the Model (The Logistic Way)¶
We don’t use least squares here — we use Maximum Likelihood Estimation (MLE).
The goal is to find parameters ( w ) and ( b ) that maximize the probability of correctly predicting all outcomes.
In other words:
“Adjust weights until the model lies with confidence — but statistically.” 😅
The loss function is called Binary Cross-Entropy:
[ L = -\frac{1}{N} \sum_i [y_i \log(\hat{y}_i) + (1 - y_i) \log(1 - \hat{y}_i)] ]
💼 Business Example: Churn Probability¶
Let’s say you have data on customer engagement:
calls_per_monthmonthly_spendsupport_tickets
and you want to predict if a customer will churn.
import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
# Simple dataset
data = pd.DataFrame({
'calls_per_month': [3, 5, 8, 10, 12],
'monthly_spend': [50, 70, 100, 120, 150],
'churn': [0, 0, 0, 1, 1]
})
X = data[['calls_per_month', 'monthly_spend']]
y = data['churn']
model = LogisticRegression()
model.fit(X, y)
print("Predicted probabilities:")
print(model.predict_proba([[7, 90]]))Your model now says something like:
“Customer has a 72% chance of leaving.”
That’s your cue to send a “We miss you 💌” email.
🧠 Model Coefficients = Business Insights¶
You can interpret the coefficients just like linear regression:
| Feature | Coefficient | Interpretation |
|---|---|---|
calls_per_month | -0.5 | More calls → lower churn chance 📞 |
monthly_spend | +0.8 | Higher spend → slightly higher churn risk 💸 |
The signs and magnitudes tell you what drives customer behavior — it’s like X-ray vision for business data 👓.
🧩 Practice Challenge¶
Create a new feature:
discount_received.Re-train the model and see if discounts reduce churn probability.
Plot your new sigmoid curve.
Interpret the coefficients like a data-driven detective 🕵️.
🎯 TL;DR¶
| Concept | Meaning |
|---|---|
| Sigmoid | Converts scores → probabilities |
| Weights | Show direction & strength of influence |
| Cross-Entropy Loss | Penalizes wrong confidence |
| Output | Probability between 0 and 1 |
💬 “Logistic Regression is like a polite fortune teller — it won’t say you’ll churn, just that you probably will (with 87% confidence).” 🔮
🔗 Next Up: Naive Bayes – Meet the ancient probabilistic wizard who assumes all your features get along… even when they clearly don’t. 😅
Binary Classification: Predicts , the probability that given input features .
Sigmoid Function: Maps linear combinations of inputs to a probability between 0 and 1.
2. The Sigmoid Function¶
The core of logistic regression is the sigmoid (logistic) function:
(linear combination of weights and features , with bias ).
Output .
3. Hypothesis Representation¶
The predicted probability is:
4. Cost Function (Log Loss)¶
Logistic regression uses cross-entropy loss:
= number of training examples.
= true label (0 or 1).
= predicted probability.
Intuition:
If , cost when .
If , cost when .
5. Gradient Descent Optimization¶
Update weights and bias to minimize :
Where:
= learning rate.
The gradients are:
6. Decision Boundary¶
If , predict .
If , predict .
The boundary is where .
7. Multiclass Logistic Regression (Softmax)¶
For classes, use softmax regression:
8. Assumptions & Limitations¶
Assumes a linear decision boundary (use feature engineering for non-linear cases).
Sensitive to outliers (but less than linear regression).
Requires large datasets for stable weight estimates.
Example in Python (Scikit-learn)¶
from sklearn.linear_model import LogisticRegression
model = LogisticRegression()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)Example: Coin Flipping with Maximum Likelihood Estimation (MLE) and Maximum A Posteriori (MAP)¶
Imagine you flip a coin 10 times and get 7 heads. You want to estimate the probability of getting heads, denoted as . Is the coin fair () or biased? We’ll use MLE and MAP to find .
MLE: Trust Only the Data¶
MLE picks the that makes the observed data (7 heads in 10 flips) most likely.
You got 7 heads and 3 tails. The probability of this happening is modeled as:
MLE finds the that maximizes this. Intuitively, it’s like asking: “What best explains 7 heads out of 10 flips?”
Without diving into calculus, the answer is the proportion of heads:
Interpretation: MLE says the probability of heads is 0.7, based purely on the data. If you got no prior beliefs, this is your best guess.
MAP: Combine Data with a Prior Belief¶
MAP also uses the data but adds a “prior belief” about . Let’s assume you think the coin is probably fair () but you’re open to it being slightly biased. We model this with a simple prior called a Beta(2, 2) prior, which favors near 0.5 but allows flexibility.
The Beta(2, 2) prior is like saying, “Before flipping, I believe the coin is fair, as if I’ve seen 2 heads and 2 tails in the past.”
MAP combines:
Data: 7 heads, 3 tails from 10 flips.
Prior: 2 “imaginary” heads, 2 “imaginary” tails.
Add them together:
Total heads = 7 (data) + 2 (prior) = 9
Total flips = 10 (data) + 4 (prior) = 14
MAP estimate:
Interpretation: MAP says the probability of heads is about 0.643. It’s closer to 0.5 than MLE (0.7) because the prior belief in a fair coin “pulls” the estimate toward 0.5.
Key Difference¶
MLE: Only uses the data (7 heads, 10 flips) → .
MAP: Combines data with a prior belief (like 2 heads, 2 tails) → , which is smoothed toward a fair coin.
This smoothing is helpful when you have little data. For example, if you flipped the coin once and got 1 head, MLE would say (100% heads), which is extreme. MAP, with the Beta(2, 2) prior, would give , a more cautious estimate.
Linear Regression Recap¶
Linear regression models a continuous target value:
The cost function is usually Mean Squared Error (MSE).
We use regularization (like L2/Ridge or L1/Lasso) to reduce overfitting.
Motivation for Classification¶
What if our target is not a continuous number, but a class label — like “spam” or “not spam”?
Linear regression doesn’t work well for classification:
It can predict values outside the range.
It doesn’t model probabilities naturally.
Logistic Regression¶
Logistic regression is a binary classification algorithm used to predict the probability that an instance belongs to one of two classes (e.g., 0 or 1). It models the relationship between input features and the probability of a specific outcome using the sigmoid function, which maps any real-valued number to the range . The model is trained by maximizing the likelihood of the observed data, typically using gradient-based optimization.
Key Components¶
Sigmoid Function: Converts a linear combination of features into a probability.
Decision Boundary: Separates the two classes based on a probability threshold (usually 0.5).
Maximum Likelihood Estimation (MLE): Optimizes the model parameters to best fit the training data.
1. Sigmoid Function¶
The sigmoid function is defined as:
where is a linear combination of:
: Input feature vector.
: Weight vector (model parameters).
: Bias term.
Properties¶
Range: , making it ideal for modeling probabilities.
Interpretation: , the probability of class 1 given input .
Complement: .
Why Use in the Sigmoid?¶
The exponential function is chosen because:
Non-linearity: Introduces non-linear behavior, allowing the model to capture complex patterns.
Differentiability: has a simple derivative, facilitating gradient-based optimization.
Convexity: The resulting log-likelihood is convex, ensuring a single global minimum during optimization.
Other bases (e.g., ) could work but lack the mathematical convenience of .
%matplotlib inline
from __future__ import division
import numpy as np
import matplotlib.pyplot as plt
def sigmoid(x):
return 1 / (1 + np.exp(-x))
def grad_sigmoid(x):
return sigmoid(x) * (1 - sigmoid(x))
X = np.arange(-10, 10, 0.1)
fig = plt.figure(figsize=(8, 6))
plt.plot(X, sigmoid(X), label='sigmoid', c='blue', linewidth=3)
plt.plot(X, grad_sigmoid(X), label='derivative of sigmoid', c='green', linewidth=3)
plt.legend(loc='upper left')
plt.xlabel('X')
plt.grid(True)
plt.ylim([-0.5, 1.5])
plt.ylabel('output')
plt.title('Sigmoid')
plt.axhline(y=0, color='k')
plt.axvline(x=0, color='k')
plt.show()
2. Decision Boundary¶
The decision boundary is the set of points where the predicted probability is 0.5:
This equation defines a hyperplane in the feature space:
If , then , and the instance is classified as class 1.
If , then , and the instance is classified as class 0.
3. Model Formulation¶
Logistic regression assumes the class posterior probabilities follow the sigmoid form:
Here, may be raw features or transformed features (e.g., polynomial features).
4. Training with Maximum Likelihood Estimation (MLE)¶
The goal is to find the parameters and that maximize the likelihood of the observed data.
Training Data¶
The training set is:
where:
: Feature vector for the -th instance.
: Target label (0 or 1).
Likelihood Function¶
The likelihood of the data given the parameters is:
Since , we can write:
Thus, the likelihood is:
Basic Logarithm Rules¶
Let be the base ( and ), and let and be positive numbers.
Definition:
Product Rule:
Quotient Rule:
Power Rule:
Change of Base Rule:
where is any valid base ( and ). A common form uses the natural logarithm () or the common logarithm ():
Logarithm of the Base:
Logarithm of One:
Rules Involving and (Natural Logarithm)¶
The natural logarithm () has base .
Definition:
Inverse Relationships:
These rules are fundamental for working with logarithmic and exponential functions.
Examples of Calculating or ¶
Example 1: ¶
Calculation:
Result:
Explanation: Any number raised to the power of 0 is 1.
Example 2: ¶
Calculation:
Result:
Explanation: This is the value of itself, approximated to three decimal places.
Example 3: ¶
Calculation:
Result:
Explanation: Square the value of . Using a calculator, .
Example 4: ¶
Calculation:
Result:
Explanation: A negative exponent means the reciprocal of . So, .
Example 5: ¶
Calculation:
Result:
Explanation: The exponent 0.5 means the square root of . Using a calculator, .
Notes¶
Approximations: The results are rounded to three decimal places for simplicity. In practice, you’d use a calculator or software (e.g., Python’s
math.exp()or NumPy’snp.exp()) for precise values.Context: These calculations are common in machine learning (e.g., softmax function) and other fields like statistics and physics.
How to Compute: If you don’t have a calculator, you can use software like Python:
import math print(math.exp(2)) # Outputs ~7.389
Let me know if you want more examples or help with a specific value!
Log-Likelihood¶
To simplify optimization, we maximize the log-likelihood (since the log is monotonic):
Optimization¶
We maximize (or equivalently, minimize ) using gradient-based methods (e.g., gradient ascent). The gradients are computed with respect to and .
5. Derivation of the Sigmoid Function¶
The sigmoid function arises naturally from modeling the log-odds (logit) as a linear function.
Step-by-Step Derivation¶
Odds: The odds of class 1 are:
Logit: The log-odds (logit) is assumed to be a linear function of the features:
Exponentiate Both Sides:
Solve for :
Let . Then:
This is the sigmoid function: .
6. Gradient of the Sigmoid Function¶
The derivative of the sigmoid function is used in gradient-based optimization.
Derivation¶
Let . Compute the derivative:
Using the chain rule:
The derivative of the inner function is:
So:
Simplify:
Notice that:
And:
Thus:
This derivative is crucial for computing gradients during optimization.
X = np.arange(-5, 5, 0.01)
fig = plt.figure(figsize=(8, 6))
plt.plot(sigmoid(X), -np.log(sigmoid(X)), label='Log(X)', c='blue', linewidth=3)
plt.legend()
plt.xlabel('Sigmoid(X)')
plt.grid(True)
plt.ylabel('Log(Sigmoid(X))')
plt.title('Plot for y = 1')
plt.axhline(y=0, color='k')
plt.axvline(x=0, color='k')
plt.show()
X = np.arange(-10, 10, 0.01)
fig = plt.figure(figsize=(8, 6))
plt.plot(sigmoid(X), -np.log(1 - sigmoid(X)), label='Log(Sigmoid(X))', c='blue', linewidth=3)
plt.legend(loc='upper left')
plt.xlabel('Sigmoid(X)')
plt.grid(True)
plt.ylabel('-Log(1 - Sigmoid(X))')
plt.title('Plot for y = 0')
plt.axhline(y=0, color='k')
plt.axvline(x=0, color='k')
plt.show()
Cost Function and Optimization Objective for Logistic Regression¶
Logistic regression aims to find parameters that best predict the probability of an instance belonging to one of two classes. To achieve this, we define a cost function (or loss function) that measures how well the model’s predictions match the observed data. The cost function is optimized using gradient-based methods to update the model parameters.
Why a Specific Cost Function?¶
Unlike linear regression, which uses mean squared error, logistic regression uses a log-loss (or binary cross-entropy) derived from the negative log-likelihood. This choice is motivated by:
Mathematical Convenience: Logarithms convert products to sums, simplifying differentiation.
Smoothness: The function is smooth and differentiable, aiding optimization.
Numerical Stability: Logarithms handle very small probabilities (e.g., ) effectively.
Convexity: The negative log-likelihood is convex, ensuring a single global minimum and no local optima.
Probabilistic Interpretation: It aligns with maximum likelihood estimation (MLE), directly modeling the likelihood of the data.
1. Cost Function¶
The cost function measures the error for a single prediction. For a given instance with features , true label , and predicted probability , the cost is:
where the sigmoid function is:
Intuition¶
If , we want . If is close to 1, is small (low penalty). If is close to 0, is large (high penalty).
If , we want . If is close to 0, is small. If is close to 1, the penalty is large.
Compact Form¶
The piecewise cost can be combined into a single expression:
When , the equation becomes .
When , it becomes .
This is the binary cross-entropy loss for a single instance.
Full Cost Function¶
For a dataset with instances , the total cost function (average loss) is:
This is the negative log-likelihood averaged over all instances. Minimizing is equivalent to maximizing the likelihood of the observed data.
1. Entropy: Measure of Uncertainty¶
Entropy comes from information theory. It quantifies the uncertainty or impurity in a probability distribution.
Definition:¶
For a binary classification problem with a probability of success (e.g., class 1), the entropy is:
Entropy is 0 when the outcome is certain (i.e., or ).
Entropy is maximum (1 bit) when .
import numpy as np
import matplotlib.pyplot as plt
def entropy(p):
# Clip to avoid log(0)
p = np.clip(p, 1e-10, 1 - 1e-10)
return -p * np.log2(p) - (1 - p) * np.log2(1 - p)
# Plot entropy for p in [0, 1]
p_vals = np.linspace(0.001, 0.999, 100)
entropy_vals = entropy(p_vals)
plt.plot(p_vals, entropy_vals)
plt.xlabel("Probability (p)")
plt.ylabel("Entropy (bits)")
plt.title("Entropy of a Binary Distribution")
plt.grid(True)
plt.show()
2. Log-Loss / Binary Cross-Entropy¶
While entropy measures uncertainty of a distribution, log-loss (or binary cross-entropy) measures the error of a predicted probability compared to the true label .
Formula:¶
For a single prediction:
If , only the term remains.
If , only the term remains.
This is the same formula as entropy, but with as the true class and as the predicted probability.
import numpy as np
import matplotlib.pyplot as plt
def log_loss(y_true, y_pred):
# Clip to avoid log(0)
y_pred = np.clip(y_pred, 1e-10, 1 - 1e-10)
return - (y_true * np.log(y_pred) + (1 - y_true) * np.log(1 - y_pred))
# Example: true label is 1
y_true = 1
y_preds = np.linspace(0.001, 0.999, 100)
losses = [log_loss(y_true, y) for y in y_preds]
plt.plot(y_preds, losses, label=r'True Label = 1')
plt.xlabel(r"Predicted Probability ($\hat{y}$)")
plt.ylabel(r"Log-Loss")
plt.title(r"Binary Cross-Entropy Loss for $y = 1$")
plt.grid(True)
plt.legend()
plt.show()
2. Connection to Maximum Likelihood Estimation (MLE)¶
Likelihood Function¶
For a binary classification dataset, the probability of the observed label given features is:
Assuming independence of instances, the likelihood of the entire dataset is:
Log-Likelihood¶
To simplify optimization, we take the logarithm:
The cost function is the negative log-likelihood scaled by :
Maximizing (likelihood) is equivalent to minimizing (loss).
3. Gradient Descent and Parameter Updates¶
To minimize , we use gradient descent. We need the gradient of with respect to each parameter .
Gradient of the Sigmoid¶
Recall the sigmoid function and its derivative:
For , the partial derivative with respect to is:
Since , we have:
Thus:
Gradient of the Cost Function¶
Now, compute the partial derivative of :
Differentiate with respect to :
For a single term:
Apply the chain rule:
First term:
Second term:
Since , we get:
Combine:
Factor out :
Simplify the expression inside:
Numerator:
So:
Now substitute :
The denominator cancels, leaving:
Thus, the full gradient is:
For gradient descent, we update in the direction of the negative gradient, so:
Gradient Descent Update Rule¶
The update rule for each parameter is:
where is the learning rate. All are updated simultaneously until convergence.
Similarity to Linear Regression¶
This update rule resembles linear regression’s gradient descent, but here, is the sigmoid function, not a linear function.
4. Interpretation of Weights¶
In logistic regression, the model predicts the log-odds:
The weight represents the change in the log-odds for a unit increase in feature , holding other features constant. Exponentiating gives the odds ratio:
This is different from linear regression, where directly represents the change in the output .
5. Maximum A Posteriori (MAP) Estimation and Regularization¶
MAP Estimation¶
MLE maximizes the likelihood . MAP incorporates a prior distribution and maximizes the posterior:
Taking the logarithm:
Gaussian Prior and L2 Regularization¶
Assume a Gaussian prior on : . The log-prior is:
The MAP objective becomes:
Equivalently, minimize:
This is the cross-entropy loss plus an L2 regularization term, which penalizes large weights to prevent overfitting, similar to ridge regression.
1. MAP Estimation: Incorporating Prior Knowledge¶
Maximum Likelihood Estimation (MLE) focuses solely on maximizing the likelihood of the data given the model parameters, i.e., . However, MLE doesn’t account for prior beliefs about the parameters . MAP estimation improves on this by incorporating a prior distribution , which reflects what we believe about the parameters before seeing the data.
The posterior probability of the parameters given the data is given by Bayes’ theorem:
Since is a constant (it doesn’t depend on ), we can maximize the unnormalized posterior:
To make optimization easier, we take the logarithm (since the log is a monotonic function, it preserves the maximum):
Here:
is the log-likelihood, same as in MLE.
is the log-prior, which encodes our prior beliefs about .
The prior acts as a regularizer, guiding the solution toward parameter values that are more likely based on prior knowledge.
2. Gaussian Prior and L2 Regularization¶
A common choice for the prior is a Gaussian distribution centered at zero, . This assumes that the parameters are likely to be small (close to zero), which encourages simpler models and helps prevent overfitting.
The probability density of a Gaussian prior is:
Taking the logarithm:
Now, substitute this into the MAP objective:
Let’s define , where controls the strength of the prior (smaller means a stronger belief that should be close to zero). The objective becomes:
where is the log-likelihood.
To turn this into a minimization problem (common in optimization), we negate the objective and scale the log-likelihood by (where is the number of data points) for numerical stability:
This is the loss function (e.g., cross-entropy loss for classification) plus an L2 regularization term . The L2 term penalizes large values of , encouraging smaller weights, which leads to simpler models and reduces overfitting. This is mathematically equivalent to ridge regression in linear regression.
3. Why L2 Regularization Works¶
Preventing Overfitting: Large weights can lead to overly complex models that fit noise in the training data. The L2 penalty discourages large weights, favoring smoother, more generalizable models.
Connection to Gaussian Prior: The L2 term arises naturally from the Gaussian prior, which assumes parameters are likely to be small. The strength of regularization () is inversely proportional to the variance of the prior ().
Geometric Interpretation: The L2 term constrains the solution to lie closer to the origin in parameter space, balancing the trade-off between fitting the data (likelihood) and keeping parameters small (prior).
Small Example: Logistic Regression with MAP and L2 Regularization¶
Let’s apply MAP estimation with a Gaussian prior to a logistic regression problem.
Problem Setup¶
Suppose we have a binary classification dataset with data points:
, .
Model parameters: .
We assume a Gaussian prior: , with , so .
Step 1: Log-Likelihood for Logistic Regression¶
For logistic regression, the likelihood is:
where is the sigmoid function.
The log-likelihood is:
For our data:
Point 1: , , so .
Point 2: , , so .
The log-likelihood is:
Step 2: Gaussian Prior¶
The log-prior with is:
Step 3: MAP Objective¶
The MAP objective is:
Equivalently, minimize the loss function (with ):
Step 4: Interpretation¶
The term is the average negative log-likelihood (cross-entropy loss).
The term is the L2 regularization term with .
To find , we’d typically use gradient descent or another optimization method to minimize .
Step 5: Numerical Intuition¶
Suppose we test :
Compute , so .
Compute , so .
Log-likelihood: .
L2 term: .
Loss: .
The optimizer would adjust to minimize , balancing the fit to the data (log-likelihood) and the penalty on large weights (L2 term).
6. MLE and MAP in Context¶
Maximum Likelihood Estimation (MLE)¶
Goal: Find that maximizes the likelihood .
Process: Define the likelihood, take the log, and optimize.
Example: For a coin with heads in flips, the likelihood is . The MLE is .
Maximum A Posteriori (MAP)¶
Goal: Incorporate prior beliefs to maximize the posterior.
Process: Combine log-likelihood and log-prior, then optimize.
Example: If you believe a coin is fair (prior ), MAP adjusts the estimate based on both data and prior.
MLE vs. MAP¶
MLE: Data-driven, no assumptions about .
MAP: Incorporates prior knowledge, leading to regularization in practice.
7. Example: Biased Coin¶
Consider a dataset of coin flips: (3 heads, 2 tails).
Model¶
Likelihood¶
Log-Likelihood¶
MLE Solution¶
Differentiate and set to zero:
The MLE estimate is , matching the proportion of heads.
8. Conditional Maximum Likelihood¶
Logistic regression models the conditional probability . The log-likelihood is:
For logistic regression:
Maximizing pushes predictions toward 1 for and 0 for .
# our dataset is {H, H, T, H, T}; if theta = P(x=H), we get:
coin_likelihood = lambda theta: theta*theta*(1-theta)*theta*(1-theta)
theta_vals = np.linspace(0,1)
plt.ylabel('Likelihood of the Data'); plt.xlabel(r'Parameter $\theta$')
plt.scatter([0.6], [coin_likelihood(0.6)])
plt.plot(theta_vals, coin_likelihood(theta_vals));
def log_likelihood(theta, X, y):
"""The cost function, J(theta0, theta1) describing the goodness of fit.
We added the 1e-6 term in order to avoid overflow (inf and -inf).
Parameters:
theta (np.array): d-dimensional vector of parameters
X (np.array): (n,d)-dimensional design matrix
y (np.array): n-dimensional vector of targets
"""
return (y*np.log(f(X, theta) + 1e-6) + (1-y)*np.log(1-f(X, theta) + 1e-6)).mean()def cross_entropy(yHat, y):
if y == 1:
return -np.log(yHat)
else:
return -np.log(1 - yHat)
fig, ax = plt.subplots(figsize=(6*fig_scale,2*fig_scale))
x = np.linspace(0,1,100)
ax.plot(x,cross_entropy(x, 1),lw=2*fig_scale,c='b',label='true label = 1', linestyle='-')
ax.plot(x,cross_entropy(x, 0),lw=2*fig_scale,c='r',label='true label = 0', linestyle='-')
ax.set_xlabel(r"Predicted probability $\hat{y}$")
ax.set_ylabel("Log loss")
plt.grid()
plt.legend();Logistic Regression from Scratch¶
Below is a Python implementation of logistic regression using NumPy, including gradient descent optimization and L2 regularization.
Explanation of Code:¶
• Sigmoid Function: Implements (\sigma(z) = \frac{1}{1 + e^{-z}}).
• Loss Function: Computes binary cross-entropy plus L2 regularization.
• Gradient Descent: Updates weights and bias using the gradients of the loss.
• Prediction: Outputs probabilities (via predict_proba) or class labels (via predict).
• Example: Generates synthetic 2D data, trains the model, and plots the loss curve.Notes:¶
• The implementation includes L2 regularization (controlled by lambda_reg), which corresponds to the Gaussian prior in MAP estimation.
• A small constant (1e-15) is added to logarithms to avoid numerical issues with (\log(0)).
• The synthetic dataset is linearly separable for simplicity, but the model can handle non-separable data.from __future__ import division
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from scipy.optimize import minimize
df = pd.read_csv('ex2data1.csv', header=None)
df.rename(columns={0: 'exam1', 1: 'exam2', 2: 'y'}, inplace=True)
df.head()fig = plt.figure(figsize=(12, 8))
plt.scatter(df[df['y'] == 0]['exam1'], df[df['y'] == 0]['exam2'],
label='Not admitted', color='yellow', edgecolor='black')
plt.scatter(df[df['y'] == 1]['exam1'], df[df['y'] == 1]['exam2'],
label='Admitted', marker='+', color='black')
plt.xlabel('Exam 1 score')
plt.ylabel('Exam 2 score')
plt.legend(loc='upper right')
plt.title('Scores indicating admission')
plt.show()
import numpy as np
class LogisticRegression:
def __init__(self, learning_rate=0.01, n_iterations=1000, lambda_reg=0.01):
self.learning_rate = learning_rate
self.n_iterations = n_iterations
self.lambda_reg = lambda_reg # Regularization strength
self.weights = None
self.bias = None
self.losses = []
def sigmoid(self, z):
return 1 / (1 + np.exp(-z))
def compute_loss(self, X, y, y_pred):
# Binary cross-entropy loss + L2 regularization
N = len(y)
cross_entropy = -np.mean(y * np.log(y_pred + 1e-15) + (1 - y) * np.log(1 - y_pred + 1e-15))
l2_penalty = (self.lambda_reg / (2 * N)) * np.sum(self.weights ** 2)
return cross_entropy + l2_penalty
def fit(self, X, y):
n_samples, n_features = X.shape
# Initialize weights and bias
self.weights = np.zeros(n_features)
self.bias = 0
# Gradient descent
for _ in range(self.n_iterations):
# Forward pass
linear_model = np.dot(X, self.weights) + self.bias
y_pred = self.sigmoid(linear_model)
# Compute loss
loss = self.compute_loss(X, y, y_pred)
self.losses.append(loss)
# Compute gradients
dw = (1 / n_samples) * np.dot(X.T, (y_pred - y)) + (self.lambda_reg / n_samples) * self.weights
db = (1 / n_samples) * np.sum(y_pred - y)
# Update parameters
self.weights -= self.learning_rate * dw
self.bias -= self.learning_rate * db
def predict_proba(self, X):
linear_model = np.dot(X, self.weights) + self.bias
return self.sigmoid(linear_model)
def predict(self, X, threshold=0.5):
return (self.predict_proba(X) >= threshold).astype(int)
# Example usage
if __name__ == "__main__":
# Generate synthetic data
np.random.seed(0)
X = np.random.randn(100, 2)
y = (X[:, 0] + X[:, 1] > 0).astype(int)
# Train model
model = LogisticRegression(learning_rate=0.1, n_iterations=1000, lambda_reg=0.1)
model.fit(X, y)
# Predict
y_pred = model.predict(X)
accuracy = np.mean(y_pred == y)
print(f"Accuracy: {accuracy:.4f}")
# Plot loss curve
import matplotlib.pyplot as plt
plt.plot(model.losses)
plt.xlabel("Iteration")
plt.ylabel("Loss")
plt.title("Loss Curve")
plt.show()Accuracy: 0.9900
Softmax Regression Explained¶
Softmax regression (also known as multinomial logistic regression) is a multi-class classification algorithm that generalizes logistic regression to handle more than two classes. It predicts the probability that an input belongs to each of possible classes, making it suitable for problems like classifying images into multiple categories (e.g., digits 0–9 in MNIST).
Key Components¶
Score Computation: Computes a score for each class based on a linear combination of input features and class-specific parameters.
Softmax Function: Converts scores into probabilities that sum to 1, representing the likelihood of each class.
Optimization: Trains the model by maximizing the likelihood of the observed data using a loss function (negative log-likelihood).
Despite the name “regression,” softmax regression is used for classification, as it outputs class probabilities.
1. Model Formulation¶
Softmax regression maps an input to a probability distribution over classes, producing a vector of probabilities:
Parameters¶
Parameters: , where each is the parameter vector for class .
Parameter Space: , a matrix where each row corresponds to a class.
Input: , the feature vector (often includes a bias term by appending a 1).
Step 1: Score Computation¶
For each class , compute a score (or logit):
The vector of scores is:
Step 2: Softmax Function¶
The scores are converted to probabilities using the softmax function, which ensures the outputs are positive and sum to 1:
For a given input , the probability of class is:
Intuition¶
Exponentiation: ensures all scores are positive, emphasizing larger scores.
Normalization: Dividing by the sum makes the probabilities sum to 1.
Competition: Classes with higher scores get higher probabilities, but the softmax ensures a smooth distribution across all classes.
Identifiability Note¶
The model is over-parametrized. Adding a constant vector to all (i.e., ) doesn’t change the probabilities because:
The terms cancel out in the softmax denominator. To address this, we can fix one parameter vector (e.g., ) without loss of generality, reducing the parameter space.
Deeper Explanation of the Softmax Function¶
1. What is the Softmax Function?¶
The softmax function is a mathematical operation used to transform a vector of real-valued scores (often called logits) into a probability distribution. It is widely used in multi-class classification problems, where the goal is to assign an input to one of possible classes.
Given a vector of scores , the softmax function computes probabilities for each class as follows:
Here:
: The exponential function ensures that the output for each class is positive, as for any real .
: The denominator normalizes the outputs by summing the exponentials of all scores, ensuring that the resulting probabilities sum to 1:
The output represents the probability of class , and the vector is a valid probability distribution.
2. Why Use the Softmax Function?¶
The softmax function is ideal for multi-class classification because:
Positivity: The exponential ensures all outputs are positive, which is necessary for probabilities.
Normalization: The denominator ensures the outputs sum to 1, making them interpretable as probabilities.
Amplifies Differences: The exponential function exaggerates differences between scores. A larger leads to a much larger , assigning higher probability to the corresponding class.
Differentiability: The softmax function is smooth and differentiable, which is crucial for optimization in machine learning (e.g., gradient-based methods like gradient descent).
3. Softmax in Multi-Class Classification¶
In a multi-class classification model (e.g., logistic regression for multiple classes or a neural network), the model computes a score for each class based on the input and model parameters . For class , the score (or logit) is often computed as a linear combination:
where:
is the input feature vector.
is the parameter vector (e.g., weights) for class .
is the collection of parameters for all classes.
The scores are then passed through the softmax function to obtain the probability of each class:
This probability represents the model’s confidence that the input belongs to class . The predicted class is typically the one with the highest probability:
4. Properties of Softmax¶
Invariance to Constant Shifts: Adding a constant to all scores doesn’t change the output probabilities, because:
This property is useful numerically, as subtracting the maximum score (i.e., ) from all scores can prevent overflow in computations:
Sensitivity to Score Differences: The exponential amplifies differences between scores. If is much larger than the others, will be close to 1, and the others will be close to 0.
Small Example: Applying the Softmax Function¶
Let’s walk through a concrete example to illustrate how the softmax function works in a multi-class classification setting.
Problem Setup¶
Suppose we have a classification problem with classes (e.g., classifying an image as a cat, dog, or bird). For a given input , a model computes scores for each class based on parameters . Let’s assume:
Input: (e.g., features extracted from an image).
Parameters for each class:
Class 1 (cat):
Class 2 (dog):
Class 3 (bird):
Step 1: Compute Scores (Logits)¶
Calculate the score for each class using :
Class 1:
Class 2:
Class 3:
So, the score vector is:
Step 2: Apply the Softmax Function¶
Compute the softmax probabilities:
First, calculate the exponentials:
Sum the exponentials:
Now compute the probabilities:
Class 1:
Class 2:
Class 3:
The probability distribution is:
Step 3: Interpretation¶
The model assigns the highest probability (57.4%) to the “cat” class, so the predicted class is “cat”:
The probabilities sum to 1: .
The exponential in the softmax amplifies the effect of the highest score (), giving the “cat” class a significantly higher probability than the others.
Step 4: Numerical Stability (Optional)¶
To avoid numerical overflow (e.g., if scores are very large), we can subtract the maximum score () from all scores:
Adjusted scores:
Exponentials: , ,
Sum:
Probabilities:
Class 1:
Class 2:
Class 3:
The results are identical, but the computation is more stable.
2. Probabilistic Interpretation¶
Softmax regression models the conditional probability of the class label given the input :
This defines a categorical distribution over the classes. The model assigns higher probabilities to classes whose parameter vectors are more aligned with the input (i.e., larger ).
Decision Rule¶
To classify an input, choose the class with the highest probability:
Since the softmax is monotonic, this is equivalent to picking the class with the highest score .
3. Learning Objective: Maximum Likelihood Estimation (MLE)¶
The goal is to find the parameters that maximize the likelihood of the observed data.
Dataset¶
Consider a dataset:
where:
: Feature vector for the -th instance.
: True class label.
Likelihood Function¶
The probability of observing label given is:
Assuming the instances are independent, the likelihood is:
Log-Likelihood¶
To simplify optimization, take the logarithm:
Substitute the probability:
Cost Function¶
In practice, we minimize the negative log-likelihood (cross-entropy loss), averaged over the dataset:
This is the multinomial cross-entropy loss. Minimizing is equivalent to maximizing the likelihood.
4. Gradient of the Cost Function¶
To optimize using gradient descent, we need the gradient with respect to for each class .
Gradient Derivation¶
The cost function is:
Define the loss for a single instance :
We compute the partial derivative with respect to (parameters for class ):
For a single instance, differentiate:
First term:
This is non-zero only if (since only involves class ):
Using an indicator function, this is:
Second term:
Let . Then:
Compute:
So:
Since , the derivative of the second term is:
Combine:
The full gradient is:
Intuition¶
If , the term , and the gradient pushes toward 1.
If , the gradient pushes toward 0.
The difference measures the error in the predicted probability.
Gradient Descent Update¶
Update each using:
where is the learning rate.
5. Connection to Logistic Regression¶
Softmax regression generalizes logistic regression:
For , softmax regression reduces to logistic regression.
Let class 1 have probability . Set , then:
This matches the logistic regression sigmoid function.
is same as
starting with:
Then you performed the following algebraic manipulations:
Divide numerator and denominator by :
Introduce : Notice that . Substituting this, you get:
Recognize the sigmoid function: You correctly identified this as the sigmoid function , where in this case, .
So, the format in this context is:
With , the probability can be expressed as:
Alternatively, if you wanted the numerator to explicitly have , you could multiply the numerator and denominator of the original expression by :
Can't use function '$' in math mode at position 155: …^\top \vec{x})}$̲$$$p = \frac{e(…
p = \frac{e(\theta_1^\top \vec{x}) \cdot e(-\theta_2^\top \vec{x})}{(e(\theta_1^\top \vec{x}) + e(\theta_2^\top \vec{x})) \cdot e(-\theta_2^\top \vec{x})}$$$$p = \frac{e(\theta_1^\top \vec{x} - \theta_2^\top \vec{x})}{e(\theta_1^\top \vec{x} - \theta_2^\top \vec{x}) + e(\theta_2^\top \vec{x} - \theta_2^\top \vec{x})}$$$$p = \frac{e((\theta_1 - \theta_2)^\top \vec{x})}{e((\theta_1 - \theta_2)^\top \vec{x}) + e(0)}If we still define , then this form is:
Both and are equivalent forms of the sigmoid function and represent the probability in terms of . Your derivation correctly arrived at the first form. The second form is just another way to express the same relationship.
6. Implementation¶
Softmax regression can be implemented using scikit-learn’s LogisticRegression with the multi_class='multinomial' option:
from sklearn.linear_model import LogisticRegression
model = LogisticRegression(multi_class='multinomial', solver='lbfgs')
model.fit(X_train, y_train)
y_pred = model.predict(X_test)solver=‘lbfgs’: Uses a second-order optimization method for efficiency.
The model automatically handles the softmax function and optimizes the cross-entropy loss.
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
# 1. Generate synthetic business-style 3-class data (e.g., low/medium/high value customers)
X, Y = make_classification(
n_samples=500,
n_features=2,
n_redundant=0,
n_clusters_per_class=1,
n_classes=3,
class_sep=2.0,
random_state=42
)
# 2. Scale features (for stability and visualization)
scaler = StandardScaler()
X = scaler.fit_transform(X)
# 3. Train multinomial logistic regression (softmax)
logreg = LogisticRegression(C=1e5, multi_class='multinomial', solver='lbfgs')
logreg.fit(X, Y)
# 4. Meshgrid for decision boundary plot
xx, yy = np.meshgrid(
np.arange(X[:, 0].min() - 0.5, X[:, 0].max() + 0.5, 0.02),
np.arange(X[:, 1].min() - 0.5, X[:, 1].max() + 0.5, 0.02)
)
Z = logreg.predict(np.c_[xx.ravel(), yy.ravel()])
Z = Z.reshape(xx.shape)
# 5. Plot
plt.figure(figsize=(10, 6))
plt.pcolormesh(xx, yy, Z, cmap=plt.cm.Paired, shading='auto')
plt.scatter(X[:, 0], X[:, 1], c=Y, edgecolors='k', cmap=plt.cm.Paired, s=50)
plt.xlabel('Customer Feature 1 (scaled)')
plt.ylabel('Customer Feature 2 (scaled)')
plt.title('Multinomial Logistic Regression (3-Class Business Example)')
plt.grid(True)
plt.show()
/Users/chandraveshchaudhari/anaconda3/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1247: FutureWarning: 'multi_class' was deprecated in version 1.5 and will be removed in 1.7. From then on, it will always use 'multinomial'. Leave it to its default value to avoid this warning.
warnings.warn(
import numpy as np
from scipy.special import softmax
from sklearn.preprocessing import LabelBinarizer
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
class SoftmaxRegression:
def __init__(self, learning_rate=0.01, max_iter=1000, tol=1e-4, random_state=None):
self.learning_rate = learning_rate
self.max_iter = max_iter
self.tol = tol
self.random_state = random_state
self.weights = None
self.classes_ = None
self.bias = None # Separate bias term
def _initialize_weights(self, n_features, n_classes):
if self.random_state is not None:
np.random.seed(self.random_state)
# Initialize weights and bias separately
self.weights = np.random.randn(n_features, n_classes) * 0.01
self.bias = np.zeros(n_classes)
def _one_hot_encode(self, y):
self.label_binarizer = LabelBinarizer()
return self.label_binarizer.fit_transform(y)
def _gradient(self, X, y_true, y_pred):
# Gradient of cross-entropy loss
error = y_pred - y_true
grad_weights = X.T @ error / X.shape[0]
grad_bias = np.mean(error, axis=0)
return grad_weights, grad_bias
def fit(self, X, y):
# Convert labels to one-hot encoding
y_one_hot = self._one_hot_encode(y)
self.classes_ = self.label_binarizer.classes_
n_samples, n_features = X.shape
n_classes = y_one_hot.shape[1]
# Initialize weights and bias
self._initialize_weights(n_features, n_classes)
prev_loss = float('inf')
# Gradient descent
for i in range(self.max_iter):
# Forward pass: compute predictions
logits = X @ self.weights + self.bias # Add bias here
y_pred = softmax(logits, axis=1)
# Compute loss (cross-entropy)
loss = -np.mean(np.sum(y_one_hot * np.log(y_pred + 1e-15), axis=1))
# Check for convergence
if np.abs(prev_loss - loss) < self.tol:
print(f"Converged at iteration {i}")
break
prev_loss = loss
# Backward pass: compute gradients and update parameters
grad_weights, grad_bias = self._gradient(X, y_one_hot, y_pred)
self.weights -= self.learning_rate * grad_weights
self.bias -= self.learning_rate * grad_bias
if i % 100 == 0:
print(f"Iteration {i}, Loss: {loss:.4f}")
def predict_proba(self, X):
logits = X @ self.weights + self.bias
return softmax(logits, axis=1)
def predict(self, X):
proba = self.predict_proba(X)
return self.classes_[np.argmax(proba, axis=1)]
# Example usage
if __name__ == "__main__":
# Load and prepare data
iris = load_iris()
X = iris.data
y = iris.target
# Split data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Standardize features
mean = X_train.mean(axis=0)
std = X_train.std(axis=0)
X_train = (X_train - mean) / std
X_test = (X_test - mean) / std
# Train model
model = SoftmaxRegression(learning_rate=0.1, max_iter=1000, random_state=42)
model.fit(X_train, y_train)
# Evaluate
y_pred = model.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
print(f"\nTest Accuracy: {accuracy:.4f}")
# Show predictions
print("\nSample predictions:")
print("True:", y_test[:5])
print("Pred:", y_pred[:5])Iteration 0, Loss: 1.1053
Iteration 100, Loss: 0.3389
Iteration 200, Loss: 0.2699
Iteration 300, Loss: 0.2291
Iteration 400, Loss: 0.2013
Iteration 500, Loss: 0.1810
Iteration 600, Loss: 0.1657
Iteration 700, Loss: 0.1536
Converged at iteration 737
Test Accuracy: 1.0000
Sample predictions:
True: [1 0 2 1 1]
Pred: [1 0 2 1 1]
import matplotlib.pyplot as plt
from matplotlib.colors import ListedColormap
import numpy as np
def plot_decision_regions(X, y, model, resolution=0.02):
# Setup marker generator and color map
markers = ('s', 'x', 'o', '^', 'v')
colors = ('red', 'blue', 'lightgreen', 'gray', 'cyan')
cmap = ListedColormap(colors[:len(np.unique(y))])
# Plot the decision surface
x1_min, x1_max = X[:, 0].min() - 1, X[:, 0].max() + 1
x2_min, x2_max = X[:, 1].min() - 1, X[:, 1].max() + 1
xx1, xx2 = np.meshgrid(np.arange(x1_min, x1_max, resolution),
np.arange(x2_min, x2_max, resolution))
# Predict for each point in meshgrid
Z = model.predict(np.array([xx1.ravel(), xx2.ravel()]).T)
Z = Z.reshape(xx1.shape)
plt.contourf(xx1, xx2, Z, alpha=0.4, cmap=cmap)
plt.xlim(xx1.min(), xx1.max())
plt.ylim(xx2.min(), xx2.max())
# Plot class samples
for idx, cl in enumerate(np.unique(y)):
plt.scatter(x=X[y == cl, 0],
y=X[y == cl, 1],
alpha=0.6,
c=cmap(idx),
edgecolor='black',
marker=markers[idx],
label=cl)
# Select two features for visualization (sepal length and width)
X_vis = X_train[:, :2] # Using first two features for visualization
# Retrain model on just these two features
model_vis = SoftmaxRegression(learning_rate=0.01, max_iter=10000, random_state=42)
model_vis.fit(X_vis, y_train)
# Create plot
plt.figure(figsize=(10, 6))
plot_decision_regions(X_vis, y_train, model_vis)
plt.xlabel('Sepal length (standardized)')
plt.ylabel('Sepal width (standardized)')
plt.title('Softmax Regression Decision Boundaries on Iris Dataset')
plt.legend(loc='upper left')
plt.show()
# Also plot the actual test results
plt.figure(figsize=(10, 6))
plot_decision_regions(X_test[:, :2], y_test, model_vis)
plt.xlabel('Sepal length (standardized)')
plt.ylabel('Sepal width (standardized)')
plt.title('Test Set Classification Results')
plt.legend(loc='upper left')
plt.show()Iteration 0, Loss: 1.0934
Iteration 100, Loss: 0.8577
Iteration 200, Loss: 0.7380
Iteration 300, Loss: 0.6699
Iteration 400, Loss: 0.6268
Iteration 500, Loss: 0.5973
Iteration 600, Loss: 0.5759
Iteration 700, Loss: 0.5595
Iteration 800, Loss: 0.5466
Converged at iteration 876
/var/folders/93/7lt42x5j7m39kz7wxbcghvrm0000gn/T/ipykernel_9647/1777411610.py:27: UserWarning: *c* argument looks like a single numeric RGB or RGBA sequence, which should be avoided as value-mapping will have precedence in case its length matches with *x* & *y*. Please use the *color* keyword-argument or provide a 2D array with a single row if you intend to specify the same RGB or RGBA value for all points.
plt.scatter(x=X[y == cl, 0],
/var/folders/93/7lt42x5j7m39kz7wxbcghvrm0000gn/T/ipykernel_9647/1777411610.py:27: UserWarning: You passed a edgecolor/edgecolors ('black') for an unfilled marker ('x'). Matplotlib is ignoring the edgecolor in favor of the facecolor. This behavior may change in the future.
plt.scatter(x=X[y == cl, 0],
Information-Theoretic View: Entropy & KL Divergence in Classification¶
In the context of classification, an information-theoretic perspective provides a framework to understand and quantify uncertainty, information content, and divergence between probability distributions. Two key concepts in this view are entropy and Kullback-Leibler (KL) divergence. Below, we explore these concepts and their relevance to classification tasks.
1. Entropy in Classification¶
Entropy is a measure of uncertainty or randomness in a probability distribution. In classification, it quantifies the uncertainty associated with predicting the class label of a data point.
Definition: For a discrete random variable representing class labels with probability distribution , the entropy is defined as:
where:
is the number of classes.
is the probability of class .
The logarithm is typically base 2 (for bits) or base (for nats).
Interpretation:
High entropy: The class distribution is uniform (e.g., ), indicating maximum uncertainty. For example, in a binary classification problem with and , the entropy is bit.
Low entropy: The distribution is skewed toward one class (e.g., ), indicating low uncertainty.
Zero entropy: The outcome is certain (e.g., ).
Role in Classification:
Decision Trees: Entropy is used in algorithms like ID3 or C4.5 to measure the impurity of a node. The goal is to split the data to minimize entropy (i.e., reduce uncertainty about class labels).
Model Evaluation: Entropy can help assess the uncertainty in a model’s predictions. For example, a model outputting high-entropy (uniform) probability distributions for test samples may indicate poor confidence.
Cross-Entropy Loss: In training classifiers (e.g., neural networks), the cross-entropy loss measures the difference between the true label distribution and the predicted distribution, effectively penalizing high uncertainty in incorrect predictions.
2. KL Divergence in Classification¶
Kullback-Leibler (KL) divergence measures how much one probability distribution differs from another. In classification, it is often used to compare the predicted probability distribution to the true distribution.
Definition: For two probability distributions (true distribution) and (approximated/predicted distribution) over the same set of class labels, the KL divergence is:
, with equality if and only if .
It is not symmetric: .
Interpretation:
KL divergence quantifies the “extra information” (in bits or nats) needed to encode samples from using a code optimized for .
A small KL divergence indicates that is a good approximation of .
A large KL divergence suggests that the predicted distribution deviates significantly from the true distribution .
Role in Classification:
Cross-Entropy Loss and KL Divergence: The cross-entropy loss used in classification can be decomposed as:
where:
is the cross-entropy.
is the entropy of the true distribution.
is the KL divergence. Since is constant for a given true distribution, minimizing the cross-entropy loss is equivalent to minimizing the KL divergence between the true and predicted distributions.
Model Calibration: KL divergence can be used to evaluate how well a model’s predicted probabilities align with the true class distribution. A high KL divergence may indicate miscalibration.
Regularization: In some models, KL divergence is used as a regularization term. For example, in variational inference or Bayesian neural networks, KL divergence measures the difference between the learned parameter distribution and a prior distribution.
Domain Adaptation: KL divergence can quantify the difference between source and target domain distributions, helping to align feature distributions in transfer learning or domain adaptation tasks.
3. Practical Implications in Classification¶
Entropy and Decision Making:
Entropy guides feature selection in decision trees by identifying splits that reduce class uncertainty.
In ensemble methods (e.g., random forests), entropy can help assess the diversity of predictions across trees.
In active learning, entropy is used to select uncertain samples for labeling, maximizing information gain.
KL Divergence and Model Optimization:
During training, minimizing the cross-entropy loss implicitly minimizes the KL divergence, aligning the predicted probabilities with the true labels.
In generative models (e.g., GANs or VAEs) used for classification, KL divergence may appear in the objective function to ensure the generated distribution matches the true data distribution.
In knowledge distillation, KL divergence is used to transfer knowledge from a teacher model (with a “soft” probability distribution) to a student model.
Evaluation Metrics:
While entropy and KL divergence are not typically used directly as evaluation metrics, they underpin metrics like log-loss (cross-entropy) and can inform diagnostic tools for model performance.
For example, a model with high cross-entropy loss may have a large KL divergence, indicating poor alignment with the true distribution.
4. Example: Binary Classification¶
Consider a binary classification problem with true labels , and a model’s predicted probabilities .
Entropy of True Distribution:
Cross-Entropy:
KL Divergence:
The KL divergence is small, indicating that the predicted distribution is close to the true distribution .
5. Limitations and Considerations¶
Entropy:
Entropy assumes a discrete distribution. For continuous variables, differential entropy is used, but it has different properties (e.g., it can be negative).
Entropy is sensitive to the number of classes; more classes generally increase entropy unless the distribution is highly skewed.
KL Divergence:
KL divergence is asymmetric and not a true distance metric.
It can be undefined if for any where . Smoothing techniques (e.g., adding a small to probabilities) are often used to avoid this.
KL divergence is sensitive to small differences in low-probability events, which may or may not be desirable depending on the application.
Computational Considerations:
Estimating entropy and KL divergence requires reliable probability estimates, which can be challenging with limited data or poorly calibrated models.
In high-dimensional settings, approximating these quantities may require techniques like Monte Carlo sampling or kernel density estimation.
6. Broader Context¶
Mutual Information: Entropy is closely related to mutual information, which measures the reduction in uncertainty about one variable (e.g., class labels) given another (e.g., features). Mutual information can guide feature selection in classification.
Information Bottleneck: This framework uses entropy and KL divergence to balance the trade-off between compressing input features and preserving information about the class labels.
Robustness and Uncertainty: Information-theoretic measures can help quantify model robustness to adversarial attacks or distributional shifts by analyzing changes in entropy or KL divergence under perturbations.
Summary¶
Entropy quantifies the uncertainty in a class distribution, guiding decision-making in algorithms like decision trees and informing loss functions like cross-entropy.
KL Divergence measures the difference between true and predicted distributions, playing a central role in optimizing classifiers and evaluating model calibration.
Together, these concepts provide a principled way to understand, train, and evaluate classification models, ensuring that predictions align with the true underlying distribution while minimizing uncertainty.
1. Multinomial Cross-Entropy Loss (Multi-Class)¶
The provided cost function is the multinomial cross-entropy loss, used for multi-class classification with classes. It is derived from the negative log-likelihood of the data under a multinomial logistic regression model (softmax classifier).
Formula¶
For a dataset with examples, the cost function is:
Explanation¶
Input: For the -th example, is the feature vector, is the true class label, and is the parameter vector for class .
Probability: The model predicts the probability of class using the softmax function:
Log-Likelihood: The log-likelihood for the -th example is the log of the probability of the true class :
Cost Function: The negative average log-likelihood over examples gives , which we minimize to find the optimal parameters .
Key Features¶
Applies to classes.
Uses the softmax function to compute probabilities across all classes.
The loss penalizes the model when the predicted probability for the true class is low.
2. Binary Cross-Entropy Loss (Two Classes)¶
For binary classification (2 classes, typically labeled ), the binary cross-entropy loss (also called log loss) is used. It’s a special case of the multinomial cross-entropy loss when , but it’s often written in a simpler form using the sigmoid function.
Formula¶
For a dataset with examples, the binary cross-entropy loss is:
where:
is the predicted probability of class 1, and is the sigmoid function.
is the true label for the -th example.
is the parameter vector (single set of weights, unlike the multi-class case).
Explanation¶
Probability: The model predicts the probability of class 1 as:
The probability of class 0 is:
Log-Likelihood: The log-likelihood for the -th example is:
Cost Function: The negative average log-likelihood gives , which is minimized to optimize .
Key Features¶
Applies to exactly 2 classes ( or ).
Uses the sigmoid function to compute the probability of class 1.
The loss penalizes incorrect predictions by assigning higher loss when the predicted probability is far from the true label .
3. Comparison¶
| Aspect | Multinomial Cross-Entropy (Multi-Class) | Binary Cross-Entropy (Two Classes) |
|---|---|---|
| Number of Classes | (general case, including 2 or more classes) | Exactly 2 classes () |
| Probability Model | Softmax: $P(y = k | \vec{x}, \theta) = \frac{e(\theta_k^\top \vec{x})}{\sum_{l=1}^K e(\theta_l^\top \vec{x})}$ |
| Parameters | parameter vectors: (one per class) | Single parameter vector: |
| Loss Formula | ||
| Special Case | Reduces to binary cross-entropy when (see below) | Is a special case of multinomial cross-entropy when |
| Interpretation | Measures how well the predicted probability distribution matches the true class across classes | Measures how well the predicted probability for class 1 (or 0) matches the true binary label |
Key Insight: Binary Case as a Special Case¶
When in the multinomial case, the softmax model can be shown to be equivalent to the sigmoid-based binary model. Let’s derive this briefly:
For , classes are (or relabeled as ). The softmax probabilities are:
Divide numerator and denominator by :
Let . This matches the sigmoid form: .
The multinomial loss for simplifies to the binary cross-entropy loss, confirming they are equivalent up to a reparameterization.
4. Small Examples¶
Example 1: Binary Cross-Entropy (2 Classes)¶
Setup:
Dataset: examples.
Example 1: , (class 1).
Example 2: , (class 0).
Parameters: .
Calculation:
For example 1: , .
Loss term: .
For example 2: , .
Loss term: .
Total loss: .
Example 2: Multinomial Cross-Entropy (3 Classes)¶
Setup:
Dataset: examples, classes.
Example 1: , (class 2).
Example 2: , (class 1).
Parameters: , , .
Calculation:
For example 1: Scores are , , .
Sum of exponentials: , , , total .
Loss term: .
For example 2: Scores are , , .
Sum of exponentials: , , , total .
Loss term: .
Total loss: .
# Your code here