Perceptron & MLP

Perceptron & MLP#

“The neuron that started it all — before ChatGPT was even a glimmer in a GPU.”

🧠 What’s Happening Here?#

Welcome to the origin story of neural networks — the perceptron. It’s the humble little unit that decided:

“What if math could make decisions and feel cool about it?”

Back in the 1950s, Frank Rosenblatt introduced the perceptron, hoping to make machines that “see” and “think.” He succeeded… until someone asked it to solve XOR. It failed miserably, funding got cut, and AI went into a winter. 🥶

Fast-forward 60 years — we added layers, non-linearities, and GPUs, and now the same idea powers everything from fraud detection to TikTok recommendations. 🤯

🧩 Concept in 10 Seconds#

A perceptron is a simple model that:

  • Takes inputs ( x_1, x_2, …, x_n )

  • Multiplies them by weights ( w_1, w_2, …, w_n )

  • Adds a bias ( b )

  • Passes the result through an activation function

Mathematically:

[ y = \sigma(w_1x_1 + w_2x_2 + \dots + w_nx_n + b) ]

Where ( \sigma ) is an activation (like ReLU or sigmoid).

🧮 Quick PyTorch Demo: One Neuron to Rule Them All#

Let’s start ridiculously small — a single neuron deciding whether a transaction is fraudulent or not (based only on “how suspicious it looks,” of course).

import torch
import torch.nn as nn

# One data point = 3 features (e.g., transaction amount, time gap, country risk)
x = torch.tensor([[0.2, 0.7, 0.5]])
y_true = torch.tensor([[1.0]])  # 1 = fraud

# A single neuron
neuron = nn.Linear(3, 1)
activation = nn.Sigmoid()

# Forward pass
y_pred = activation(neuron(x))
print(y_pred)

🎯 Output: A single number between 0 and 1 — the probability of fraud. That’s it — one neuron doing business analytics.

🧱 Add More Neurons: Multi-Layer Perceptron (MLP)#

Now let’s go beyond the single neuron and build a small neural network. Think of it as a bunch of interns (neurons) connected in teams (layers), all trying to guess your KPIs.

import torch
import torch.nn as nn
import torch.optim as optim

class MLPClassifier(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(input_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, output_dim),
            nn.Sigmoid()  # Binary output
        )

    def forward(self, x):
        return self.net(x)

# Example: Predict customer churn
model = MLPClassifier(input_dim=5, hidden_dim=10, output_dim=1)
criterion = nn.BCELoss()
optimizer = optim.Adam(model.parameters(), lr=0.01)

# Dummy data
X = torch.randn(100, 5)
y = (X.sum(dim=1) > 0).float().unsqueeze(1)

for epoch in range(10):
    optimizer.zero_grad()
    y_pred = model(X)
    loss = criterion(y_pred, y)
    loss.backward()
    optimizer.step()
    print(f"Epoch {epoch+1} | Loss: {loss.item():.4f}")

🧠 Activation Functions: The Neuron’s Personality#

Each activation gives the neuron a personality trait:

Activation

Personality

When to Use

Sigmoid

“Everything is a probability.”

Binary classification

ReLU

“If life gives me negatives, I ignore them.”

Deep networks, CNNs

Tanh

“Centered and balanced, but a bit moody.”

Sometimes used for small models

LeakyReLU

“ReLU with a backup plan.”

When you fear dead neurons

💡 Why It Works (and When It Fails)#

✅ Works great for

  • Credit scoring

  • Customer churn prediction

  • Sales forecasting with non-linear patterns

❌ Fails spectacularly when

  • You forget to normalize data

  • You don’t shuffle batches

  • You call it “AI” after training for 3 epochs

🧭 Business Analogy#

Think of each neuron as an employee:

  • Each gets some features (inputs)

  • Adds their own “bias” (you know who you are 😏)

  • The output goes through a corporate hierarchy (layers)

  • The CEO (output layer) says “Yes, we’ll launch that campaign”

⚙️ Quick Visualization: Decision Boundary (optional)#

If you have 2D data, you can visualize what the MLP learns:

import matplotlib.pyplot as plt
import numpy as np

# Simulate data
X = torch.randn(500, 2)
y = (X[:, 0]**2 + X[:, 1] > 0).float().unsqueeze(1)

# Model
model = MLPClassifier(2, 8, 1)
optimizer = optim.Adam(model.parameters(), lr=0.01)
criterion = nn.BCELoss()

for _ in range(100):
    optimizer.zero_grad()
    y_pred = model(X)
    loss = criterion(y_pred, y)
    loss.backward()
    optimizer.step()

# Plot
xv, yv = np.meshgrid(np.linspace(-3,3,100), np.linspace(-3,3,100))
grid = torch.tensor(np.c_[xv.ravel(), yv.ravel()]).float()
preds = model(grid).detach().numpy().reshape(100,100)

plt.contourf(xv, yv, preds, cmap='coolwarm', alpha=0.7)
plt.scatter(X[:,0], X[:,1], c=y[:,0], cmap='coolwarm', edgecolors='k')
plt.title("Decision Boundary Learned by MLP")
plt.show()

🎓 Mini Exercises#

  1. Modify the MLP to use two hidden layers. Does it reduce loss faster?

  2. Replace ReLU with Tanh — how does it change convergence?

  3. Try to classify customer churn using synthetic data (hint: generate features that mimic account activity).

  4. Bonus: Plot training loss vs epochs and annotate where overfitting begins.

💬 Funny Wisdom from Neural Networks#

  • “Sigmoid is that one friend who’s always between 0 and 1 — never fully committed.”

  • “If you think your MLP is overfitting, it probably is.”

  • “Dropout: making neurons unemployed for the greater good.”

🚀 Summary#

Concept

Meaning

Perceptron

A single decision-making unit

MLP

Stacked perceptrons for complex patterns

Activation

Introduces non-linearity (a.k.a. intelligence)

Loss

How wrong we are

Optimizer

The thing trying to fix our wrongness

“Neural networks don’t think — they approximate. But with enough layers, even your boss might think it’s magic.”

# Your code here
Contents