CS / data / stats students
Students preparing for ML internships and new-grad roles who want the practical path, not only theory.
Software engineers
Backend and full-stack developers moving into ML engineering, who already write good code and want the ML on top.
Data analysts
Analysts who know SQL and spreadsheets and want to build and ship models, not just report on data.
QA & test engineers
Test automation engineers heading into ML: evaluation, data checks and model testing use the same mindset.
Data engineers
Pipeline builders who want to own the models their data feeds, and the MLOps around them.
Career switchers
Engineers from other fields (EE, mechanical, finance) with some Python, aiming for an ML engineer job.
Prerequisites
| You need | Level | Where to get it |
|---|---|---|
| Python and NumPy arrays | Comfortable | Area 01 |
| School algebra (y = mx + c, powers) | Basic | Any refresher; Module 2 recaps slopes |
| Calculus | None | Module 2 starts from slopes |
Where this area sits in the ML lifecycle — and the roles that use it
| Role | How you use this area on the job | Typical first title |
|---|---|---|
| ML engineer | Debug training (exploding loss, bad learning rates) and read model code fluently | ML Engineer I / Machine Learning Engineer |
| Data scientist | Choose distributions, build confidence intervals, avoid causal claims from correlations | Data Scientist / Applied Scientist |
| AI / LLM engineer | Understand embeddings, cosine similarity and softmax over tokens | AI Engineer / LLM Engineer |
| MLOps / platform | Reason about numerical stability and monitoring statistics | MLOps Engineer / ML Platform Engineer |
| Data engineer | Compute correct aggregate statistics at scale | Data Engineer |
- Compute dot products and matrix products and predict their shapes.
- Measure distance and cosine similarity between vectors.
- Explain derivatives and gradients as slopes and use them to minimise a loss.
- Choose a learning rate by its effect on convergence.
- Apply the chain rule to a small model.
- Use Bayes' rule and explain the base-rate effect.
- Turn scores into probabilities with sigmoid and softmax.
- Summarise data, compute correlations and bootstrap a confidence interval.
| Tool | What it does | Used in this area for | Cost |
|---|---|---|---|
| Python 3.10 – 3.13 | The language of ML: every library on this page is a Python package | Every example and the lab | Free |
| venv + pip | An isolated environment per project and the package installer | Installing the area's packages | Free |
| NumPy | Fast n-dimensional arrays and maths | Vectors, matrices, gradients, sampling | Free |
| Matplotlib | Charts: histograms, scatter plots, learning curves | Plotting losses and distributions | Free |
| JupyterLab | Notebooks: run code cell by cell and see tables and plots inline | Playing with formulas cell by cell | Free |
| Google Colab | Hosted notebooks in the browser, with a free GPU on request | No-install option | Free |
Platforms you will meet at work
| Platform | Vendor | What teams use it for |
|---|---|---|
| SageMaker | AWS | Managed notebooks, training jobs, model registry and endpoints |
| Vertex AI | Google Cloud | Training, pipelines, feature store, model serving and Gemini APIs |
| Azure Machine Learning | Microsoft | Workspaces, MLflow tracking, managed endpoints |
| Databricks | Databricks | Spark + notebooks + MLflow on a lakehouse |
| Weights & Biases | W&B | Experiment tracking and model evaluation dashboards |
The same NumPy maths is what PyTorch and JAX run on GPUs; the notation does not change.
Recommended: WSL 2 (Ubuntu inside Windows) — the same commands as Linux, cloud servers and most ML teams.
- Turn on WSL and install UbuntuOpen PowerShell as Administrator, run the command, then restart.
wsl --install
After the restart an Ubuntu window opens: choose a user name and password. - Install Python's venv support
sudo apt update sudo apt install -y python3 python3-venv python3-pip git make
- Create an environment and install the packages
python3 -m venv ~/ml source ~/ml/bin/activate pip install --upgrade pip pip install numpy matplotlib jupyterlab
Runsource ~/ml/bin/activatein every new terminal. - EditorInstall VS Code, the WSL, Python and Jupyter extensions, then type
code .in Ubuntu. - Check it works
python -c "import numpy; print(numpy.__version__)"
Install Python 3.12 from python.org (tick Add python.exe to PATH), then in PowerShell: py -m venv %USERPROFILE%\ml, %USERPROFILE%\ml\Scripts\activate and the same pip install line. Every lab runs the same.
- Install Apple's command-line tools
xcode-select --install
- Install Homebrew and PythonFollow the one-line command on brew.sh, then:
brew install python@3.12 git make
- Create an environment and install the packages
python3 -m venv ~/ml source ~/ml/bin/activate pip install --upgrade pip pip install numpy matplotlib jupyterlab
- EditorInstall VS Code with the Python and Jupyter extensions. Apple-silicon Macs run every lab natively.
- Check it works
python -c "import numpy; print(numpy.__version__)"
- Ubuntu / Debian
sudo apt update sudo apt install -y python3 python3-venv python3-pip git make
- Fedora
sudo dnf install -y python3 python3-pip git make
- Arch
sudo pacman -S python python-pip git make
- Create an environment and install the packages
python3 -m venv ~/ml source ~/ml/bin/activate pip install --upgrade pip pip install numpy matplotlib jupyterlab
- Check it works
python -c "import numpy; print(numpy.__version__)"
- Open Google ColabGo to colab.research.google.com and sign in with a Google account.
- New notebookFile → New notebook. NumPy, pandas, scikit-learn, Matplotlib and PyTorch are already installed.
- Upload the labClick the folder icon, upload the lab zip, then run
!unzip lab.zip && cd lab && python check.pyin a cell (prefix shell commands with!). - Check the versions
python -c "import numpy; print(numpy.__version__)"
- Create
first.pyimport numpy as np X = np.array([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]]) w = np.array([0.5, -1.0]) print("X @ w =", X @ w) # a dot product per row f = lambda v: (v - 3) ** 2 h = 1e-6 print("slope at 1 =", round((f(1 + h) - f(1 - h)) / (2 * h), 4)) # numerical derivative print("mean, std =", X.mean(axis=0), X.std(axis=0).round(3))
- Run it
python first.py
Output (captured from a real run):X @ w = [-1.5 -2.5 -3.5] slope at 1 = -4.0 mean, std = [3. 4.] [1.633 1.633]
- What to noticeThe slope of (w − 3)² at w = 1 is −4: the loss goes down if w increases. That one number is all gradient descent needs to take a step.
| Session | Do this | Time |
|---|---|---|
| Day 1 | Install · first run · Module 1 (vectors, matrices) + dot-product widget | 3.5 h |
| Day 2 | Module 2 (derivatives, gradient descent) + playground | 4 h |
| Day 3 | Module 3 (probability, Bayes, softmax) + calculator | 3.5 h |
| Day 4 | Module 4 (statistics, bootstrap) | 3 h |
| Day 5 | Lab: gradient descent · area test | 2 h |
Start from zero
New to this? Work through these steps first, in order. Already comfortable? Skim them and jump to the reference below.
A vector is a list of measurements
Describe one thing with several numbers — a house by rooms, age and size — and you have a vector.
A recipe card's ingredient amounts: flour 200, sugar 50, eggs 2. Same card format for every recipe.
| rooms | age | size | |
|---|---|---|---|
| house A | 3 | 1 | 4 |
| house B | 2 | 5 | 1 |
Every example a model sees is a vector of features; the whole dataset is a matrix X.
The dot product is a weighted total
Multiply each number by a weight and add everything up.
A shopping bill: quantities × prices, summed. Quantities are the features, prices the weights.
- x = [3, 1, 4], w = [2, −1, 0.5]
- 3×2 = 6, 1×(−1) = −1, 4×0.5 = 2
- 6 − 1 + 2 = 7
- add the bias b = 1 → prediction 8
Linear regression, logistic regression and every neuron in a network compute exactly this.
[1, 2] · [3, 4] = ?
1×3 + 2×4 = 11.
Matrix multiplication = many dot products at once
Put all examples in a matrix X and all weights in w; X @ w computes every prediction in one go.
Pricing a whole basket of orders at once instead of one receipt at a time.
import numpy as np X = np.array([[3, 1, 4], [2, 5, 1]]) w = np.array([2, -1, 0.5]) print(X @ w + 1)
# output
[8. 0.5]GPUs are matrix-multiplication machines — this is why deep learning is fast.
Similar things point the same way
Two vectors are similar if they point in the same direction, even if one is longer.
Two people walking north-east: one walked further, but they head the same way.
| pair | cosine |
|---|---|
| same direction | 1.0 |
| right angle | 0.0 |
| opposite | −1.0 |
Search, recommendations and RAG compare embeddings with cosine similarity.
Going further: the reference
The same ideas at the next level: precise definitions, tables, rules and interview questions.
Key definitions
- Scalar
- A single number, like a price or a learning rate.
- Vector
- An ordered list of numbers. One example's features form a vector, e.g. [rooms, age, size].
- Matrix
- A 2-D grid of numbers. A dataset X is a matrix: one row per example, one column per feature.
- Dot product
- Multiply matching entries and add: a·b = Σ aᵢbᵢ. A linear model's prediction is x·w + b.
- Transpose
- Flip rows and columns: Xᵀ. A (5, 3) matrix becomes (3, 5).
- Matrix multiplication
- (n, d) @ (d, k) → (n, k): each result entry is a row-column dot product.
- Norm
- A vector's length: ‖x‖ = √(Σ xᵢ²). Used for distances and regularization.
- Cosine similarity
- a·b / (‖a‖‖b‖): direction similarity from −1 to 1; the basis of text and image embeddings search.
import numpy as np x = np.array([3, 1, 4]) # one house: rooms, age, size w = np.array([2, -1, 0.5]) # learned weights b = 1.0 print("x · w + b =", x @ w + b) X = np.array([[3, 1, 4], # 3 houses → a (3, 3) matrix [2, 5, 1], [4, 0, 6]]) print("all predictions:", X @ w + b) print("X.T shape:", X.T.shape)
# output
x · w + b = 8.0
all predictions: [ 8. 0.5 12. ]
X.T shape: (3, 3)Distance and similarity
Two vectors can be compared by how far apart they are (Euclidean distance, used by k-NN and k-means) or by the angle between them (cosine similarity, used to compare text embeddings in search and RAG).
import numpy as np a = np.array([1.0, 2.0, 2.0]) b = np.array([2.0, 4.0, 4.0]) # same direction, twice as long c = np.array([-2.0, 1.0, 0.0]) # at a right angle to a def cosine(u, v): return u @ v / (np.linalg.norm(u) * np.linalg.norm(v)) print("‖a‖ =", np.linalg.norm(a)) print("distance(a, b) =", np.linalg.norm(a - b)) print("cosine(a, b) =", round(cosine(a, b), 3), " cosine(a, c) =", round(cosine(a, c), 3))
# output
‖a‖ = 3.0
distance(a, b) = 3.0
cosine(a, b) = 1.0 cosine(a, c) = 0.0Try it: dot product and angle
| Operation | NumPy | Shape rule |
|---|---|---|
| Dot product | a @ b or np.dot(a, b) | (d,)·(d,) → scalar |
| Matrix × vector | X @ w | (n,d)@(d,) → (n,) |
| Matrix × matrix | A @ B | (n,d)@(d,k) → (n,k) |
| Transpose | X.T | (n,d) → (d,n) |
| Length | np.linalg.norm(x) | (d,) → scalar |
| Element-wise | a * b | same shape (or broadcast) |
* is not matrix multiplicationIn NumPy A * B multiplies element by element; A @ B is the matrix product. Mixing them up gives an answer with the wrong shape — or, worse, the right shape and wrong numbers.
Using * where you meant @ · mismatched inner dimensions · comparing embeddings with Euclidean distance when their lengths differ (use cosine) · forgetting the bias term b.
Interview corner
Q. What does a dot product tell you? A. A weighted sum; geometrically ‖a‖‖b‖cosθ, so it is large when vectors point the same way.
Q. Shapes of X @ W for X (64, 784), W (784, 128)? A. (64, 128): the inner 784s match and disappear.
Key takeaways
- A model's prediction is a dot product plus a bias.
- (n, d) @ (d, k) → (n, k): check shapes first.
- Distance for nearness, cosine for direction.
Start from zero
New to this? Work through these steps first, in order. Already comfortable? Skim them and jump to the reference below.
A derivative is a slope
The derivative tells you how much the output changes when you nudge the input a little.
The steepness of a road at the spot where you stand — uphill, flat or downhill.
f = lambda w: (w - 3) ** 2 h = 0.001 for w in (1, 3, 5): print(w, round((f(w + h) - f(w - h)) / (2 * h), 3))
# output
1 -4.0
3 0.0
5 4.0The slope of the loss tells you which way to move each weight.
Walk downhill: gradient descent
Look at the slope, take a small step the opposite way, repeat. The loss gets smaller each time.
Finding the bottom of a valley in fog: feel the ground, step downhill, repeat.
- w = 0, slope f′(0) = −6 (downhill is to the right)
- step: w ← 0 − 0.1 × (−6) = 0.6
- slope at 0.6 is −4.8 → w ← 1.08
- … steps shrink as the slope flattens near w = 3
This loop — compute slope, step, repeat — trains almost every model in this roadmap.
At w = 5 the slope of (w − 3)² is +4. With η = 0.1, where does w go?
5 − 0.1 × 4 = 4.6: downhill, toward 3.
Many weights → a gradient
With many weights, measure the slope for each one separately. Together these slopes are the gradient.
A hillside map with a slope arrow at each spot: the arrow points the steepest way up.
| weight | slope ∂L/∂w | step direction |
|---|---|---|
| w₁ | +2.0 | decrease |
| w₂ | −0.5 | increase |
| b | 0.0 | stay |
A network with a million weights gets a million slopes per step; the update rule is the same.
Chain rule: slopes multiply
If A affects B and B affects C, the effect of A on C is (effect of A on B) × (effect of B on C).
Exchange rates: dollars → euros → yen. Multiply the two rates to go dollars → yen.
- u = 2w, L = u²
- dL/du = 2u, du/dw = 2
- dL/dw = 2u × 2 = 8w
It is how deep learning gets gradients for every layer.
Going further: the reference
The same ideas at the next level: precise definitions, tables, rules and interview questions.
Key definitions
- Derivative
- How fast a function's output changes as its input changes: the slope of the curve at a point.
- Partial derivative
- The derivative with respect to one input while holding the others fixed: ∂L/∂w₁.
- Gradient
- The vector of all partial derivatives. It points uphill — the direction of steepest increase.
- Loss function
- A number that says how wrong the model is, e.g. mean squared error. Training minimises it.
- Gradient descent
- Repeat: compute the gradient, take a small step the opposite way: w ← w − η∇L.
- Learning rate (η)
- The step size. Too small is slow; too large overshoots or diverges.
- Chain rule
- The derivative of a composition is the product of the local derivatives: (f∘g)′ = f′(g)·g′.
- Backpropagation
- The chain rule applied backwards through a network to get every weight's gradient at once.
| Function | Derivative | Used in |
|---|---|---|
| c (constant) | 0 | biases |
| x² | 2x | squared error |
| xⁿ | n·xⁿ⁻¹ | polynomials |
| eˣ | eˣ | softmax, sigmoid |
| ln x | 1/x | log-loss, cross-entropy |
| σ(x) = 1/(1+e⁻ˣ) | σ(x)(1 − σ(x)) | logistic regression, neurons |
| max(0, x) (ReLU) | 1 if x > 0 else 0 | deep networks |
Gradient descent on one weight
def f(w): return (w - 3) ** 2 # loss, minimum at w = 3 def df(w): return 2 * (w - 3) # its derivative w, lr = 0.0, 0.1 for step in range(1, 26): w = w - lr * df(w) # step against the slope if step in (1, 2, 5, 10, 25): print(f"step {step:2}: w = {w:.4f} loss = {f(w):.6f}")
# output
step 1: w = 0.6000 loss = 5.760000
step 2: w = 1.0800 loss = 3.686400
step 5: w = 2.0170 loss = 0.966368
step 10: w = 2.6779 loss = 0.103763
step 25: w = 2.9887 loss = 0.000128Try it: gradient descent playground
Many weights: the gradient
With weights w₁…w_d the loss depends on all of them. The gradient ∇L collects one partial derivative per weight. For linear regression with mean squared error the gradient has a closed form you will implement in the lab:
import numpy as np rng = np.random.default_rng(0) X = rng.normal(size=(100, 2)) y = X @ np.array([3.0, -2.0]) + 1.0 w, b = np.zeros(2), 0.0 for epoch in range(200): err = X @ w + b - y dw = (2 / len(y)) * X.T @ err # ∂L/∂w, shape (2,) db = (2 / len(y)) * err.sum() # ∂L/∂b w, b = w - 0.1 * dw, b - 0.1 * db print("w =", w.round(3), " b =", round(b, 3))
# output
w = [ 3. -2.] b = 1.0loss.backward() (Area 07) does exactly this for millions of weights.Nudge a weight by ε in each direction and measure the loss change: (L(w+ε) − L(w−ε)) / 2ε. If it disagrees with your formula, the formula is wrong. The lab uses this to catch the gradient bug.
Stepping with the gradient instead of against it (planted bug 3) · forgetting to average over the batch (planted bug 2 — the step size then grows with the dataset) · a learning rate so large the loss becomes NaN · not scaling features, so one weight needs a tiny step and another a big one.
Interview corner
Q. Why subtract the gradient? A. The gradient points uphill (steepest increase of the loss); minimising means moving the opposite way.
Q. What happens if the learning rate is too high? A. Steps overshoot the minimum; the loss oscillates or explodes to inf/NaN.
Key takeaways
- Derivative = slope; gradient = all the slopes.
- w ← w − η·∇L, repeated, is how models learn.
- The chain rule (backprop) makes it work for deep models.
Start from zero
New to this? Work through these steps first, in order. Already comfortable? Skim them and jump to the reference below.
Probability measures how likely
A probability is a number from 0 (never) to 1 (certain).
A weather forecast: 0.3 chance of rain means it rains on about 3 of 10 days like this one.
import numpy as np rng = np.random.default_rng(0) print((rng.random(10_000) < 0.3).mean())
# output
0.3008Classifiers output probabilities; you pick the threshold that turns them into decisions.
“Given that” changes everything
The chance of A when you already know B can be very different from the chance of B given A.
Most basketball players are tall; most tall people are not basketball players.
| question | answer |
|---|---|
| P(tall | basketball player) | high |
| P(basketball player | tall) | low |
It explains why rare-event detectors (fraud, disease) produce many false alarms.
Fraud is 0.1% of transactions. Your model flags 2% of all transactions. At best, what share of flags can be real fraud?
At most 0.1 / 2 = 5%.
Scores become probabilities
A model produces a raw score; sigmoid squashes it between 0 and 1, softmax turns several scores into probabilities that add to 1.
Converting raw votes into percentages of the total.
import numpy as np s = np.array([2.0, 1.0, 0.1]) print((np.exp(s) / np.exp(s).sum()).round(2))
# output
[0.66 0.24 0.1 ]Every classifier and every LLM next-token prediction ends in one of these.
Going further: the reference
The same ideas at the next level: precise definitions, tables, rules and interview questions.
Key definitions
- Probability
- A number from 0 to 1 for how likely an event is.
- Conditional probability
- P(A | B): the probability of A given that B happened.
- Bayes' rule
- P(A | B) = P(B | A)·P(A) / P(B): flip a conditional using the base rate P(A).
- Random variable
- A quantity whose value is uncertain, e.g. “clicks tomorrow”.
- Distribution
- All possible values of a random variable and how likely each is.
- Bernoulli / binomial
- Yes/no outcome with probability p; the count of yeses in n tries.
- Normal (Gaussian)
- The bell curve N(μ, σ²); sums and averages of many small effects tend to be normal.
- Expected value
- The long-run average: E[X] = Σ x·P(x).
Try it: Bayes calculator
Distributions you will meet
import numpy as np rng = np.random.default_rng(1) flips = rng.binomial(n=1, p=0.3, size=10_000) # Bernoulli: did the user click? heights = rng.normal(loc=170, scale=8, size=10_000) # normal print("click rate ≈", flips.mean()) print("mean height ≈", heights.mean().round(1), " std ≈", heights.std().round(1)) print("share within 1 std:", np.mean(np.abs(heights - 170) < 8).round(3))
# output
click rate ≈ 0.3039
mean height ≈ 169.9 std ≈ 7.9
share within 1 std: 0.69| Distribution | Models | NumPy | ML use |
|---|---|---|---|
| Bernoulli | one yes/no | rng.binomial(1, p) | the target of a binary classifier |
| Binomial | count of yeses in n | rng.binomial(n, p) | conversions per day |
| Categorical | one of k classes | rng.choice(k, p=…) | softmax outputs, next-token prediction |
| Normal | sums of many effects | rng.normal(μ, σ) | noise, weight initialisation |
| Uniform | anything equally likely | rng.uniform(a, b) | random search over hyperparameters |
| Poisson | counts of rare events | rng.poisson(λ) | visits, failures per hour |
From scores to probabilities
Classifiers output a score z; the sigmoid squashes it into a probability between 0 and 1, and softmax does the same for many classes so the probabilities add to 1. The loss used to train them, cross-entropy, is the negative log of the probability given to the true answer — confident wrong answers are punished hardest.
import numpy as np z = np.array([-2.0, 0.0, 3.0]) print("sigmoid:", (1 / (1 + np.exp(-z))).round(3)) scores = np.array([2.0, 1.0, 0.1]) # 3 classes p = np.exp(scores) / np.exp(scores).sum() print("softmax:", p.round(3), "sum =", p.sum()) print("cross-entropy if class 0 is right:", round(-np.log(p[0]), 3)) print("cross-entropy if class 2 is right:", round(-np.log(p[2]), 3))
# output
sigmoid: [0.119 0.5 0.953]
softmax: [0.659 0.242 0.099] sum = 1.0
cross-entropy if class 0 is right: 0.417
cross-entropy if class 2 is right: 2.317A fraud model that says 0.97 is more useful than one that says “fraud”: the business picks the threshold. That is why calibration and thresholds get a whole module in Area 05.
Confusing P(A | B) with P(B | A) · ignoring the base rate · assuming everything is normal (incomes, wait times and clicks are skewed) · treating a model score as a calibrated probability without checking.
Interview corner
Q. A test is 99% accurate and the disease affects 1 in 1,000. A positive result means? A. Mostly a false positive: ~1 true positive per ~10 false positives, so P(sick | +) ≈ 9%.
Q. Why softmax? A. It turns any real-valued scores into positive probabilities that sum to 1, and it is differentiable.
Key takeaways
- Conditional probability is directional — base rates matter.
- Know Bernoulli, categorical and normal by heart.
- Sigmoid/softmax make probabilities; cross-entropy scores them.
Start from zero
New to this? Work through these steps first, in order. Already comfortable? Skim them and jump to the reference below.
Centre and spread
The mean is the balance point; the median is the middle value; the standard deviation is the typical distance from the mean.
Average salary vs median salary when one billionaire walks into the room.
import numpy as np x = np.array([3, 4, 4, 5, 100]) print(x.mean(), np.median(x), x.std().round(1))
# output
23.2 4.0 38.4Skewed features (prices, durations) need the median and often a log transform.
Moving together isn't causing
Correlation says two numbers rise and fall together. It doesn't say one causes the other.
Ice-cream sales and sunburns both rise in summer; ice cream doesn't cause sunburn.
| pair | r | causal? |
|---|---|---|
| ad spend, sales (A/B tested) | 0.9 | yes (tested) |
| ice cream, sunburn | 0.8 | no (summer) |
Models find correlations; only experiments prove causes.
Customers who use feature X churn less (r = −0.6). Should you force everyone onto X?
Not yet: engaged customers may both use X and stay. Run an A/B test.
A number from a sample wobbles
Measure a different sample and you'd get a slightly different answer. The wobble shrinks as samples grow.
Tasting soup: one spoonful can mislead; ten spoonfuls from different spots tell you more.
| test-set size | typical wobble (1 std error, accuracy ≈ 80%) |
|---|---|
| 100 | ± 4% |
| 1,000 | ± 1.3% |
| 10,000 | ± 0.4% |
Before declaring a new model better, check the difference is bigger than the wobble.
Going further: the reference
The same ideas at the next level: precise definitions, tables, rules and interview questions.
Key definitions
- Mean / median
- The average, and the middle value; the median ignores extreme outliers.
- Variance / std
- Average squared distance from the mean, and its square root (same units as the data).
- z-score
- (x − mean) / std: how many standard deviations from the mean. Standardization uses it.
- Correlation (r)
- Strength of a linear relationship, from −1 to +1. Not causation.
- Population / sample
- Everyone you care about, and the subset you actually observed.
- Sampling variability
- Different samples give different statistics; that wobble is uncertainty.
- Central limit theorem
- Averages of many independent values are approximately normal, with spread σ/√n.
- Confidence interval
- A range that would contain the true value in, e.g., 95% of repeated samples.
import numpy as np wait_min = np.array([2, 3, 3, 4, 4, 5, 5, 6, 7, 48]) # one very slow day print("mean =", wait_min.mean()) print("median =", np.median(wait_min)) print("std =", wait_min.std().round(2)) z = (wait_min - wait_min.mean()) / wait_min.std() print("z-scores:", z.round(2))
# output
mean = 8.7
median = 4.5
std = 13.18
z-scores: [-0.51 -0.43 -0.43 -0.36 -0.36 -0.28 -0.28 -0.2 -0.13 2.98]import numpy as np rng = np.random.default_rng(3) ads = rng.uniform(0, 10, 200) sales = 5 * ads + rng.normal(0, 8, 200) icecream = rng.uniform(0, 10, 200) print("corr(ads, sales) =", np.corrcoef(ads, sales)[0, 1].round(2)) print("corr(ads, ice cream) =", np.corrcoef(ads, icecream)[0, 1].round(2))
# output
corr(ads, sales) = 0.86
corr(ads, ice cream) = -0.0How sure are you? The bootstrap
A model's accuracy on a test set is itself an estimate from a sample. Resampling the test set with replacement many times shows how much it would wobble — a confidence interval without formulas.
import numpy as np rng = np.random.default_rng(0) correct = rng.random(500) < 0.82 # 500 test predictions, ~82% right acc = correct.mean() boot = [rng.choice(correct, size=correct.size, replace=True).mean() for _ in range(2000)] lo, hi = np.percentile(boot, [2.5, 97.5]) print(f"accuracy = {acc:.3f} 95% CI ≈ [{lo:.3f}, {hi:.3f}]")
# output
accuracy = 0.788 95% CI ≈ [0.752, 0.824]“Model B is 0.6% better” means nothing if the 95% interval is ±3%. Bootstrap the test set before claiming a win — interviewers love this question.
The lab: gradient descent, by hand
Reporting the mean of skewed data without the median · reading causation into correlation · comparing models on tiny test sets · computing std with the wrong ddof without knowing (NumPy uses ddof=0, pandas ddof=1).
Interview corner
Q. Correlation vs causation? A. Correlation is co-movement; causation needs an intervention or a design (A/B test) that rules out confounders.
Q. Why standardize features? A. So features on different scales get comparable weights and gradient descent converges faster.
Key takeaways
- Median for skewed data; std for spread; z-scores for scaling.
- Correlation is linear and not causal.
- Every metric is a sample estimate — bootstrap it.
[1, 2, 3] · [4, 0, −1]?
Shape of (32, 10) @ (10, 4)?
f(w) = w², what is f′(3)? Which way does gradient descent step from w = 3?
Loss turns to NaN after a few steps. First thing to try?
Chain rule: L = (2w)², dL/dw?
Rare disease 0.1%, test 99% sensitive, 1% false positives. Is a positive likely sick?
Softmax of [0, 0]?
Accuracy 0.84 on 50 test examples vs 0.83 on 5,000. Which is more trustworthy?
Explain gradient descent to a non-technical person.
Batch vs stochastic gradient descent?
Why do we need non-linear activation functions?
What is the vanishing gradient problem?
Bayes' theorem — give an ML example.
Why cross-entropy instead of MSE for classification?
How would you build a confidence interval for a model's accuracy?
What does cosine similarity measure and why use it for embeddings?
| You see | Why | Fix |
|---|---|---|
| ValueError: matmul: Input operand 1 has a mismatch | Inner dimensions differ | Print both shapes; transpose one (.T) if needed |
| Loss becomes inf or nan | Learning rate too high, or unscaled features | Lower η by 10×; standardize X |
| Loss barely changes | Learning rate too low, or gradient formula wrong | Raise η; compare to a numerical gradient |
| RuntimeWarning: overflow in exp | Large scores in softmax/sigmoid | Subtract the max score before exp (stable softmax) |
| Std differs between NumPy and pandas | ddof 0 vs 1 | Pass ddof=1 to NumPy or ddof=0 to pandas |
| My bootstrap interval is huge | Small test set | Collect more evaluation data before comparing models |
| Resource | Why read it |
|---|---|
| Deisenroth, Faisal & Ong, Mathematics for Machine Learning (Cambridge, free PDF at mml-book.github.io) | Linear algebra, calculus and probability written for ML |
| 3Blue1Brown, Essence of Linear Algebra and Essence of Calculus (YouTube) | Visual intuition for every idea in Modules 1–2 |
| Goodfellow, Bengio & Courville, Deep Learning, ch. 2–4 (free at deeplearningbook.org) | The maths background chapters used by deep learning |
| Efron & Hastie, Computer Age Statistical Inference (free PDF) | The bootstrap and modern statistics, by the bootstrap's inventor |
Lab 02 · Gradient descent from scratch
gd.py trains linear regression on 200 samples using only NumPy. The loss, the gradient and the update each hide one bug. check.py compares against a loss you can compute by hand, numerical gradients and the exact least-squares answer.
pip install -r requirements.txt, thenpython check.py.- Start with check 1: do the hand calculation in the message. What does
mseforget? - Checks 2–3: which gradient disagrees with the numerical one, and by roughly what factor of n?
- Checks 4–5: the loss explodes. Which way should the update step go?
- Re-run after each fix until all five checks are ok.
Done when you see PASS: all 5 checks passed — gradient descent recovers w=[ 1.99 -1.01 0.51], b=4.00.