CS / data / stats students
Students preparing for ML internships and new-grad roles who want the practical path, not only theory.
Software engineers
Backend and full-stack developers moving into ML engineering, who already write good code and want the ML on top.
Data analysts
Analysts who know SQL and spreadsheets and want to build and ship models, not just report on data.
QA & test engineers
Test automation engineers heading into ML: evaluation, data checks and model testing use the same mindset.
Data engineers
Pipeline builders who want to own the models their data feeds, and the MLOps around them.
Career switchers
Engineers from other fields (EE, mechanical, finance) with some Python, aiming for an ML engineer job.
Prerequisites
| You need | Level | Where to get it |
|---|---|---|
| Programming basics (variables, if, loops) in any language | Beginner | Any intro course; Module 1 recaps the Python |
| Spreadsheets (rows, columns, sum) | Helpful | Everyday experience |
| Maths | None yet | Area 02 covers the maths for ML |
Where this area sits in the ML lifecycle — and the roles that use it
| Role | How you use this area on the job | Typical first title |
|---|---|---|
| ML engineer | Write the data-loading and feature code that feeds every model | ML Engineer I / Machine Learning Engineer |
| Data scientist | Explore and summarise data in notebooks every day | Data Scientist / Applied Scientist |
| AI / LLM engineer | Prepare and clean datasets for evaluation and fine-tuning | AI Engineer / LLM Engineer |
| MLOps / platform | Build reproducible data steps that run in pipelines | MLOps Engineer / ML Platform Engineer |
| Data engineer | Hand clean tables to ML teams; check them with pandas | Data Engineer |
- Set up Python in a virtual environment and install packages with pip.
- Use lists, dicts, comprehensions and functions to transform data.
- Create, index and slice NumPy arrays and use boolean masks.
- Reduce along the right axis and explain broadcasting.
- Load a CSV into pandas and inspect shape, dtypes and missing values.
- Filter, add columns, group and aggregate with pandas.
- Choose the right join and verify row counts.
- Find and fix three silent data bugs in a real script.
| Tool | What it does | Used in this area for | Cost |
|---|---|---|---|
| Python 3.10 – 3.13 | The language of ML: every library on this page is a Python package | Every example and the lab | Free |
| venv + pip | An isolated environment per project and the package installer | Installing the area's packages | Free |
| NumPy | Fast n-dimensional arrays and maths | Arrays, axes, broadcasting (Module 2) | Free |
| pandas | Tables (DataFrames): load, clean, group and join data | Tables, NaN, groupby and joins (Module 3) | Free |
| Matplotlib | Charts: histograms, scatter plots, learning curves | Histograms, scatter and line plots (Module 4) | Free |
| JupyterLab | Notebooks: run code cell by cell and see tables and plots inline | Exploring data cell by cell | Free |
| VS Code + Python & Jupyter extensions | Editor, debugger and notebooks in one place | Writing and debugging code | Free |
| Google Colab | Hosted notebooks in the browser, with a free GPU on request | No-install option for every example | Free |
Platforms you will meet at work
| Platform | Vendor | What teams use it for |
|---|---|---|
| SageMaker | AWS | Managed notebooks, training jobs, model registry and endpoints |
| Vertex AI | Google Cloud | Training, pipelines, feature store, model serving and Gemini APIs |
| Azure Machine Learning | Microsoft | Workspaces, MLflow tracking, managed endpoints |
| Databricks | Databricks | Spark + notebooks + MLflow on a lakehouse |
| Weights & Biases | W&B | Experiment tracking and model evaluation dashboards |
The pandas and NumPy you learn here are exactly what runs on SageMaker, Vertex AI or Databricks notebooks; only the machine gets bigger.
Recommended: WSL 2 (Ubuntu inside Windows) — the same commands as Linux, cloud servers and most ML teams.
- Turn on WSL and install UbuntuOpen PowerShell as Administrator, run the command, then restart.
wsl --install
After the restart an Ubuntu window opens: choose a user name and password. - Install Python's venv support
sudo apt update sudo apt install -y python3 python3-venv python3-pip git make
- Create an environment and install the packages
python3 -m venv ~/ml source ~/ml/bin/activate pip install --upgrade pip pip install numpy pandas matplotlib jupyterlab
Runsource ~/ml/bin/activatein every new terminal. - EditorInstall VS Code, the WSL, Python and Jupyter extensions, then type
code .in Ubuntu. - Check it works
python -c "import numpy, pandas, matplotlib; print(numpy.__version__, pandas.__version__)"
Install Python 3.12 from python.org (tick Add python.exe to PATH), then in PowerShell: py -m venv %USERPROFILE%\ml, %USERPROFILE%\ml\Scripts\activate and the same pip install line. Every lab runs the same.
- Install Apple's command-line tools
xcode-select --install
- Install Homebrew and PythonFollow the one-line command on brew.sh, then:
brew install python@3.12 git make
- Create an environment and install the packages
python3 -m venv ~/ml source ~/ml/bin/activate pip install --upgrade pip pip install numpy pandas matplotlib jupyterlab
- EditorInstall VS Code with the Python and Jupyter extensions. Apple-silicon Macs run every lab natively.
- Check it works
python -c "import numpy, pandas, matplotlib; print(numpy.__version__, pandas.__version__)"
- Ubuntu / Debian
sudo apt update sudo apt install -y python3 python3-venv python3-pip git make
- Fedora
sudo dnf install -y python3 python3-pip git make
- Arch
sudo pacman -S python python-pip git make
- Create an environment and install the packages
python3 -m venv ~/ml source ~/ml/bin/activate pip install --upgrade pip pip install numpy pandas matplotlib jupyterlab
- Check it works
python -c "import numpy, pandas, matplotlib; print(numpy.__version__, pandas.__version__)"
- Open Google ColabGo to colab.research.google.com and sign in with a Google account.
- New notebookFile → New notebook. NumPy, pandas, scikit-learn, Matplotlib and PyTorch are already installed.
- Upload the labClick the folder icon, upload the lab zip, then run
!unzip lab.zip && cd lab && python check.pyin a cell (prefix shell commands with!). - Check the versions
python -c "import numpy, pandas, matplotlib; print(numpy.__version__, pandas.__version__)"
- Activate your environment and create
first.pyimport numpy as np import pandas as pd rng = np.random.default_rng(42) df = pd.DataFrame({ "price": rng.normal(40, 12, 5).round(2), "qty": rng.integers(1, 6, 5), }) df["revenue"] = df["price"] * df["qty"] print(df) print("total revenue:", round(df["revenue"].sum(), 2)) print("numpy", np.__version__, "· pandas", pd.__version__)
- Run it
python first.py
Output (captured from a real run):price qty revenue 0 43.66 3 130.98 1 27.52 5 137.60 2 49.01 4 196.04 3 51.29 4 205.16 4 16.59 4 66.36 total revenue: 736.14 numpy 2.5.3 · pandas 3.0.5
- What to noticeThe same seed (42) gives everyone the same numbers.
df["revenue"] = df["price"] * df["qty"]multiplied whole columns at once — no loop. That is the idea behind everything in this area.
| Session | Do this | Time |
|---|---|---|
| Day 1 | Install · first run · Module 1 (Python essentials) | 3 h |
| Day 2 | Module 2 (NumPy: axis and broadcasting) + both widgets | 3.5 h |
| Day 3 | Module 3 (pandas: NaN, groupby, joins) | 3.5 h |
| Day 4 | Module 4 (plots and workflow) · Lab: fix prep.py | 3 h |
| Day 5 | Lab review · area test | 1 h |
Start from zero
New to this? Work through these steps first, in order. Already comfortable? Skim them and jump to the reference below.
A variable is a labelled box
A variable gives a name to a value so you can use it later. The value can be a number, some text, or a whole collection.
A sticky label on a storage box: the label is the name, the contents are the value. Re-labelling doesn't change what's inside.
price = 19.99 qty = 2 revenue = price * qty print(revenue)
# output
39.98=), expression.Every data step is a chain of named values. Good names (revenue, not x2) make ML code readable months later.
Lists and dicts hold many values
A list keeps values in order; a dictionary looks values up by a name (key).
A list is a queue at a shop — first, second, third. A dict is a phone book — find the number by the name.
prices = [19.99, 5.50, 42.00] print(prices[0], len(prices)) region = {"east": 119.0, "west": 40.0} print(region["west"])
# output
19.99 3
40.0Rows of data, feature names, model settings (hyperparameters) and results are all lists and dicts.
What does [19.99, 5.50, 42.00][1] give?
5.5 — Python counts from 0, so index 1 is the second item.Comprehensions transform a whole list
A comprehension builds a new list by doing the same thing to every item, in one line.
A stamping machine: every item that goes in comes out stamped the same way.
prices = [19.99, 5.50, 42.00] with_tax = [round(p * 1.08, 2) for p in prices] cheap = [p for p in prices if p < 20] print(with_tax, cheap)
# output
[21.59, 5.94, 45.36] [19.99, 5.5][expression for item in items if condition].It's the bridge to vectorized thinking: “do this to every value” is exactly what NumPy and pandas do — only much faster.
Functions name a step
A function packages steps under a name. You give it inputs, it gives back an output.
A recipe: ingredients in (parameters), dish out (return value). Cook it again with different ingredients any time.
def mean(xs): return sum(xs) / len(xs) print(mean([2, 4, 9]))
# output
5.0Data cleaning in real projects is a set of small tested functions — exactly like clean_orders in the lab.
What does a function without return give back?
None. A very common bug: the function computes the answer but never returns it.Going further: the reference
The same ideas at the next level: precise definitions, tables, rules and interview questions.
Key definitions
- Interpreter
- The program (
python) that reads your code and runs it line by line. - Virtual environment
- A private folder of packages for one project, made with
python -m venv, so projects never fight over versions. - List
- An ordered, changeable collection:
[3, 1, 2]. Can hold anything, including mixed types. - Dictionary
- Key → value lookup:
{"east": 119}. Keys are unique. - Comprehension
- A one-line loop that builds a list or dict:
[x * 2 for x in xs]. - Function
- Named, reusable code with inputs (parameters) and an output (
return). - Package
- Installable code others wrote — NumPy, pandas, scikit-learn — added with
pip install. - f-string
- Formatted text:
f"{mean:.2f}"puts a value into a string with 2 decimals.
The Python you actually need for data work
ML code is mostly ordinary Python gluing libraries together. Four things carry most of the weight: lists and dicts to hold things, loops and comprehensions to transform them, functions to name a step, and f-strings to report results.
prices = [19.99, 5.50, 42.00, 7.25] qty = [2, 10, 1, 4] revenue = [p * q for p, q in zip(prices, qty)] # comprehension + zip by_month = {"2026-01": 39.98, "2026-02": 84.00} # dict by_month["2026-03"] = 24.00 # add a key def mean(xs): """Average of a list of numbers.""" return sum(xs) / len(xs) print(revenue) print(f"mean price = {mean(prices):.2f}") print(sorted(by_month.items(), key=lambda kv: kv[1], reverse=True)[0])
# output
[39.98, 55.0, 42.0, 29.0]
mean price = 18.68
('2026-02', 84.0)| Need | Python | Example |
|---|---|---|
| Keep items in order | list | [3, 1, 2] |
| Look up by name | dict | d["east"] |
| Unique values | set | set(regions) |
| Fixed record | tuple | ("east", 119) |
| Transform every item | comprehension | [f(x) for x in xs] |
| Pair two lists | zip | zip(prices, qty) |
| Index and value | enumerate | for i, x in enumerate(xs) |
Loops vs vectorized code: measured
import time import numpy as np xs = list(range(1_000_000)) t0 = time.perf_counter(); ys = [x * 2.0 for x in xs]; t_list = time.perf_counter() - t0 a = np.arange(1_000_000, dtype=float) t0 = time.perf_counter(); b = a * 2.0; t_np = time.perf_counter() - t0 print(f"list comprehension: {t_list * 1000:.1f} ms") print(f"numpy vectorized: {t_np * 1000:.1f} ms") print(f"speed-up: about {t_list / t_np:.0f}x")
# output
list comprehension: 38.9 ms
numpy vectorized: 2.7 ms
speed-up: about 15xCreate a venv for each project and keep a requirements.txt (pip freeze > requirements.txt). The commonest “works on my machine” bug in ML is two projects sharing one Python with different library versions.
Mutating a list while looping over it · using a list of numbers where an array belongs · forgetting return (the function gives None) · installing packages into the system Python instead of a venv · comparing floats with == (use math.isclose / np.isclose).
Interview corner
Q. List vs NumPy array? A. A list holds arbitrary Python objects (pointers); an array holds one dtype in contiguous memory, so maths on it runs in compiled loops and is far faster.
Q. Why use a virtual environment? A. To pin each project's package versions independently and make the setup reproducible with requirements.txt.
Key takeaways
- Lists, dicts, comprehensions and functions are 90% of the Python you need.
- Numbers in bulk belong in NumPy arrays.
- One venv and one requirements.txt per project.
Start from zero
New to this? Work through these steps first, in order. Already comfortable? Skim them and jump to the reference below.
An array is a grid of numbers of one type
NumPy's array stores many numbers of the same type, packed together. It can be a row, a table, or more dimensions.
An egg carton: fixed slots, all holding the same kind of thing, in neat rows and columns.
import numpy as np X = np.array([[1, 2, 3], [4, 5, 6]]) print(X.shape, X.dtype) print(X * 10)
# output
(2, 3) int64
[[10 20 30]
[40 50 60]]Every ML model takes its input as an array (often called X): one row per example, one column per feature.
Vectorization: whole-array maths, no loop
Instead of writing a loop, you write the maths once and NumPy applies it to every element.
Painting a fence with a roller instead of a tiny brush, one plank at a time.
import numpy as np price = np.array([19.99, 5.50, 42.00]) qty = np.array([2, 10, 1]) print(price * qty) # element by element print((price * qty).sum())
# output
[39.98 55. 42. ]
136.98It's typically 50–100× faster than a Python loop and shorter to read.
axis: which way to collapse
When you take a mean or sum of a table, axis says whether to squash the rows (one answer per column) or the columns (one answer per row).
A class grade sheet: averaging down each column gives the average per subject; averaging across each row gives each student's average.
| Maths | Physics | |
|---|---|---|
| Ana | 80 | 70 |
| Ben | 60 | 90 |
| axis=0 mean | 70 | 80 |
Across each row (axis=1) gives Ana 75 and Ben 75.
Features are columns. Scaling features needs per-column statistics — axis=0. Forgetting it is lab bug 2.
A (100, 5) array of 100 examples and 5 features. Shape of X.mean(axis=0)?
(5,) — one mean per feature.
Broadcasting: different shapes, one operation
NumPy can combine a big array with a smaller one by reusing the smaller one along a dimension.
Adding the same discount row to every customer's bill — you write the discount once, it applies to every row.
- X has shape (3, 4); the column means have shape (4,).
- Line shapes up from the right: 4 matches 4.
- The (4,) row is reused for all 3 rows.
X - X.mean(axis=0)subtracts each column's mean from that column.
It's how feature scaling, normalization and many model computations are written in one line.
Going further: the reference
The same ideas at the next level: precise definitions, tables, rules and interview questions.
Key definitions
- ndarray
- NumPy's n-dimensional array: one dtype, a fixed shape, contiguous memory.
- shape
- Size along each axis:
(3, 4)is 3 rows and 4 columns. - dtype
- Element type:
float64,int64,bool… - axis
- Which dimension an operation runs along.
axis=0= down the rows (per column). - Vectorization
- Expressing a computation on whole arrays instead of element-by-element Python loops.
- Broadcasting
- NumPy's rule for combining arrays of different shapes by stretching size-1 axes.
- Boolean mask
- An array of True/False used to select elements:
a[a > 0]. - View vs copy
- Slices are views of the same memory; changing a view changes the original.
import numpy as np X = np.array([[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]], dtype=float) print(X.shape, X.dtype) print(X[1]) # second row print(X[:, 2]) # third column print(X[X > 6]) # boolean mask print(X.mean(axis=0)) # one value per column print(X.mean(axis=1)) # one value per row
# output
(3, 4) float64
[5. 6. 7. 8.]
[ 3. 7. 11.]
[ 7. 8. 9. 10. 11. 12.]
[5. 6. 7. 8.]
[ 2.5 6.5 10.5]axis=0 collapses rows → one result per column (per feature) — what you almost always want for scaling features.Try it: axis explorer
Broadcasting: combining different shapes
When two arrays have different shapes, NumPy compares their shapes from the right. Each pair of sizes must be equal, or one of them must be 1 — a size-1 axis is stretched (without copying) to match. That is how you subtract a per-column mean from every row in one line.
import numpy as np X = np.array([[1., 2., 3., 4.], [5., 6., 7., 8.], [9., 10., 11., 12.]]) col_mean = X.mean(axis=0) # shape (4,) Z = (X - col_mean) / X.std(axis=0) # (3,4) - (4,) broadcasts per column print(Z.round(2)) print("column means:", Z.mean(axis=0).round(6)) row_mean = X.mean(axis=1) # shape (3,) try: X - row_mean # (3,4) - (3,) does NOT broadcast except ValueError as e: print("ValueError:", e) print((X - row_mean[:, None]).round(1)) # make it (3,1) and it works
# output
[[-1.22 -1.22 -1.22 -1.22]
[ 0. 0. 0. 0. ]
[ 1.22 1.22 1.22 1.22]]
column means: [0. 0. 0. 0.]
ValueError: operands could not be broadcast together with shapes (3,4) (3,)
[[-1.5 -0.5 0.5 1.5]
[-1.5 -0.5 0.5 1.5]
[-1.5 -0.5 0.5 1.5]]Try it: will these shapes broadcast?
| Operation | NumPy | Note |
|---|---|---|
| Create | np.array(...), np.zeros((3,4)), np.arange(10), np.linspace(0,1,5) | set dtype= if needed |
| Random (reproducible) | rng = np.random.default_rng(42); rng.normal(size=5) | new Generator API |
| Reshape | a.reshape(3, 4), a[:, None] | -1 lets NumPy infer one size |
| Select | a[1:3], a[a > 0], a[[0, 2]] | slices are views |
| Reduce | a.sum(axis=0), a.mean(), a.argmax() | axis = which dim disappears |
| Combine | np.concatenate, np.stack, a @ b | @ is matrix multiply |
Without an axis, NumPy averages every number in the array into one value. Standardizing features with it mixes price and quantity together — the result has the wrong mean and spread per column. That is planted bug 2 in the lab.
Forgetting the axis in reductions · assuming a slice is a copy (it is a view — use .copy()) · integer arrays silently truncating (np.array([1, 2]) / 2 is fine, but assigning 0.5 into an int array stores 0) · shape (n,) vs (n,1) confusion · using np.random.seed globals instead of a Generator.
Interview corner
Q. What is broadcasting? A. Combining arrays of different shapes by comparing sizes right to left; sizes must match or be 1, and size-1 axes are stretched virtually.
Q. What does axis=0 mean in X.mean(axis=0)? A. Reduce along the rows: the row dimension disappears and you get one mean per column.
Key takeaways
- Arrays: one dtype, a shape, fast vectorized maths.
- axis tells NumPy which dimension to collapse.
- Broadcasting: right to left, equal or 1.
Start from zero
New to this? Work through these steps first, in order. Already comfortable? Skim them and jump to the reference below.
A DataFrame is a spreadsheet you program
pandas gives you tables with named columns. Each column has one type, like a column of prices or a column of dates.
An Excel sheet you control with code: every change is written down and can be re-run.
import pandas as pd df = pd.read_csv("orders.csv") print(df)
# output
order_id customer_id price qty coupon
0 1 3.0 19.99 2 SPRING10
1 2 1.0 5.50 10 NaN
2 3 NaN 42.00 1 NaN
3 4 2.0 7.25 4 SPRING10Almost every dataset you train on starts as a DataFrame.
Missing values show up as NaN
When a cell is empty, pandas writes NaN. You must decide: drop that row, or fill the gap.
A form with blanks: if the name is missing you can't process it; if the optional coupon box is blank, that's fine.
import pandas as pd df = pd.read_csv("orders.csv") print(df.isna().sum()) print(len(df.dropna()), "vs", len(df.dropna(subset=["customer_id", "price"])))
# output
order_id 0
customer_id 1
price 0
qty 0
coupon 2
dtype: int64
2 vs 3isna(), dropna(subset=…), fillna().Dropping too much data biases your model; keeping broken rows crashes it.
Order 3 has no customer_id; orders 2 and 3 have no coupon. Which rows should dropna(subset=["customer_id", "price"]) drop?
Only order 3. The coupon is optional.
groupby answers “per something” questions
To get a total or average per category, group the rows by that category and aggregate each group.
Sorting receipts into piles by month, then adding up each pile.
import pandas as pd df = pd.DataFrame({"region": ["east", "west", "east"], "revenue": [40, 29, 55]}) print(df.groupby("region")["revenue"].sum())
# output
region
east 95
west 29
Name: revenue, dtype: int64“Average purchase per customer”, “churn rate per plan” — these features come from groupby.
Joins combine two tables
A join matches rows from two tables using a shared key column, like customer_id.
Matching name tags to a guest list: some guests aren't on the list, some names on the list never arrived. You choose who to keep.
| how= | keeps |
|---|---|
| inner | only matches |
| left | every row of the left table |
| right | every row of the right table |
| outer | everything |
on= and how=.An inner join silently drops rows whose key is missing on the other side — lab bug 3.
Going further: the reference
The same ideas at the next level: precise definitions, tables, rules and interview questions.
Key definitions
- DataFrame
- A table: labelled rows (the index) and named columns, each column one dtype.
- Series
- One column (or row) of a DataFrame: values plus an index.
- loc / iloc
locselects by label,ilocby position.- NaN
- “Not a Number”: pandas' marker for a missing value.
- groupby
- Split rows by a key, apply an aggregation to each group, combine the results.
- merge / join
- Combine two tables on a key column.
howdecides which rows survive. - dtype object
- pandas' type for text (or mixed) columns.
- Method chaining
- Writing steps as
df.dropna(...).assign(...).groupby(...)— one readable pipeline.
import pandas as pd df = pd.read_csv("orders.csv") print(df.shape) print(df.dtypes) print(df.isna().sum())
# output
(7, 6)
order_id int64
customer_id float64
month str
price float64
qty int64
coupon str
dtype: object
order_id 0
customer_id 1
month 0
price 1
qty 0
coupon 5
dtype: int64Select, filter, add columns
import pandas as pd df = pd.read_csv("orders.csv") cheap = df.loc[df["price"] < 10, ["order_id", "price", "qty"]] # rows by mask, columns by name print(cheap) df = df.assign(revenue=df["price"] * df["qty"]) # new column print(df[["order_id", "revenue"]].head(3)) print(df.iloc[0, :3].to_dict()) # first row, first 3 columns by position
# output
order_id price qty
1 2 5.50 10
3 4 7.25 4
order_id revenue
0 1 39.98
1 2 55.00
2 3 42.00
{'order_id': 1, 'customer_id': 3.0, 'month': '2026-01'}Missing values: drop only what you must
import pandas as pd df = pd.read_csv("orders.csv") print("dropna(): ", len(df.dropna()), "rows") # drops rows with ANY NaN, incl. coupon print("dropna(subset=[...]): ", len(df.dropna(subset=["customer_id", "price"])), "rows") print(df["coupon"].fillna("none").value_counts().to_dict())
# output
dropna(): 2 rows
dropna(subset=[...]): 5 rows
{'none': 5, 'SPRING10': 2}coupon is optional, so most rows have NaN there. df.dropna() throws those rows away along with the genuinely broken ones — 7 orders become 1. Always say which columns are required: dropna(subset=["customer_id", "price"]). Planted bug 1.
groupby: split, apply, combine
df.groupby("region")["revenue"].sum().import pandas as pd df = pd.read_csv("orders.csv").dropna(subset=["customer_id", "price"]) df = df.assign(revenue=df["price"] * df["qty"]) print(df.groupby("month")["revenue"].agg(["count", "sum", "mean"]).round(2))
# output
count sum mean
month
2026-01 2 94.98 47.49
2026-02 1 29.00 29.00
2026-03 2 54.00 27.00Joins: which rows survive
how="left".import pandas as pd orders = pd.read_csv("orders.csv").dropna(subset=["customer_id", "price"]) customers = pd.read_csv("customers.csv") for how in ["inner", "left"]: j = orders.merge(customers, on="customer_id", how=how) print(f"{how:5}: {len(j)} rows, regions = {j['region'].fillna('unknown').unique().tolist()}")
# output
inner: 4 rows, regions = ['east', 'west']
left : 5 rows, regions = ['east', 'west', 'unknown']Try it: join explorer
| Task | pandas |
|---|---|
| Rows and columns | df.shape, df.info() |
| Summary stats | df.describe() |
| Count values | df["col"].value_counts() |
| Rename / drop | df.rename(columns={...}), df.drop(columns=[...]) |
| Sort | df.sort_values("revenue", ascending=False) |
| Dates | pd.to_datetime(df["month"]) |
| Save | df.to_parquet("clean.parquet") or to_csv(index=False) |
Print len(df) before and after each merge. Fewer rows than you expected means an inner join dropped keys; more rows means duplicate keys on the right side multiplied your data.
dropna() with no subset · inner joins that silently drop rows (planted bug 3) · chained assignment df[df.x > 0]["y"] = 1 (use .loc) · apply with a Python function where a vectorized column operation exists · forgetting index=False when saving CSV.
Interview corner
Q. loc vs iloc? A. loc selects by label (index values, column names); iloc by integer position.
Q. Why did my row count change after merge? A. An inner join drops keys missing on either side; duplicate keys on the right side multiply rows. Check counts before and after.
Key takeaways
- A DataFrame is named columns of arrays.
- Say which columns are required when dropping NaN.
- Pick the join on purpose and check the row count.
Start from zero
New to this? Work through these steps first, in order. Already comfortable? Skim them and jump to the reference below.
Look before you model
Before any model, plot each column. You'll spot typos, outliers, gaps and skew that numbers alone hide.
A doctor looks at an X-ray before operating.
| Plot | Shows |
|---|---|
| Histogram | the shape of one column |
| Scatter | how two columns relate |
| Line | change over time |
Most model problems are data problems you could have seen in a plot.
Notebooks remember what you ran
A notebook keeps every variable from every cell you ran — in the order you ran them, not the order on the page.
A whiteboard that never erases: old scribbles stay and can confuse the next person.
- Run cell 1: load data.
- Run cell 3: count rows → 400.
- Edit and run cell 2: filter to 300 rows.
- Cell 3 still shows 400 until you rerun it.
A result you can't reproduce isn't a result. Interviewers ask about this.
Your teammate's notebook works for them but crashes for you at cell 5. Most likely cause?
Hidden state: they defined something in a cell they later deleted or ran out of order. Restart & Run All reproduces the crash.
Seeds make random numbers repeatable
Random number generators start from a seed. The same seed gives the same “random” numbers every time.
Shuffling cards in exactly the same way every time you say “42”.
import numpy as np print(np.random.default_rng(42).integers(0, 100, 5)) print(np.random.default_rng(42).integers(0, 100, 5))
# output
[ 8 77 65 43 43]
[ 8 77 65 43 43]np.random.default_rng(seed).Train/test splits and model initialisation are random. Seeds let you and reviewers get the same numbers.
Going further: the reference
The same ideas at the next level: precise definitions, tables, rules and interview questions.
Key definitions
- Notebook
- A document of code cells, outputs and notes (Jupyter), run one cell at a time.
- Kernel
- The Python process behind a notebook; it keeps variables between cells.
- Hidden state
- Variables left over from cells run earlier or out of order.
- Figure / Axes
- Matplotlib's canvas and one plot area on it.
- Histogram
- Counts of values in ranges (bins): the shape of one column.
- Reproducibility
- Anyone can rerun your code from scratch and get the same numbers.
Three plots you will make every week
import matplotlib.pyplot as plt import numpy as np rng = np.random.default_rng(0) price = rng.normal(40, 12, 500) qty = rng.integers(1, 6, 500) fig, axes = plt.subplots(1, 3, figsize=(12, 3.5)) axes[0].hist(price, bins=30) # distribution of one column axes[0].set_title("price") axes[1].scatter(price, price * qty, s=6) # relationship of two columns axes[1].set_xlabel("price"); axes[1].set_ylabel("revenue") axes[2].plot(np.cumsum(price * qty)) # trend over time / order axes[2].set_title("cumulative revenue") fig.tight_layout() fig.savefig("eda.png", dpi=120) print("saved eda.png", fig.get_size_inches())
# output
saved eda.png [12. 3.5]| Question | Plot | Matplotlib |
|---|---|---|
| What does this column look like? | histogram | ax.hist(x, bins=30) |
| Do two columns move together? | scatter | ax.scatter(x, y) |
| How does it change over time? | line | ax.plot(t, y) |
| How do groups compare? | bar | ax.bar(labels, values) |
| Outliers per group? | box plot | ax.boxplot(groups) |
A clean workflow
| Step | Do |
|---|---|
| 1 | One folder per project: data/, notebooks/, src/, requirements.txt |
| 2 | Explore in a notebook; move code you reuse into functions in src/ |
| 3 | Set seeds: rng = np.random.default_rng(42) |
| 4 | Restart & Run All before sharing a result |
| 5 | Save cleaned data to a file; never edit raw data in place |
| 6 | Commit code (not big data files) to git |
Notebooks are great for looking at data. Code that must run again — cleaning, features, training — belongs in .py files with functions you can test, exactly like prep.py in the lab.
The lab: three bugs in a data-prep script
Trusting a notebook without Restart & Run All · plots without labels or titles · editing raw CSVs by hand · no seed, so numbers change every run · huge notebooks that nobody can rerun.
Interview corner
Q. Why can't you trust a notebook that ran fine? A. Cells may have run out of order, leaving hidden state; a clean Restart & Run All proves the result is reproducible.
Q. First plots for a new dataset? A. A histogram per numeric column, value counts for categories, and scatter plots of features against the target.
Key takeaways
- Histogram, scatter, line: shape, relationship, trend.
- Restart & Run All before you trust a number.
- Explore in notebooks, keep code in .py files.
np.array([[1,2],[3,4]]).sum(axis=0)?
[4 6] — the rows collapse, one sum per column.Can shapes (5,3) and (3,) broadcast? (5,3) and (5,)?
What does df.dropna() drop?
Keep every order and add the region when known: which join?
orders.merge(customers, on="customer_id", how="left").loc or iloc for “rows where price > 10, column qty”?
df.loc[df.price > 10, "qty"] — labels and a mask.Revenue per month in one line?
df.groupby("month")["revenue"].sum()Your notebook gives a different answer after a restart. Why?
Why is a * 2 on an array faster than a list comprehension?
How do you inspect a new dataset quickly?
df.shape, df.dtypes/info(), df.isna().sum(), df.describe(), value counts for categories, then histograms.Explain broadcasting with an example.
How do you handle missing values?
What is the difference between merge and concat?
Why avoid apply() on large DataFrames?
View vs copy in NumPy?
How do you make an analysis reproducible?
What does groupby().agg() return?
| You see | Why | Fix |
|---|---|---|
| ModuleNotFoundError: No module named 'pandas' | Package not installed in the active environment | Activate the venv (source ~/ml/bin/activate) then pip install pandas |
| ValueError: operands could not be broadcast together | Shapes don't line up right-to-left | Print .shape of both; add an axis with [:, None] |
| SettingWithCopyWarning | Chained indexing assigned to a temporary copy | Use df.loc[mask, "col"] = value |
| Row count jumped after a merge | Duplicate keys in the right table | Check customers.customer_id.is_unique; deduplicate first |
| Column of numbers has dtype object | A stray text value (e.g. "n/a") | pd.to_numeric(col, errors="coerce"), then inspect the NaNs |
| Plot window never appears (WSL/SSH) | No display | Save with fig.savefig("plot.png") or use Jupyter |
| pip: externally-managed-environment | Installing into system Python on new Ubuntu/macOS | Create and activate a venv first |
| Resource | Why read it |
|---|---|
| Wes McKinney, Python for Data Analysis (3rd ed., O'Reilly) — free online at wesmckinney.com/book | By the author of pandas: NumPy and pandas in depth |
| NumPy user guide — Broadcasting (numpy.org/doc) | The exact broadcasting rules with diagrams |
| pandas user guide — 10 minutes to pandas; Merge, join, concatenate | Official quick start and the join reference |
| Jake VanderPlas, Python Data Science Handbook — free online | NumPy, pandas and Matplotlib chapters with examples |
Lab 01 · Fix a data-prep script
prep.py cleans 400 orders, standardizes three numeric columns and totals revenue per region. It runs without errors — and three of its answers are wrong. check.py tests six rules from the specification.
pip install -r requirements.txt, thenpython check.py. Read every FAIL line: which rule, and what number did you get?- Bug 1 (clean_orders): the row count is far too low. Which rows should survive?
- Bug 2 (standardize): column means are not 0. Which argument is missing?
- Bug 3 (revenue_by_region): regions don't add up to the total and there is no “unknown”. Which join keeps every order?
- Fix one bug at a time and re-run until every line says ok.
Done when you see PASS: all 6 checks passed — prep.py matches the specification.