When the data is too big for your laptop it lives in a database — and the language to ask it questions is SQL. Three patterns cover most of your needs
Day 34 of 66
A CSV on your laptop holds last week's labels. But a lab's full annotation history — millions of rows across many tasks and model versions — lives in a database you can't load into pandas. To ask it questions you use SQL (Structured Query Language). The good news: if you understand pandas groupby, you already understand the hard part of SQL. They're the same ideas in different clothes.
SQL queries say: SELECT these columns FROM this table WHERE these conditions hold, GROUP BY this column, computing these aggregates. That clause order is 90% of what a data lead ever writes. Everything else is detail.
SELECT item_id, score FROM labels WHERE score < 3 returns just the columns you name, for just the rows that pass the condition. This is pandas' df[df.score < 3] — the "show me the failures" query.
SELECT annotator, COUNT(*), AVG(score) FROM labels GROUP BY annotator collapses the table to one row per annotator with their volume and average. This is your per-annotator profile — exactly pandas' groupby("annotator")["score"].agg([...]).
SELECT * FROM items JOIN labels ON items.item_id = labels.item_id glues two tables together on their shared key. This is pandas' pd.merge(items, labels, on="item_id"). Same join, same purpose: pull a dimension from one table onto rows in another.
Now the real question a lead actually asks: "every output rated below 3 last week, broken down by failure type." Watch it become SQL — it's just the three patterns stacked.
SELECT i.failure_type,
COUNT(*) AS n,
AVG(l.score) AS avg_score
FROM labels l
JOIN items i ON l.item_id = i.item_id -- pull failure_type in
WHERE l.score < 3 -- only failures
AND l.labeled_at >= '2026-06-09' -- only last week
GROUP BY i.failure_type -- one row per type
ORDER BY n DESC; -- worst offenders firstJOIN brings failure_type from the items table, WHERE keeps only last week's sub-3 scores, GROUP BY collapses to one row per failure type, and COUNT/AVG summarize each. The l and i are short aliases for the tables. This single query is a full weekly failure report.They overlap, so the choice is about where the data is and how big it is, not which you like better.
| Reach for SQL when… | Reach for pandas when… |
|---|---|
| The data lives in a warehouse / database | You have a CSV or a small export on disk |
| It's too big to fit in memory (millions of rows) | It fits comfortably in memory |
| You want one shared, repeatable query others can run | You're exploring ad-hoc and iterating fast |
| You need to join across many stored tables | You want charts, custom functions, or quick reshaping |
The fastest way to feel the equivalence: take the per-annotator profile you built in pandas on Day 33 and rewrite it as SELECT annotator, COUNT(*), AVG(score) FROM labels GROUP BY annotator. Run both against the same data and confirm the numbers match. Once you've seen one result come out identical in both tools, SQL stops being scary — it's just groupby with different keywords.
COUNT(*) and AVG(score) per annotator.No database handy? Python's built-in sqlite3 needs zero setup, or load your CSV into a quick in-memory table with df.to_sql and query it. The point is writing the three patterns, not the engine.
An IC is capped by what fits in a CSV on their laptop. A lead goes to where the full history actually lives — millions of rows across every task and model version in the warehouse — and asks it the same questions in SQL, so their findings cover the whole operation instead of last week's sample.
Say this in an interview: "I don't wait for someone to export me a slice. I'll write the SELECT, WHERE, GROUP BY and JOIN against the warehouse myself, so my failure breakdowns are over the full annotation history — not just whatever fit on my machine."