Data Science, from zero, in 30 minutes

What data science is

Data science = combining data + methods (= maths, statistics, computer programs) + domain knowledge (= knowing the real field, for example how a café or a bus company works). The goal is understanding and action you can defend.

It is NOT only maintaining computers (that is IT work). It is NOT making charts without checking where the data came from. It does NOT replace human judgement.

The 5 steps from data to outcome

The 4 types of analysis (very important)

Association is not causation

10 records show more waste on rainy days. But on several rainy days, classes were also cancelled. So we cannot say "rain caused it". Two reasons:

• Small sample (= too few records, only 10)
• Overlapping factors (= another thing happened at the same time. The technical name is a confounder)

The correct attitude: the pattern is a reason to investigate, not proof.

What databases do

Database systems give: reliable structures (tables), relationships (= links between tables), access controls (= rules for who can see or change data) and traceable records (= you can follow where each record came from and how it changed). They do NOT make decisions, prove causes or choose trade-offs (= giving up some of one thing to get more of another).

The 4 kinds of prediction results

A "positive" means "the alarm says YES", whatever the alarm is about. Here the alert means "high waste".

Memory trick: the second word (positive/negative) is what the model said. The first word (true/false) says whether the model was right.

Real human review

A human check of a model is meaningful (= real, not just for show) only if the reviewer has:

• enough time and information to judge the recommendation, and
• the authority to change or reject it.

Bad review: "always accept when confidence is high", or a screen that hides the model's limitations (= weak points). This is called rubber-stamping (= approving without really checking).

The project lifecycle and defining the problem

A lifecycle (= the stages a project goes through from start to end). Remember two numbers:

Both are iterative (= you can go back to an earlier stage when you learn something new). Neither is a strict one-time order.

Your files don't list the names of the 8 teaching stages. If you need them, check lecture W1L2, slide 4.

Step 1 is always: define the problem

Problem formulation (= clearly stating the problem) comes before choosing algorithms, designing dashboards or collecting data. You must say:

• Who is the accountable (= responsible, must answer for results) user?
• What decision do they make, and when?
• What measurable (= can be counted or checked) outcome do we want?

"Use AI to improve student success" is too vague. It names no decision and no way to measure success.

A good target

The target (= the thing we try to predict) must be reproducible (= if anyone repeats it, they get the same answer). It needs three parts:

• an observable event (= something you can actually see happen): "a required assessment is missed"
• a time window: "in the next 14 days"
• a documented rule: "under the written deadline rule"

Bad targets use opinions ("seems uncommitted", "the adviser believes") or vague far-away things ("will eventually be successful").

The cut-off time rule

Every record has two times. The event time is when it happened. The receipt time is when it reached the database.

Using information that was not really available yet is called data leakage. It makes a model look better than it really is.

Baseline

A baseline (= a simple method you compare against, for example "same as last week"). You compare your candidate method (= the new method you are testing) with it to see if the new method adds real value. A complex method is NOT automatically better.

Fair testing

To know how a method will work in the future, test both methods on the same later, unseen cases (= data the model never saw while it was being built), using measures agreed in advance. A score on development data (= the data used to build the model) looks too good. This is called overfitting (= the model memorised old data and works worse on new data).

Success has several parts

After launch: monitoring

After deployment (= putting the model into real use), keep watching it. If the overall error stays the same but one group gets much worse, investigate that group and review under the monitoring plan. Don't ignore it, hide the group or wait.

Data preparation stage

This stage turns understood raw data into an analysis-ready dataset, and records every change. For example:

• applying a checked unit conversion, and writing it down
• building one row per unit with documented aggregation (= combining many rows into one: count, sum, average)

Deciding which decision to support belongs to problem formulation. Measuring the real outcome after launch belongs to evaluation and monitoring. Neither is preparation.

A good model can still be stopped

Pause or redesign the project, even if predictions are great, when:

• the data is not authorised (= not officially allowed) for this purpose, or
• the user has no feasible (= practically possible) action, or no capacity (= time, staff, resources) to use the result.

Data types, sources and collection

Grain: what one row means

Grain (= exactly what ONE row in a table represents). Say it precisely. A student reviewed every Monday gives: "one student at one weekly review cut-off". The same student on two Mondays = two rows, and they are not duplicates.

Measurement scales

Discrete (= whole numbers you count: 7 items, never 7.5) vs continuous (= any value in a range: 4.237 km). So item_count = 7 is a discrete numerical count.

Data sources: two separate questions

Ask two questions. 1. Where does it come from? Answer: internal or external. 2. Why was it collected? Answer: primary (for this study) or secondary (for some other purpose).

Data structure

Combining tables safely

Orders = 1 row per order. GPS = many rows per order. If you join (= connect tables by a shared ID) them directly, each order appears many times. This is row multiplication. The correct way:

• first define the aggregation from events to one row per order
• check the keys (= ID columns used for matching), the timing, and the unmatched records (= rows with no partner in the other table)

Sampling

A target population (= the whole group you want to make claims about) vs a sampling frame (= the list you actually choose people from). If the frame leaves out groups, the results cannot speak for everyone. Survey sent only to app users with delivered orders misses telephone orders and cancellations. This is a coverage problem. A big number of answers does not fix it.

A stronger sampling plan:

• sample across all relevant groups (districts, channels, statuses). This is stratified sampling (= split into groups, sample from each)
• compare invitation and response rates (= % of invited people who answered) between groups, to find non-response bias (= error because some kinds of people don't answer)

Weak plans: "a large response count proves it represents everyone"; "only invite customers who gave positive feedback before".

Metadata

Metadata (= data ABOUT data, explaining how to read it): units (km or miles?), what starts the delivery clock, time zone, the source system that produced the record. Not metadata: an actual value like "4.2" or a customer's comment. Those are the data itself. Related word: provenance (= where data came from, its history).

Data quality, privacy and governance

Data quality dimensions

A dimension (= one aspect or side of something). Learn these:

The 3 calculations in the test

Fixing bad values

distance_km = 4500 but the allowed range is 0–50. Maybe it was written in metres instead of km, but you don't know yet. So: flag it (= mark it as a problem) → verify the source unit → then make a documented correction. Don't overwrite it right away, don't cap it at 50, and don't ignore it.

Privacy words

• Pseudonymisation (= replacing names with codes). People can still be found through linkage (= combining with other data), especially precise location and time. So it is NOT anonymous, and access and disclosure (= sharing or revealing) still need controls.
• Anonymisation (= no one can be identified at all, in any way). This is much harder to achieve.
• Data minimisation (= collect only the data you really need for the stated purpose).
• Purpose: data must have an approved reason for its use. Encryption is security. It protects data, but it does not prove the data is necessary or the purpose is right.
• Retention (= how long data is kept). Keeping data "indefinitely (= forever, no end date) for possible future uses" is bad.

Governance roles

Governance (= the rules and people who manage data responsibly).

Memory trick: the steward looks after the meaning and quality of data. The custodian looks after the boxes it is stored in.

Release conditions (go / no-go)

A pilot (= small trial before full launch) needs, for example: at least 95% coverage in every route-period group, plus role-based access (= each person has their own account with rights based on their job). If two groups have only 60% and staff share accounts, then both conditions fail:

• A 96% overall average does not satisfy a per-group (= for each group separately) rule. Investigate and repair the weak groups.
• Missing readings are not zero passengers. Never fill them with 0.
• Replace shared accounts with individual approved access, and verify the controls before release.

And collect only what the decision needs. For monthly timetables by route, direction and 30-minute period: use validated occupancy counts (= checked counts of how full vehicles are) aggregated to that grain. Exclude card numbers, home addresses and continuous phone tracking.