Data & AI Engineering
Begin with one explainable decision and hand-calculable baseline, then earn more complex statistics, models, retrieval, serving, monitoring, privacy, human control, and recovery claims.
Problem definition and data audit
Objective Run one hand-checkable decision baseline and distinguish unit, prediction time, label, action, error costs, and leakage.
Core explanation
An AI project begins with a decision, not a model. Name the decision maker, affected population, possible action, intended outcome, and cost of each error. A prediction unit is the exact thing represented by one row; the first laboratory uses one learner-course-week. Prediction time is the boundary after which information is unavailable to the decision. Features are facts available by that boundary, while the label is the later outcome used to evaluate the prediction. Leakage occurs when a feature contains that outcome, other future information, or a transformation fitted using evaluation data. The baseline flags support after at least one missed deadline, and a confusion matrix compares its decisions with four frozen later outcomes. Recall answers how many learners who needed support the rule found; it does not show that the support caused improvement. The audit therefore rejects a deliberately prohibited next-week score field. This standard-library experiment is an explainable comparison point, not a trained model or causal claim; later chapters earn each additional layer of complexity through explicit evidence.
Begin with a decision, not with a model
A data project exists to improve a decision. Before opening a notebook, write who makes that decision, what action they may take, which person or system is affected, when the action occurs, and what a better outcome means. “Predict students at risk” is incomplete: it does not say whether an adviser will offer tutoring, whether the prediction happens before enrollment or after the third week, or whether false alarms consume scarce support. A usable statement might be: every Monday morning, advisers review currently enrolled learners and may offer an optional study session; the system ranks cases using only information available by Sunday night. That sentence establishes actor, population, prediction time, intervention, and availability boundary.
Separate prediction from causation. A model may estimate who is likely to miss a deadline, but that does not prove which intervention will help them. A feature can predict an outcome because it records an underlying cause, a consequence, a collection artifact, or an inequitable process. Write the target as an observable event with a time window, not a vague label such as “success.” Then define a simple baseline and the cost of each error. If support is harmless and plentiful, missing a learner may be worse than reviewing extra cases. If an intervention is intrusive or limited, false positives matter more. The metric and threshold must follow this decision table; they cannot be selected only because a library reports them.
Define unit, population, time, and target without leakage
The unit is the thing represented by one prediction row: learner, enrollment, course session, account, or event. Mixing units creates subtle errors. One learner may have several enrollments, and randomly splitting enrollment rows can place the same person in training and evaluation. The model then recognizes the person rather than generalizing to unseen learners. State the eligible population and exclusions explicitly. Ask who never enters the dataset: people without reliable internet, learners who declined tracking, users of an older client, or those whose records were deleted. A result applies to the observed sampling and measurement process, not automatically to everyone the product hopes to serve.
Draw a timeline for every candidate field. Event time says when something happened; recording time says when the system received it; availability time says when the production decision could safely use it. A final grade recorded in June cannot be used for an April intervention, even if it appears on the same historical row. Labels also need a cutoff: a course completion label may remain unknown for weeks, and treating unknown as failure changes the task. Build each training example “as of” its prediction timestamp and join only records whose availability precedes that timestamp. Test this with deliberately late records and boundary timestamps. Leakage usually produces unusually impressive evaluation, so an exceptional score should trigger more skepticism, not celebration.
Audit provenance, quality, representation, and rights
Create a source inventory before cleaning. For each table, file, stream, or external API, record its owner, purpose of collection, collection method, update cadence, schema version, stable identifiers, time semantics, retention, access restrictions, and known outages. Inspect raw examples and distributions, but keep raw material immutable so every transformation can be reproduced. Distinguish a missing value from zero, “not applicable,” “not yet observed,” “collection failed,” and “person declined.” These mechanisms carry different information and require different handling. Count duplicates by the domain identity, not merely identical rows, and investigate whether repeated records are legitimate events, retries, corrections, or ingestion defects.
Data suitability includes law, consent, fairness, and user expectations, not only file quality. Minimize fields to those justified by the decision. Sensitive or proxy attributes may be needed to audit unequal performance even when they must not drive the model; define separate access and retention policies. Ask whether the intervention could disadvantage a group, create a self-fulfilling label, or shift staff attention away from people the data underrepresents. Record these risks in a data sheet with prohibited uses. If provenance, authorization, target validity, or representation is too weak, the correct output of the audit is “do not model yet,” followed by a plan to improve measurement or redesign the decision.
Turn the audit into executable acceptance evidence
A useful audit ends with checks another person can run. Freeze a small, privacy-safe fixture containing ordinary, empty, boundary, duplicate, late, contradictory, and malformed cases. Add assertions for schema, identifier uniqueness at the declared grain, allowed ranges, relationships, time ordering, target availability, cohort coverage, and reconciliation totals. Produce a table showing how many records each rule accepts, rejects, quarantines, or leaves unknown. A schema checker alone is insufficient: a timestamp can have the right type while occurring after the prediction cutoff, and a valid category can have a newly changed meaning. Semantic checks should name stable error codes and source versions so failures can be investigated without printing private rows.
Finish with a go/no-go review. The document should contain the decision contract, population and unit, time diagram, target construction, baseline, error-cost table, source inventory, quality findings, leakage tests, group coverage, privacy and harm review, unresolved assumptions, and an owner for each action. Give the artifact to another learner and ask them to add a new data source. They should be able to determine whether its timestamps are usable, how identities join, which permission applies, and which checks must change. If they need unwritten knowledge from the original analyst, the project is not reproducible and the audit is not complete.