A model is a proposed answer to a particular question
Imagine a fictional service trying to identify which support requests may need follow-up. Before choosing an algorithm, decide what “needs follow-up” means, when the prediction would be made and what information is available at that time. A vague target makes an impressive score difficult to interpret.
Data Science combines investigation, statistics, programming and model-building. In a data or product team, work can include preparing information, comparing an approach with a baseline and communicating uncertainty. Some questions need descriptive analysis rather than a predictive model at all.
Question a prediction that looks too good.
A team predicts next month’s cancellations. A tempting feature is the final cancellation confirmation—which is only known after the prediction should have been made.
Your role: Inspect a toy prediction, remove future information and review how a later-period test should work.
What you’ll learn: Check that a prediction uses only information available when the decision is needed.
No account needed. Nothing is sent or saved. You can restart at any point.
Read the full guided story without the interactive controls
You can follow the same lesson here without JavaScript.
Inspect the perfect-looking shortcut
Run the prediction that copies the final cancellation confirmation. Examine when that information becomes available.
What happens and why: Copying the outcome matches all four fictional records, but the confirmation is unavailable at prediction time. This is leakage: information from outside the permitted decision-time data enters the experiment.
Use information available at the decision
Keep prior usage and exclude the later confirmation. Account age is available context, but it is optional and unused by this simple rule.
What happens and why: The rule predicts cancellation when prior usage is below three visits. It matches three of four fictional later records. This fixed teaching rule is not a trained or validated business model.
Keep later records for a later-period check
Choose the time boundary between earlier development data and the later evaluation period.
What happens and why: For this time-dependent scenario, develop using January–March and reserve April–June. Mixing later outcomes into development can hide temporal changes. Split strategy depends on the real deployment setting.
Report an honest experiment, not a promise
Review the available features, time split and four-record result before writing a cautious conclusion.
What happens and why: Three out of four is an observation in a tiny fictional example, not evidence of real-world accuracy or fairness. Validate transformations, availability, drift and performance on appropriate independent data.
Takeaway: A perfect-looking result can come from information that would not exist when a prediction is needed. Respect decision time, reserve suitable evaluation data and state the limits of a small experiment.
Where this is useful: Useful when reviewing forecasting, risk models and machine-learning evaluation.
A fictional educational example, not a confirmed TWA assignment, assessment or learner outcome. Simplified to explore one idea; real situations need context and authorised procedures.
Optional deeper exploration: open the original technical workspace
Want to explore the technical model?
The guided story above introduces the idea. This original workspace lets you try actions in different orders. It has its own reset and does not inherit your story choices.
- Feature
- Information used to make a prediction.
- Leakage
- Using information the real prediction process would not legitimately have.
- Holdout
- Data kept apart from development for evaluation.
- Temporal change
- A change over time that can affect future performance.
A great score can hide a bad experiment.
Learning objective: Identify unavailable future information and inappropriate data splits.
A fictional team predicts whether a customer will cancel next month, at the start of that month.
Evidence desk
- Available now
- Prior usage and account age
- Only known later
- Final cancellation confirmation
- Evaluation
- Future deployment must work on later periods
Your workspace
Worked explanation & takeaway — available without JavaScript
Exclude final cancellation confirmation: it is unavailable at prediction time and leaks the target. A chronological holdout better reflects later deployment in this time-dependent scenario. No accuracy number is computed; removing leakage can lower an inflated score while making evaluation more credible.
Fictional educational model. No live system, personal assessment or confirmed TWA assignment. All actions run locally in your browser.
Would you know this at prediction time?
The example is fictional and contains no real model score or customer information.
| Candidate input | Available at arrival? | Question to ask |
|---|---|---|
| Submitted category | Potentially | Is it present and defined consistently? |
| Replies after closure | No | Does this use information from the future? |
| Processing time after completion | No | Is the experiment answering the real-time question? |
Potentially available at the stated prediction point. Check data quality and whether using it is appropriate; availability does not prove predictive value.
Read both scenarios and explanations
Category supplied when the request arrives
Potentially available at the stated prediction point. Check data quality and whether using it is appropriate; availability does not prove predictive value.
Replies counted after the case closes
Not available when the new case arrives. Using it for that prediction leaks future information and can mislead the evaluation.
An educational illustration using fictional situations. It is not a confirmed TWA assignment, live system, assessment or promise of a particular tool.
Understand the data and evaluation boundary
Useful foundations include Python, data cleaning, summary statistics, probability and asking testable questions. TWA’s existing direction includes these areas, exploratory analysis, visualisation, regression/classification/clustering, validation, responsible data use and communication. The starting point depends on prior programming, maths and data experience.
scikit-learn’s guidance explains why preprocessing must be applied consistently and why learning from test information can produce optimistic evaluation. These are industry concepts, not a promise of a particular library or fixed assessment. A strong first explanation should distinguish the data used to develop a model from the data used to evaluate it.
Find the future information hiding in a feature
In the fictional follow-up example, “number of replies after case closure” would not be known when a new case arrives. Including it as an input can make a historical experiment appear easier than the real task. The model has received information from after the decision it is supposed to support.
The illustration below asks whether a field is available at the prediction point. It does not claim that one remaining field is a good predictor. Define the timing, separate development and evaluation appropriately and choose measures that reflect the consequences of errors. If records are time-dependent or grouped, an arbitrary split may need more thought.
A prediction-readiness and model-evaluation note
Use a fictional case dataset to define a prediction point, identify unsuitable future information and propose a baseline and evaluation plan.
What you could document
- A question/target and feature-timing definition
- A data-preparation and split rationale
- A baseline/model comparison or documented experiment plan
- A findings report with error analysis and limitations
What makes the explanation useful?
The result should make leakage, evaluation choices and uncertainty understandable. It should not claim deployment readiness or business benefit from one score.
The learning plan determines actual activities and assessment. These deliverables are examples, not contractual curriculum outcomes.
A good model report includes an unglamorous baseline
Compare the proposed approach with a simple documented rule. Record the dataset, target definition, splitting/evaluation choices and known limitations. Explain which kinds of mistakes matter and why. A higher number is not automatically a better decision if its meaning, data coverage or cost is unclear.
Deeper directions can include applied machine learning, predictive analytics and more specialised statistical work. Data Engineering focuses on dependable data delivery; AI Engineering may focus on applications around models. Discuss the direction you want with TWA without assuming that every Data Science plan includes deployment, advanced maths or a fixed three-level syllabus.