Scoping a Machine Learning Project So It Survives Contact With Reality

Begin with the decision

The first question is never which model. It is: what decision changes because of this system, and who makes it?

If no decision changes, the project produces a dashboard nobody opens. If the decision-maker will not act on a probabilistic output, accuracy is irrelevant — you have a change-management problem wearing a technical costume.

Write the sentence: "When the system outputs X, [role] will do Y instead of Z." If that cannot be completed, stop and fix the framing before writing any code.

Establish a baseline before modelling

You cannot claim improvement without knowing the starting point. Three baselines are worth measuring:

• Current process performance — how often does the human, or the existing rule, get it right?
• Trivial predictor — always predict the majority class, or persistence for a time series
• Simple rules — a handful of thresholds a domain expert would write in an afternoon

Simple baselines are startlingly competitive. When a rules engine reaches 85% and a neural network reaches 87%, the rules engine may still be the correct choice once interpretability, latency and maintenance are counted.

Interrogate the labels

Supervised learning requires labelled examples, and this is where projects quietly die.

• Do labels exist, or must they be created? Manual labelling is frequently the largest line item in the budget.
• Are they consistent? Have two annotators label the same hundred examples. Disagreement above roughly 10% caps achievable accuracy and needs resolving through a clearer definition, not through more data.
• When do they arrive? If the outcome is only known 60 days later, that determines the retraining cadence and the entire feedback loop.
• Are they contaminated? A field populated by the very process you are automating leaks the answer. Models trained on it look extraordinary in evaluation and fail immediately in production.

Check for leakage explicitly

Leakage is the most common cause of a model that tests brilliantly and deploys terribly. Ask of every feature: would this value be available, with this value, at the moment of prediction?

Classic sources are fields updated after the outcome is known, aggregates computed over the full dataset before splitting, identifiers correlated with the target through collection order, and random splits applied to data with temporal structure. Time-ordered data requires a time-ordered split, always.

Define success as a threshold, not a direction

"Improve accuracy" is unfalsifiable. "Reduce manual review volume by 40% while keeping false negatives below 2%" can be tested and can be failed.

Choose the metric to match the cost structure. Accuracy is misleading under class imbalance; precision, recall and the trade-off between them usually matter more, and the right operating point depends on the relative cost of each error type. Decide this before seeing results, or the threshold will drift to wherever the model happened to land.

Plan for the model getting worse

Deployment is the beginning of the work, not the end. Data distributions drift, upstream systems change format without warning, and behaviour adapts to the model's presence.

Before launch, decide what is monitored, what threshold triggers investigation, who owns the response, how retraining happens, and how to roll back. A model nobody is watching is a liability accruing quietly.

A short scoping checklist

1. The decision that changes, written as a complete sentence
2. A measured baseline
3. Label availability, consistency and latency
4. An explicit leakage review
5. A success threshold with a chosen operating point
6. A monitoring and rollback plan

If any of the six is missing, the project is not ready to start — and starting anyway is how the expensive discoveries get made late.