Split before learning

Evaluation data must simulate future unknown cases.

8 minutes. Mission: measure generalization without leakage.

Three different jobs

Train fits model parameters. Validation guides model and threshold choices. Test gets one final use after choices are frozen.

Repeatedly choosing against test results makes the test part of training. Google calls out that validation and test sets can “wear out” through repeated use.^[1]

Choose split by deployment

Random rows work only when examples are independent and future data resembles the sample. Split by customer when one customer has many rows. Split by time when predicting future periods. Duplicates and preprocessing learned before splitting leak information.

Leakage test Could this value or statistic exist when the real prediction is made? If not, exclude it or move its computation inside each training fold.

Check retrieval

Win: you can choose random, grouped, or temporal splitting from deployment conditions.

Primary source

Read Google: Dividing the original dataset, then scan scikit-learn’s cross-validation guide.

PreviousNext: fit the first model