Compare honestly

Prefer repeated evidence over one lucky split.

10 minutes. Mission: compare candidates and diagnose fit.

Cross-validation rotates validation rows

from sklearn.model_selection import cross_validate

scores = cross_validate(
    model, X_train, y_train, cv=5,
    scoring="roc_auc", return_train_score=True,
)

Each fold fits the entire pipeline on different training rows and evaluates on its held-out fold. Report the score distribution, not only its mean.^[1]

Read the gap

Low train and validation: underfit, weak features, noisy target, or metric mismatch. High train, lower validation: overfit or split mismatch. Similar and useful: candidate generalizes under this validation design.

Model quality is conditional on the split. Time and grouped data need matching cross-validators; ordinary random folds can give false confidence.

Check retrieval

Win: you can compare score distributions and diagnose the train-validation gap.

Primary source

Read scikit-learn’s cross-validation guide, especially grouped and time-series sections.

PreviousNext: make the go decision