cross_validate

Note

This page is a reference documentation. It only explains the function signature, and not how to use it. Please refer to the Habitat Guide and Python API guide (v2.0) for usage.

cross_validate(table: FeatureTable, spec: MLSpec, *, n_splits: int = 5, seed: int | None = None, inner_cv: int | None = None, param_grid: Mapping[str, Sequence[Any]] | Sequence[Mapping[str, Sequence[Any]]] | None = None, strategy: str = 'grid', n_iter: int = 10, objective: str | None = None) → CVResult[source]

Estimate generalisation with stratified K-fold cross-validation.

Every fold builds a FRESH pipeline from the spec and fits it on that fold’s training rows only, so preprocessing statistics and feature selection can never leak in from the validation rows. Folds are stratified on the outcome when the endpoint family has strata (binary, multiclass, survival).

Nested cross-validation. Passing inner_cv together with a param_grid re-tunes the hyperparameters inside every outer fold, on that fold’s TRAINING rows only, and scores the winner on the untouched validation rows. The reported panel then estimates the whole tuning PROCEDURE, which is the quantity a reviewer asks for: tuning once on all the data and cross-validating afterwards reuses the validation rows for selection and reports an optimistically biased number. The two arguments are required together – a grid without inner_cv would have to tune on the outer validation rows, and inner_cv without a grid would tune nothing.

Parameters:

  • table – Feature table with a declared outcome.

  • spec – The modelling definition to evaluate. Under nested CV this is the definition MINUS the tuned parameters: each outer fold overwrites the searched ones with its own winners.

  • n_splits – Number of OUTER folds; must be at least 2.

  • seed – Optional seed override, folded into the spec (and therefore into the fold shuffling, the component seeding, and the manifest) before anything runs.

  • inner_cv – Number of INNER folds used to tune inside each outer fold’s training rows; must be at least 2. None (the default) runs plain cross-validation with no tuning.

  • param_grid – The search space, in the key syntax search_hyperparameters() documents.

  • strategy – "grid" or "random"; see search_hyperparameters().

  • n_iter – Candidate budget for strategy="random".

  • objective – Registered metric name the inner search maximises (or minimises, honouring the metric’s own direction); see search_hyperparameters().

Returns:

Per-fold panels, their across-fold summary, the per-fold fitted pipelines, and – under nested CV – each fold’s winning parameters.

Raises:

HABITAPIError – If the table declares no outcome, n_splits or inner_cv is below 2, a stratum is smaller than the fold count, or exactly one of inner_cv / param_grid is given.

Examples

>>> from habit.datasets import make_synthetic_feature_table
>>> from habit.spec import MLSpec, Spec
>>> import habit.recipes as recipes
>>> table = make_synthetic_feature_table(n_rows=60, n_features=8, rng=42)
>>> spec = MLSpec(
...     name="demo",
...     steps=(Spec("zscore"),),
...     classifier=Spec("LogisticRegression", {"max_iter": 500}),
...     metrics=(Spec("accuracy"), Spec("auc")),
... )
>>> result = recipes.cross_validate(table, spec, n_splits=3, seed=42)
>>> result.n_splits
3
>>> sorted(result.mean_metrics)
['accuracy', 'auc']