TablePipeline

Note

This page is a reference documentation. It only explains the class signature, and not how to use it. Please refer to the Habitat Guide and Python API guide (v2.0) for usage.

class TablePipeline(steps: Sequence[Any], model: Classifier | Regressor | SurvivalModel | None = None, *, classifier: Classifier | None = None, **pipeline_options: Any)[source]

Bases: Pipeline

Fitted preprocessing/selection chain plus model over feature tables.

The structural answer to the train/predict leakage class of bugs: the preprocessing and feature-selection steps are fitted ONCE on the training table and their fitted state is what predict/transform apply to any later table – the prediction data is normalised with the TRAINING statistics and reduced with the TRAINING selection, never re-fitted.

The fitted pipeline is also the artefact a study publishes for external validation of its tabular model, which is why save() persists the steps and the model together in one versioned, self-describing file (a JSON manifest recording every component’s Spec alongside the pickled fitted state).

This IS an ``sklearn.pipeline.Pipeline``. Subclassing rather than re-implementing composition is what gives HABIT’s tabular models clone, get_params/set_params, nested parameter addressing and therefore GridSearchCV / RandomizedSearchCV / cross_val_score for free, instead of a second composition engine that would drift from the one the rest of the ecosystem uses.

Two consequences a caller must know:

.steps has sklearn’s meaning – List[Tuple[str, estimator]], where the estimators are the interop adapters. It is NOT overridden, because sklearn’s _iter / _validate_steps / get_params / set_params all read and WRITE it directly. The HABIT components are reached through components (transformation steps) and model (the terminal one).
The step list always begins with a FrameToTable head named "frame_to_table" and ends with the outcome-model adapter named "model". The head is what lets an sklearn cross-validation driver pass a plain frame as X (a FeatureTable is a frozen dataclass and deliberately not row-sliceable); when the pipeline is handed a FeatureTable directly – HABIT’s own entry point – it passes straight through, with no frame round-trip and therefore no dtype promotion that could shift a later z-score.

HABIT’s verbs are kept as overrides, because a FeatureTable in must give a HABIT type out: fit(), transform(), predict() and predict_proba() return tables / labelled Series / labelled frames for FeatureTable input, and plain arrays for frame input (what an sklearn scorer expects). evaluate(), predict_survival_function(), set_random_state(), spec(), save() and load() have no sklearn equivalent and are unchanged.

Parameters:

steps – Either the HABIT form – ordered transformation components (TablePreprocessor and/or FeatureSelector implementations), optionally preceded by a FrameToTable declaring the frame schema; may be empty, in which case the pipeline is the bare model – or the sklearn form [(name, estimator), ...], which is what clone, set_params(steps=...) and slicing pass back in.
model – The terminal outcome model – a Classifier, Regressor, or SurvivalModel, matched to the endpoint family of the tables it will be fitted on. Must be omitted when steps is already in sklearn form (the terminal step carries it).
classifier – Deprecated alias for model (binary/multiclass endpoints); kept so existing call sites keep working.
**pipeline_options – Forwarded verbatim to sklearn.pipeline.Pipeline (memory, verbose, and transform_input on scikit-learn >= 1.6).

Examples

Nested hyperparameter search over a HABIT component’s own parameter:

from sklearn.model_selection import GridSearchCV

from habit.pipeline.sklearn_interop import FrameToTable

pipe = TablePipeline(
    steps=[FrameToTable.from_table(train), ZScorePreprocessor()],
    model=LogisticRegressionClassifier(),
)
search = GridSearchCV(pipe, {"model__component__C": [0.1, 1, 10]})
search.fit(train.frame, outcome_series(train))

__init__(steps: Sequence[Any], model: Classifier | Regressor | SurvivalModel | None = None, *, classifier: Classifier | None = None, **pipeline_options: Any) → None[source]

property components: Tuple[TablePreprocessor | FeatureSelector, ...]

Return the ordered HABIT transformation components.

This is the successor of the pre-v1.1 .steps property. .steps now means what sklearn means by it, so the HABIT components – the objects carrying spec, the fitted statistics and the selected column names – are read here instead. The FrameToTable head and the terminal model adapter are excluded: the head is interop plumbing, and the model is model.

Returns:: The unwrapped components, in execution order.
Return type:: Tuple

property model: Classifier | Regressor | SurvivalModel

Return the terminal outcome model.

Returns:: The HABIT Classifier / Regressor / SurvivalModel.
Raises:: HABITAPIError – When the pipeline does not end in a HABIT outcome model – which happens for a slice such as pipe[:-1].

property classifier: Classifier: Return the terminal model, asserted to be a classifier.

property frame_schema: FrameToTable

Return the FrameToTable head step.

The one place to read or re-declare the column schema an sklearn driver’s frames follow. Prefer pipe.set_params(frame_to_table=FrameToTable.from_table(table)) to replace it, so the change goes through sklearn’s own parameter machinery and survives clone.

Raises:: HABITAPIError – When the pipeline has no FrameToTable head.

property spec: Spec

Return the composed specification of every stage.

The payload shape – {"steps": [...], "model": {...}} – is the FINGERPRINTED one and is deliberately unchanged by the move to sklearn.pipeline.Pipeline: the FrameToTable head and the adapters are interop plumbing, not scientific definition, so they do not appear. Renaming or reshaping this payload would move every recorded provenance fingerprint in the repository.

Raises:: HABITAPIError – When a step carries no Spec, i.e. the pipeline holds a foreign estimator that HABIT cannot describe.

set_random_state(seed: int) → None[source]

Seed every stochastic component of the pipeline.

Propagates to each transformation component and the terminal model implementing Seedable; deterministic components are untouched (v1.0 naming decisions: one seeding verb, never a constructor parameter).

Parameters:: seed – The seed to install.

fit(X: Any, y: Any | None = None, *, repeat_tables: Sequence[FeatureTable] | None = None, **params: Any) → TablePipeline[source]

Fit every step in order, then the terminal model.

Each step is fitted on the table produced by the previous step, so learned statistics compose exactly as they will at predict time.

Parameters:

X – Training data. A FeatureTable (HABIT’s entry point: the outcome rides inside it and passes through the head step untouched) or a plain frame carrying the identifier, feature and outcome columns the FrameToTable head declares.
y – Training targets. Normally None for a FeatureTable, whose outcome column already supplies them; sklearn’s cross-validation drivers pass the sliced label array, which is cross-checked against the table’s own outcome so a misaligned y fails loudly.
repeat_tables – Optional aligned repeat-measurement tables. Routed ONLY to the steps whose fit declares repeat_tables (the test-retest / ICC selectors); the rest never see the keyword.
**params – scikit-learn step-scoped fit parameters in stepname__param form, forwarded unchanged.

Returns:

self, fitted.

transform(X: Any, **params: Any) → FeatureTable[source]

Apply the fitted transformation chain to a table.

Deliberately narrower than sklearn.pipeline.Pipeline.transform, which also calls the FINAL step: a TablePipeline always ends in an outcome model, which has no transform, so sklearn’s version is never available on this class anyway. What callers have always meant by pipeline.transform(table) – “the classifier-ready table” – is what this returns.

Parameters:

X – Table (or frame) carrying the feature columns seen at fit time; each fitted step validates its own input schema.
**params – Accepted for signature compatibility; must be empty.

Returns:

The table after every fitted transformation step.

Return type:

FeatureTable

Raises:

HABITAPIError – If the pipeline is not fitted, or params is non-empty (per-step transform parameters would silently do nothing here).

predict(X: Any, **params: Any) → Any[source]

Predict the terminal model’s output for a table’s rows.

Class labels for a classifier, values for a regressor, risk scores for a survival model (routed through predict_risk).

Parameters:

X – Data to predict; transformed with the fitted state first.
**params – scikit-learn predict parameters, forwarded to the terminal adapter on the array path only.

Returns:

pd.Series indexed by the table’s identifier columns for FeatureTable input (HABIT semantics), or a plain ndarray for frame input, which is what an sklearn scorer expects.

predict_proba(X: Any, **params: Any) → Any[source]

Predict class probabilities for a table’s rows.

Only meaningful for a classifier terminal model; regressors and survival models have no class-probability output.

Parameters:

X – Data to predict; transformed with the fitted state first.
**params – scikit-learn predict parameters, forwarded to the terminal adapter on the array path only.

Returns:

A probability frame indexed by the identifier columns, one column per class, for FeatureTable input; a plain ndarray with columns aligned to self.classes_ for frame input.

Raises:

HABITAPIError – If the terminal model is not a classifier.

predict_survival_function(table: FeatureTable, times: ndarray) → DataFrame[source]

Predict per-subject survival functions at the requested times.

Parameters:

table – Table to predict; transformed with the fitted state first.
times – Ascending 1-D grid of evaluation times.

Returns:

Survival probabilities, one row per subject, one column per time.

Raises:

HABITAPIError – If the terminal model is not a survival model.

evaluate(table: FeatureTable, metrics: Sequence[Metric | RegressionMetric | SurvivalMetric]) → Dict[str, float][source]

Score the pipeline on a labelled table.

Dispatches by the table’s endpoint family:

binary / multiclass – classification Metric objects; probability metrics receive the positive-class scores (column "1" for a 0/1 outcome, else the last class column).
continuous – RegressionMetric objects on (true, predicted).
survival – SurvivalMetric objects; risk-based metrics get predict_risk, function-based ones get predict_survival_function evaluated on a grid derived from the follow-up range.

Parameters:

table – Evaluation table carrying the endpoint column(s).
metrics – Metrics to compute, keyed in the result by metric.spec.name. Must match the endpoint family.

Returns:

Mapping of metric name to value.

Raises:

HABITAPIError – If metrics is empty, the table has no endpoint, or a metric family does not match the endpoint.

save(path: str | Path) → Path[source]

Persist the fitted pipeline in a versioned, self-describing format.

The .habitpipeline file is a ZIP archive holding a JSON manifest (format name, format version, producing HABIT version, and every component’s spec and class path) plus the pickled fitted state. The manifest keeps the artefact inspectable without deserialising it.

What is pickled is the HABIT COMPONENTS, not the sklearn adapters wrapping them: the components are the fitted science, the adapters are interop plumbing that load() rebuilds. Keeping the payload at the component level is also what lets one loader read both format version 1 (written before this class became an sklearn.pipeline.Pipeline) and version 2.

Parameters:: path – Destination file path.
Returns:: The path written.

classmethod load(path: str | Path) → TablePipeline[source]

Load a pipeline previously written by save().

Version 3 is the first v2 archive format. Versions 1 and 2 pickle v1 habit.domain classes, so they are explicitly rejected before unpickling with migration guidance. A file whose format_version this build does not know is never guessed at: silently loading a wrong-but-plausible pipeline would produce numbers nobody could trace.

Security note: the fitted state is pickle-serialised (the standard serialisation for sklearn estimators), so only ever load pipeline files from sources you trust.

Parameters:: path – Source file path.
Returns:: The loaded pipeline, fitted exactly as when saved.
Raises:: CompatibilityError – If the file is not a HABIT table pipeline, was written with a newer format version, or its manifest does not match its payload.

set_fit_request(*, repeat_tables: bool | None | str = '$UNCHANGED$') → TablePipeline

Request metadata passed to the fit method.

Note that this method is only relevant if enable_metadata_routing=True (see sklearn.set_config()). Please see User Guide on how the routing mechanism works.

The options for each parameter are:

True: metadata is requested, and passed to fit if provided. The request is ignored if metadata is not provided.
False: metadata is not requested and the meta-estimator will not pass it to fit.
None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.
str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Note

This method is only relevant if this estimator is used as a sub-estimator of a meta-estimator, e.g. used inside a Pipeline. Otherwise it has no effect.

Parameters:: repeat_tables (str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED) – Metadata routing for repeat_tables parameter in fit.
Returns:: self – The updated object.
Return type:: object

set_score_request(*, sample_weight: bool | None | str = '$UNCHANGED$') → TablePipeline

Request metadata passed to the score method.

Note that this method is only relevant if enable_metadata_routing=True (see sklearn.set_config()). Please see User Guide on how the routing mechanism works.

The options for each parameter are:

True: metadata is requested, and passed to score if provided. The request is ignored if metadata is not provided.
False: metadata is not requested and the meta-estimator will not pass it to score.
None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.
str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Note

This method is only relevant if this estimator is used as a sub-estimator of a meta-estimator, e.g. used inside a Pipeline. Otherwise it has no effect.

Parameters:: sample_weight (str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED) – Metadata routing for sample_weight parameter in score.
Returns:: self – The updated object.
Return type:: object