Execution backends

Execution backends and checkpoints: optional accelerators, never required.

An entire study can run without ever constructing anything from this package: op(subject) handles one subject and Cohort.map(op) handles the cohort with an implicit serial backend. Explicit backends are constructed only when parallelism, per-subject timeouts, or resume are actually wanted.

Integrator chapter (which backend, continue vs fail_fast, resume): Parallel execution and fault tolerance. Runnable demos: 7. Running a whole cohort. YAML twins: Spec, RunPolicy, and YAML isomorphism. Habitat CLI / recipe wiring: Habitat Segmentation Configuration.

Classes

`SerialBackend`	Run subject-level work one item at a time in the current process.
`ProcessPoolBackend`	Execute subject-level work across child processes.
`SubjectTimeoutError`	A subject exceeded its wall-clock or spawn-startup budget.
`CheckpointStore`	File-based persistence for subject-level results, keyed by cache key.
`LegacyCheckpointMigrationReport`	Outcome of one v0.1 → v1 checkpoint migration attempt.

Functions

`backend_from_policy`	Build the execution backend a policy asks for.
`should_use_process_pool`	Return whether the policy requires a spawn process-pool backend.
`is_v01_checkpoint_layout`	Return whether `root` holds a v0.1 checkpoint tree.
`migrate_v01_checkpoint_if_needed`	Detect a v0.1 checkpoint under `root` and migrate it to v1 entries.

Backend selection from CLI and run_from_yaml

habit.execution.backend_from_policy (shared by cmd_habitat and run_from_yaml) selects:

policy.backend == "process" → from_policy
or workers > 1 → ProcessPoolBackend
or parallel_mode == "isolated" → ProcessPoolBackend (even when workers == 1)
otherwise → SerialBackend with the policy’s checkpoint / failure flags only

A positive subject_timeout_sec alone does not force the process pool (library default is 900.0). Timeouts, spawn / graceful shutdown, OOM backoff, GPU capping, parallel_mode, and auto_retry_rounds apply under ProcessPoolBackend only.

ProcessPoolBackend.from_policy does not copy strict_checkpoint_hash or checkpoint_dir onto the backend object. The CLI / recipe resolve checkpoint_dir (default under out_dir), bind spec.fingerprint() onto CheckpointStore, and raise CompatibilityError when strict_checkpoint_hash=True meets an incompatible fingerprint or a legacy v0.1 layout.

SerialBackend

Runs one subject at a time in the current process. Default behind map().

from habit.execution import SerialBackend

backend = SerialBackend()
results = list(backend.map(pipeline, cohort))

# Cohort.map defaults to SerialBackend
results = cohort.map(pipeline)
results = cohort.map(pipeline, backend=SerialBackend())

Constructor knobs (checkpoint subset shared with ProcessPoolBackend):

Parameter	Default	Role
`on_subject_failure`	`"continue"`	`"continue"` or `"fail_fast"`
`resume`	`True`	Honour checkpoint successes / recorded failures when a store is attached
`retry_failed_subjects`	`False`	Re-run subjects whose checkpoint records a failure
`force_rerun_subjects`	`()`	Subject IDs reprocessed even when a success exists
`clear_checkpoint_on_success`	`False`	Clear the store after a run with zero failures

SerialBackend does not accept timeout / spawn / OOM / parallel_mode / auto_retry_rounds.

Failure policy: `continue` vs `fail_fast`

Both SerialBackend and ProcessPoolBackend accept on_subject_failure:

"continue" (default) — isolate the exception in that subject’s SubjectResult.error and proceed
"fail_fast" — re-raise the first subject exception immediately

from habit.execution import SerialBackend

backend = SerialBackend(on_subject_failure="continue")
slots = list(backend.map(op, subjects))
# slots[i].error is set for failed subjects; others have .value

Important

map() defaults to raise_on_failure=True: it aggregates failures and raises ProcessingError when any slot has an error — even if the backend used continue. Pass raise_on_failure=False to receive SubjectResult slots (recipes / CLI do this so a partial cohort can finish, matching v0.1). Soft failure also remains available via backend.map directly. See Fault tolerance patterns.

ProcessPoolBackend

Important

Windows + process pool: spawning workers re-imports your script. Put any call that starts ProcessPoolBackend (RunPolicy(backend="process"), workers > 1, or parallel_mode="isolated") inside:

if __name__ == "__main__":
    ...

Running the same code at module top level (or pasting it into a .py file without this guard) raises RuntimeError: ... bootstrapping phase on Windows. The habit CLI entry point is already safe; this applies to scripts / notebooks converted to scripts / pure-Python recipes. For a quick serial check, use RunPolicy(workers=1, backend="serial") (no spawn).

Multiprocess backend. Constructor surface mirrors RunPolicy field-by-field (except backend, checkpoint_dir, and strict_checkpoint_hash — see above).

Parameter	Default	Role
`workers`	`1`	Process count (`1` still runs in a child under ProcessPoolBackend)
`subject_timeout_sec`	`900.0`	Per-subject wall-clock cap; `None` disables
`subject_spawn_timeout_sec`	`120.0`	Isolated-mode spawn startup cap; `None` disables
`graceful_shutdown_sec`	`15.0`	Seconds between `terminate()` and `kill()` on timeout
`on_subject_failure`	`"continue"`	`"continue"` or `"fail_fast"`
`oom_backoff`	`True`	Reduce workers after fatal `MemoryError`
`oom_reduce_workers_by`	`1`	Workers subtracted per OOM step (floor 1)
`cap_workers_to_gpu_pool`	`False`	Clamp `workers` to detected GPU pool when set
`parallel_mode`	`"persistent"`	`"persistent"` (long-lived workers) or `"isolated"` (one child per subject)
`auto_retry_rounds`	`2`	Extra in-run dispatch rounds for failed subjects; `0` disables
`resume`	`True`	Honour checkpoint successes / recorded failures
`retry_failed_subjects`	`False`	Re-run checkpointed failures
`force_rerun_subjects`	`()`	Force-rerun subject IDs
`clear_checkpoint_on_success`	`False`	Clear store after a clean run
`persistent_worker_max_consecutive_failures`	`1`	Restart a persistent slot after this many consecutive fatal failures
`persistent_worker_recycle_after_tasks`	`0`	Restart a persistent worker after this many successes (`0` disables)

from habit.spec import RunPolicy
from habit.execution import ProcessPoolBackend

backend = ProcessPoolBackend(
    workers=4,
    subject_timeout_sec=900.0,
    on_subject_failure="continue",  # or "fail_fast"
    parallel_mode="persistent",     # library default; use "isolated" for per-subject isolation
    auto_retry_rounds=2,            # in-run retries for flaky subjects
    oom_backoff=True,               # reduce workers after MemoryError
)
results = list(backend.map(pipeline, cohort))

# From RunPolicy (does not apply checkpoint_dir / strict_checkpoint_hash)
backend = ProcessPoolBackend.from_policy(
    RunPolicy(workers=4, backend="process")
)

Only path-backed lazy subjects cross the process boundary. Exceeding subject_timeout_sec raises SubjectTimeoutError (isolated under continue, aborting under fail_fast).

YAML equivalents (habitat v0.1 top-level): on_subject_failure, individual_subject_timeout_sec, individual_subject_auto_retry_rounds, retry_failed_subjects, … — see Habitat Segmentation Configuration and the mapping table in Spec, RunPolicy, and YAML isomorphism.

CheckpointStore

from habit.execution import CheckpointStore, SerialBackend

store = CheckpointStore("out/run/.habitat_checkpoint")
results = cohort.map(
    pipeline,
    backend=SerialBackend(),
    checkpoint=store,
)
# Re-running skips subjects already recorded as success

Recorded terminal failures are skipped on resume (v0.1 rule) unless retry_failed_subjects=True. Successful subjects restore from_cache=True. Pass the same store to fit_predict() together with a Report so one-step product files (maps, models, figures) are rewritten from cached payloads on resume (One-step habitat analysis).

On the v1 habitat CLI path, cache keys embed the spec fingerprint, and the store is also bound to run_fingerprint.json. With strict_checkpoint_hash=True, an incompatible fingerprint raises CompatibilityError (v0.1 CheckpointConfigHashError parity). With strict=False, mismatches are warned and left unreachable.

A legacy v0.1 manifest.json / subjects/ tree is auto-migrated on store open (migrate_v01_checkpoint_if_needed()): failed subject IDs become v1 .failed records under fingerprint-scoped recipe keys; completed pickles are converted to Supervoxelization when geometry and labels are present, otherwise those subjects are logged and recomputed. The legacy tree is moved under .v01_legacy_archive/. Corrupt/unreadable manifest.json still raises CompatibilityError.

Exports: SerialBackend, ProcessPoolBackend, CheckpointStore, LegacyCheckpointMigrationReport, is_v01_checkpoint_layout, migrate_v01_checkpoint_if_needed, SubjectTimeoutError, backend_from_policy, should_use_process_pool.