.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_examples\07_parallel\plot_01_backends.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_examples_07_parallel_plot_01_backends.py: Running the same study on each backend ====================================== **Background.** Once a study is declared, HABIT still has to run it on every subject. This page is about that scheduling layer. Switching the backend or ``RunPolicy`` changes speed and how failures are handled; it does not change the science, so the habitat labels must come out identical. **Purpose.** You will run one study serially and with process workers and confirm the labels match, see why serial can win on a tiny cohort, keep a run going when one subject fails, resume from checkpoints, and build policies for GPU worker caps and per-subject timeouts. **When to use.** Stay serial for a handful of subjects or while debugging one case. Switch to process workers when there are many subjects and each one is expensive. **Key terms.** * **backend** -- what schedules the subjects: serial (one after another in this Python process) or a pool of worker processes. * **RunPolicy** -- the settings object for a run: backend, number of workers, failure handling, timeouts, checkpoint and resume options. * **spawn** -- how Windows starts a worker: a fresh Python interpreter that re-imports HABIT, which costs seconds per worker. * **checkpoint** -- finished subject results saved on disk so an interrupted run resumes without recomputing them. * **supervoxel / pool / fit / assign** -- the study stages; see :doc:`/auto_examples/00_full_pipeline/plot_01_full_pipeline`. The study is the two-step list from :doc:`/auto_examples/00_full_pipeline/plot_01_full_pipeline`, with the habitat count fixed at 3 so this page can refit it on every backend. Only ``backend=`` / ``RunPolicy`` settings change. Habitat labels must match. ========================================= ========================================== Where the work runs What to pass ========================================= ========================================== This process, one subject at a time ``SerialBackend()`` CPU, several subjects at once ``RunPolicy(backend="process")`` A fresh process per subject ``parallel_mode="isolated"`` One worker per GPU ``cap_workers_to_gpu_pool=True`` ========================================= ========================================== ``backend`` is only ``"serial"`` or ``"process"``. Isolated mode and the GPU cap are settings of the process backend, not extra backend names. On one GPU only worker 0 gets the card unless ``HABIT_GPU_OVERSUBSCRIBE=wrap``. On Windows a process pool must start under ``if __name__ == "__main__":`` (a spawned worker re-imports this file). Each backend cell below is written that way so it can be copied as is. .. GENERATED FROM PYTHON SOURCE LINES 58-63 Load the cohort and declare the study ------------------------------------- Same stages as the complete analysis, except ``fit`` uses ``n_habitats=3`` instead of an elbow search. sphinx_gallery_thumbnail_number = 1 .. GENERATED FROM PYTHON SOURCE LINES 63-106 .. code-block:: Python import dataclasses import tempfile import time from pathlib import Path import matplotlib.pyplot as plt import numpy as np from habit.contracts import Cohort, cohort_from_directory from habit.datasets import fetch_demo from habit.execution import CheckpointStore, SerialBackend, backend_from_policy from habit.recipes import Study from habit.spec import HabitatSpec, RunPolicy, Spec, Stage from habit.viz import plot_habitat_overlay # Change DATA / MODALITIES / ROI to your preprocessed layout. # The guard keeps a spawned worker from loading the cohort again. if __name__ == "__main__": DATA = fetch_demo() MODALITIES = ("pre_contrast", "LAP", "PVP", "delay_3min") ROI = "LAP" cohort = cohort_from_directory(DATA, modalities=MODALITIES, roi=ROI)[:2] spec = HabitatSpec( name="parallel_backends", stages=( # extract: one intensity column per DCE phase, inside the ROI. Stage("extract", Spec("raw", {"modalities": list(MODALITIES), "roi": ROI})), # partition: 30 supervoxels per subject; these rows are clustered. Stage("partition", Spec("kmeans", {"n_supervoxels": 30})), # pool: one matrix for the whole training cohort. Stage("pool", Spec("pool")), # fit: fixed count so serial and process runs are the same model. Stage("fit", Spec("kmeans", {"n_habitats": 3, "n_init": 10})), # assign: nearest centroid writes the habitat ids. Stage("assign", Spec("nearest_centroid")), # quantify: volume fractions, so the result still has a feature table. Stage("volume", Spec("volume")), ), random_seed=0, ) study = Study(spec) print(f"Cohort: {list(cohort.subject_ids)}") .. rst-class:: sphx-glr-script-out .. code-block:: none HABIT demo data (cached) DATA (preprocessed root): C:\Users\dongm\.habit_data\demo-data-v1\preprocessed On-disk inventory of this folder: subjects (5): subj001, subj002, subj003, subj004, subj005 image series: LAP, PVP, delay_3min, pre_contrast mask keys: LAP, PVP, delay_3min, pre_contrast example image: images/subj001/delay_3min/WATER__BH_Ax_LAVA_Flex_3min_Series0012.nrrd example mask: masks/subj001/delay_3min/WATER__BH_Ax_LAVA_Flex_10min_Series0017_mask.nrrd Your own data must use the same folder tree (change IDs / series names): DATA/ images/// masks/// Then load it with the same call the demos use: cohort = cohort_from_directory(DATA, modalities=("LAP",), roi="LAP") Swap DATA / modalities / roi to match your tree. Mask key is often the same as one image series (here LAP). Cohort: ['subj001', 'subj002'] .. GENERATED FROM PYTHON SOURCE LINES 107-111 Serial backend -------------- One subject after another, in this process. This is the reference the other backends must match. .. GENERATED FROM PYTHON SOURCE LINES 111-119 .. code-block:: Python if __name__ == "__main__": timings: dict[str, float] = {} start = time.perf_counter() serial = study.fit_predict(cohort, backend=SerialBackend()) timings["serial"] = time.perf_counter() - start print(f"serial: {timings['serial']:.1f} s, backend SerialBackend") print(serial.features.frame) .. rst-class:: sphx-glr-script-out .. code-block:: none Cohort.map[_ComputeUnits]: 0%| | 0/2 [00:00` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_01_backends.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: plot_01_backends.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_