Model Comparison Configuration

This page documents model comparison YAML used by habit compare. Schema: ModelComparisonConfig. Demo: config/model_comparison/config_model_comparison_demo.yaml.

Command usage: Model comparison. Python API: Python API (Model comparison).

Example configuration:

output_dir: ../../demo_data/results/model_comparison

files_config:
  - path: ../../demo_data/results/ml/radiomics/all_prediction_results.csv
    model_name: radiomics
    subject_id_col: subject_id
    label_col: label
    prob_col: LogisticRegression_prob
    pred_col: LogisticRegression_pred
    split_col: dataset
  - path: ../../demo_data/results/ml/clinical/all_prediction_results.csv
    model_name: clinical
    subject_id_col: subject_id
    label_col: label
    prob_col: LogisticRegression_prob
    pred_col: LogisticRegression_pred
    split_col: dataset

merged_data:
  enabled: true
  save_name: combined_predictions.csv

split:
  enabled: true

visualization:
  roc:
    enabled: true
    save_name: roc_curves.pdf
    title: ROC Curves
  dca:
    enabled: true
    save_name: decision_curves.pdf
    title: Decision Curves
  calibration:
    enabled: true
    save_name: calibration_curves.pdf
    n_bins: 5
    title: Calibration Curves
  pr_curve:
    enabled: true
    save_name: precision_recall_curves.pdf
    title: Precision-Recall Curves

delong_test:
  enabled: true
  save_name: delong_results.json

metrics:
  basic_metrics:
    enabled: true
  youden_metrics:
    enabled: true
  target_metrics:
    enabled: true
    targets:
      sensitivity: 0.91
      specificity: 0.91

Top-level fields

output_dir (required)

• Type: string (directory path)

• Description: root directory for plots, merged tables, and JSON metrics. Relative paths resolve from the YAML file directory.

files_config (required)

• Type: non-empty list

• Description: one entry per model prediction table to compare.

Each files_config item

Field	Required	Description
path	yes	Prediction CSV/Excel from habit model / habit cv
subject_id_col	yes	Subject identifier column
label_col	yes	Ground-truth binary label column (0 / 1)
prob_col	yes	Predicted probability column (continuous in [0, 1])
pred_col	no	Predicted class column when available
split_col	no	Split name column (train / test / …); used when split.enabled
model_name	recommended	Display name in plots and reports; must be unique across entries
name	no	Alias for model_name when model_name is omitted

If both model_name and name are omitted, HABIT uses the file stem of path.

merged_data

• enabled (bool, default true): write a combined prediction table

• save_name (default combined_predictions.csv)

split

• enabled (bool, default false): when true, generate per-split analyses using split_col in each prediction file

visualization

Sub-blocks roc, dca, calibration, and pr_curve each accept:

• enabled (bool, default true)

• save_name (output filename under output_dir)

• title (plot title, English)

• n_bins (calibration only; number of probability bins)

delong_test

• enabled (bool, default true)

• save_name (default delong_results.json)

metrics

• basic_metrics.enabled: accuracy / sensitivity / specificity style metrics

• youden_metrics.enabled: Youden-index optimal threshold metrics

• target_metrics.enabled plus targets: evaluate at fixed operating points (each target value must be in (0, 1))

Typical outputs under output_dir

• roc_curves.pdf, decision_curves.pdf, calibration_curves.pdf, precision_recall_curves.pdf

• delong_results.json

• combined_predictions.csv (when merge is enabled)

Related machine-learning training fields remain on Machine Learning Configuration.