We describe Sibyl, A lightweight post-processor that scans LLM math outputs and marks any claim not backed by a cited source or a proof sketch as 'unproven'.. Large language models frequently introduce mathematical claims into multi-step solutions without proof or citation, presenting conjectural statements with the same confidence as theorems.
We describe Aureole, A single-figure ring-plot that renders AUC, calibration slope, and calibration-in-the-large per demographic subgroup for a clinical model.. Subgroup performance tables are tedious to read and easy to collapse into a single aggregate metric.
We describe Damselfly, A permutation-based paired-AUC comparison tuned for small and label-sparse clinical datasets where DeLong's normal approximation is unreliable.. The DeLong test is standard for comparing two AUCs on the same samples but relies on a normal approximation of the covariance of U-statistics that fails at small sample size or when the positive class is severely imbalanced.
We describe Picket, A small reporting template and helper library that makes within-fold mis-calibration visible in cross-validated clinical prediction models.. Published clinical prediction models typically report aggregate calibration (Brier score, ECE, HL test) averaged over cross-validation folds.
We describe Stagehand, A minimal pattern and library that splits every irreversible agent action into a dry-run plan and a signed commit step.. Agents performing irreversible actions (file deletion, financial transactions, external emails, database migrations) currently interleave plan and commit in one step.