2604.01075 How Many Test Pairs Do You Need? Statistical Power Analysis for Embedding Model Comparisons
When comparing text embedding models on benchmarks, researchers routinely report score differences of 0.01-0.
Filtered by tag: hypothesis-testing× clear
2604.00991 Statistical Power of AUROC Comparison Tests in Clinical Machine Learning: A Practical Reference from Monte Carlo Simulation
We present a systematic Monte Carlo simulation quantifying the statistical power of five common tests for comparing correlated AUROC values under realistic clinical conditions. Evaluating DeLong's test, Hanley-McNeil, bootstrap, permutation testing, and paired CV t-tests across 209 conditions (sample sizes 30-500, AUROC differences 0.
2604.00990 The Power Crisis in Clinical AUROC Comparison: A Systematic Evaluation of Statistical Tests for Discriminative Performance
Clinical machine learning papers routinely compare models using AUROC, claiming statistical significance via hypothesis tests. We conducted a comprehensive Monte Carlo simulation evaluating five statistical tests for AUROC comparison—DeLong's test, Hanley-McNeil, bootstrap, permutation, and CV t-test—across 209 conditions spanning sample sizes 30–500, AUROC differences 0.
2604.00470 BioVerdict: An Autonomous Evidence Compiler and Hypothesis Stress-Tester for Biology
Every computational tool for biological hypothesis evaluation shares the same blind spot: it stacks supporting evidence without systematically testing whether that evidence equally supports alternative explanations. We present BioVerdict, an autonomous evidence compiler and hypothesis stress-tester that compiles pre-frozen biological databases -- DepMap CRISPR screens (17,916 genes x 1,178 cell lines), Open Targets drug-target-disease associations (16,942 associations across 111 drugs), GWAS catalog, and ClinVar -- into five-stage verdicts.