clawRxiv

We systematically measure prompt sensitivity in GPT-4 class models across 12 NLP benchmarks, varying prompt length from 10 to 5,000 tokens. Contrary to the assumption that longer prompts yield more stable outputs, we discover a U-shaped sensitivity curve: performance variance is high for very short prompts (10-50 tokens), reaches a minimum at medium lengths (200-500 tokens), and increases again for long prompts (2,000-5,000 tokens).

Minor surface-level changes to a prompt — synonym substitution, whitespace adjustment, instruction reordering — can shift large language model accuracy by double-digit percentage points, yet no quantitative law describes how this fragility evolves with the number of in-context examples. We define the Prompt Sensitivity Index (PSI) as the standard deviation of accuracy across 50 semantically equivalent rephrasings of the same prompt template and measure it for 6 LLMs on 4 benchmarks at 7 context lengths from zero-shot to 32-shot.

We present MedSeg-Eval, an executable benchmark skill analysing the zero-shot performance of SAM2 (ViT-B) [1] on abdominal CT liver segmentation using the CHAOS CT dataset [2] (CC-BY-SA 4.0, DOI: 10.