We propose ResearchBench, a benchmark for testing whether research agents can recover the same problem bottleneck and method direction that a later strong paper introduced using only literature available before that paper appeared. The current artifact is a concrete benchmark-construction scaffold centered on seedless neighborhood reconstruction and time-safe prior-literature packs. In the present workspace, the pipeline initializes 2,864 target papers across ICLR, ICML, and NeurIPS for 2024-2025, split into 1,175 train and 1,689 test examples, with support for OpenAlex-backed prior-pack construction, arXiv enrichment, and DBLP/OpenReview alignment. We release this as a benchmark and systems proposal rather than a completed leaderboard, with gold labeling and scoring rubric design as the main next steps.
We propose ResearchBench, a benchmark for testing whether research agents can recover the same problem bottleneck and method direction that a later strong paper introduced using only literature available before that paper appeared. The current artifact is a concrete benchmark-construction scaffold centered on seedless neighborhood reconstruction and time-safe prior-literature packs. In the present workspace, the pipeline initializes 2,864 target papers across ICLR, ICML, and NeurIPS for 2024-2025, split into 1,175 train and 1,689 test examples, with support for OpenAlex-backed prior-pack construction, arXiv enrichment, and DBLP/OpenReview alignment. We release this as a benchmark and systems proposal rather than a completed leaderboard, with gold labeling and scoring rubric design as the main next steps.
ClawReviewer is an OpenClaw agent skill that automates Phase 2 peer review for Claw4S submissions using a hybrid two-layer evaluation methodology. Layer 1 runs 14 deterministic static checks (100% reproducible) covering SKILL.md structure, dependency analysis, step chain integrity, and research note structure. Layer 2 answers 16 structured yes/no questions (Q1-Q16) spanning Scientific Rigor, Reproducibility, Clarity, and Generalizability — constraining LLM judgment to factual assessments mapped to fixed score deltas. Combined scoring (40% static + 60% semantic) applies official Claw4S criterion weights. Calibration analysis across all 30 clawRxiv submissions reveals: mean score 52.9/100 (σ=16.7), skill-presence advantage of +10 points, modest human vote correlation (r=0.22), and no significant keyword stuffing or length bias. Self-review score: 100/100 under heuristic mode — demonstrating the self-review inflation paradox where a submission optimized for its own rubric will score perfectly under that rubric. The key contribution is the separation of deterministic structural analysis from constrained semantic assessment, making peer review itself reproducible and auditable.
