Introduction

Structural variants (SVs) — genomic rearrangements exceeding 50 base pairs — account for a greater proportion of divergent bases between human genomes than single-nucleotide variants (SNVs), yet remain substantially harder to detect with confidence (Sudmant et al., 2015). The advent of long-read sequencing technologies from Pacific Biosciences (HiFi) and Oxford Nanopore Technologies (ONT) has dramatically improved SV detection sensitivity by spanning repetitive and complex regions that confound short-read aligners (Logsdon et al., 2020). However, the bioinformatics community now faces a benchmarking challenge: multiple SV callers exist for long-read data — including Sniffles2, cuteSV, SVIM, pbsv, and DeBreak — each with distinct algorithmic strategies, and no single caller dominates across all SV classes and size ranges.

In this study, we present a systematic benchmarking analysis of five widely used long-read SV callers across simulated and real datasets spanning HiFi and ONT platforms. We evaluate performance using the Genome in a Bottle (GIAB) HG002 Tier 1 SV benchmark set (Zook et al., 2020) and a synthetic diploid genome with known ground-truth SVs. Our analysis stratifies results by SV type (deletion, insertion, duplication, inversion, translocation), size class, and genomic context (repetitive vs. unique sequence). We further examine the effect of sequencing depth on recall and precision, and quantify the benefit of ensemble calling strategies that merge outputs from multiple tools.

Background

Structural Variant Classes

SVs encompass deletions (DEL), insertions (INS), duplications (DUP), inversions (INV), and translocations (TRA/BND). Each class presents distinct detection challenges. Deletions and insertions are the most abundant SV types in human genomes, while inversions and translocations, though rarer, carry outsized functional impact when disrupting gene regulatory architecture (Collins et al., 2020).

Long-Read SV Callers

Sniffles2 (Smolka et al., 2024) uses a multi-sample, population-aware approach that clusters split-read and alignment signatures, then genotypes across cohorts. cuteSV (Jiang et al., 2020) employs a clustering-and-refinement strategy optimized for both HiFi and ONT reads. SVIM (Heller & Vingron, 2019) collects SV signatures from read alignments and clusters them with a partitioning approach. pbsv (Pacific Biosciences) is tailored for HiFi data, leveraging the high per-read accuracy of circular consensus sequencing. DeBreak (Ren & Bhatt, 2023) uses a local de novo assembly strategy to resolve complex SVs at base-pair resolution.

Benchmarking Standards

The GIAB consortium provides a curated benchmark set for HG002 (Ashkenazi son) containing approximately 12,745 isolated, sequence-resolved SVs with defined confident regions (Zook et al., 2020). We use Truvari (English et al., 2022) as the comparison engine, applying standard matching criteria: 70% reciprocal overlap, 70% sequence similarity, and a maximum reference distance of 1,000 bp.

Methods

Simulated Data Generation

We generated a synthetic diploid genome based on GRCh38 using VISOR (Bolognini & Magi, 2020). The simulation introduced 5,000 deletions (50 bp–100 kbp), 4,500 insertions (50 bp–50 kbp), 500 duplications (500 bp–200 kbp), 300 inversions (1 kbp–5 Mbp), and 200 translocations. SV positions were sampled uniformly across autosomes, excluding centromeric and telomeric regions. We then simulated HiFi reads at 30× coverage using pbsim3 (Ono et al., 2023) with a mean read length of 15 kbp and a per-read accuracy of 99.9%, and ONT reads at 30× coverage using the R10.4.1 error model with a mean read length of 20 kbp and a modal accuracy of 99.2%.

Real Data

For real-data evaluation, we used publicly available HG002 datasets:

• PacBio HiFi: ~34× coverage, mean read length 13.5 kbp (GIAB FTP)
• ONT R10.4.1: ~60× coverage, mean read length 22 kbp (GIAB FTP)

Reads were aligned to GRCh38 (no alt contigs) using minimap2 v2.28 (Li, 2018) with preset map-hifi or map-ont as appropriate. BAM files were sorted and indexed with samtools v1.19.

SV Calling

Each caller was run with recommended parameters for the respective platform:

# Sniffles2
sniffles --input aligned.bam --vcf sniffles2.vcf --reference GRCh38.fa

# cuteSV
cuteSV aligned.bam GRCh38.fa cutesv.vcf . --max_cluster_bias_INS 1000 \
  --diff_ratio_merging_INS 0.9 --max_cluster_bias_DEL 1000 \
  --diff_ratio_merging_DEL 0.5

# SVIM
svim alignment --min_sv_size 50 svim_output/ aligned.bam GRCh38.fa

# pbsv (HiFi only)
pbsv discover aligned.bam ref.svsig.gz
pbsv call GRCh38.fa ref.svsig.gz pbsv.vcf

# DeBreak
debreak --bam aligned.bam --outpath debreak_output/ --rescue_large_ins \
  --rescue_dup --ref GRCh38.fa

Evaluation Metrics

All VCFs were normalized and compared against the ground truth using Truvari v4.2 bench mode. We report:

• Recall ($R$): $R = \frac{TP}{TP + FN}$
• Precision ($P$): $P = \frac{TP}{TP + FP}$
• F1 score: $F_1 = \frac{2PR}{P + R}$

Results were stratified by SV type, size bins (50–300 bp, 300 bp–1 kbp, 1–10 kbp, 10–100 kbp, >100 kbp), and genomic context using RepeatMasker annotations.

Ensemble Strategy

We implemented a union-intersection ensemble: an SV call is retained if supported by at least $k$ out of $n$ callers (with $k \in {2, 3}$ and $n = 5$). Merging was performed using SURVIVOR v1.0.7 (Jeffares et al., 2017) with a maximum allowed distance of 1,000 bp between breakpoints.

Depth Titration

To assess the effect of coverage, we downsampled the HiFi and ONT BAMs to 5×, 10×, 15×, 20×, and 25× using samtools view with a fractional sampling flag, then re-ran all callers at each depth.

Results

Simulated Genome Performance

On the simulated HiFi dataset at 30× coverage, Sniffles2 achieved the highest overall F1 score of 0.953, followed closely by cuteSV (0.947) and DeBreak (0.944). pbsv reached 0.938, while SVIM trailed at 0.912. For ONT simulated data, cuteSV led with F1 = 0.931, Sniffles2 followed at 0.928, and DeBreak at 0.919.

Performance varied substantially by SV type:

Inversions and duplications were consistently the hardest SV classes, with all callers showing F1 drops of 8–17% relative to deletions. Translocations were detected with recall below 0.70 by all tools except DeBreak (0.74), which benefits from its local assembly approach for resolving complex breakpoints.

GIAB HG002 Benchmark

On the real HG002 HiFi data evaluated against the GIAB Tier 1 benchmark:

Sniffles2 and DeBreak tied for the highest F1 on real data. Notably, DeBreak achieved the highest precision (0.961), attributable to its assembly-based refinement of breakpoint coordinates. SVIM showed the largest gap between simulated and real performance, suggesting sensitivity to the error profile of real sequencing data.

Size-Stratified Analysis

Small SVs (50–300 bp) proved most challenging across all callers. In this size range, recall dropped by 5–12% compared to SVs in the 1–10 kbp range. The primary failure mode was insertions in the 50–300 bp range within homopolymer or short tandem repeat (STR) contexts, where alignment ambiguity leads to imprecise breakpoint placement. DeBreak and Sniffles2 showed the smallest performance degradation in this regime, with F1 scores of 0.921 and 0.918 respectively for small insertions.

For large SVs (>100 kbp), recall was uniformly high (>0.95) for deletions but dropped for duplications and inversions due to incomplete read-through of the variant.

Repetitive Sequence Context

SVs overlapping segmental duplications showed a mean F1 reduction of 0.087 across all callers compared to SVs in unique sequence. SVs within LINE/SINE elements showed a smaller but consistent reduction of 0.034. The assembly-based approach of DeBreak was most robust to repetitive context, with only a 0.061 F1 reduction in segmental duplications versus 0.102 for SVIM.

Depth Titration

Recall degraded gracefully with decreasing coverage for most callers. At 15× HiFi coverage, Sniffles2 retained 94.1% of its 30× recall, while SVIM retained only 87.3%. At 10× coverage, all callers showed substantial recall loss (>10%), with small insertions most affected. The precision of all callers remained relatively stable down to 10× coverage, after which false positive rates increased sharply, particularly for cuteSV.

At 5× coverage, no caller achieved F1 > 0.80, confirming that long-read SV calling requires a practical minimum of ~10× coverage for clinical-grade sensitivity.

Ensemble Calling

The $k = 2$ ensemble (call retained if supported by $\geq 2$ callers) achieved the best balance:

The $k = 2$ ensemble improved F1 by 1.4 percentage points over the best individual caller, primarily through recall gains on inversions (+6.2%) and duplications (+4.8%) — SV classes where callers exhibit complementary detection profiles. The $k = 3$ ensemble traded recall for precision, which may be preferable in clinical settings where false positives carry higher cost.

Discussion

Our benchmarking reveals several key findings for the long-read SV calling community.

First, no single caller uniformly dominates. Sniffles2 and DeBreak share the top F1 on real data, but their strengths differ: Sniffles2 excels in recall for common SV types, while DeBreak provides superior precision and better handling of complex SVs through local assembly. For HiFi-only workflows, pbsv remains competitive and benefits from tight integration with the PacBio ecosystem.

Second, ensemble strategies provide meaningful gains. A simple majority-vote ensemble ($k = 2$ of 5) boosted F1 by 1.4 points, a non-trivial improvement at the high end of the performance curve. This gain comes at modest computational cost — running five callers on a 30× HiFi genome completes in under 4 hours on a 32-core server — and we recommend ensemble calling as standard practice for research applications.

Third, small SVs in repetitive contexts remain the frontier challenge. The 50–300 bp size range, particularly insertions in STR regions, accounts for the majority of false negatives across all callers. Emerging approaches that integrate repeat-aware alignment (e.g., TRGT for tandem repeats) with general-purpose SV callers may address this gap.

Fourth, coverage requirements are platform-dependent. HiFi data at 15× provides near-saturation recall for most SV classes, while ONT data requires ~20× to achieve comparable performance, reflecting the higher per-read error rate. For cost-sensitive projects, 15× HiFi represents an efficient operating point.

Limitations

This study has several limitations. Our simulated SVs were placed in non-centromeric regions and may underestimate difficulty in highly repetitive contexts. The GIAB benchmark, while the gold standard, is biased toward isolated SVs and underrepresents complex, clustered rearrangements. We did not evaluate somatic SV calling or mosaic variant detection, which present additional challenges. Finally, our ensemble approach used a simple voting scheme; more sophisticated methods incorporating caller-specific confidence scores may yield further improvements.

Conclusion

We provide a comprehensive benchmark of five long-read SV callers across simulated and real human genome data. Sniffles2 and DeBreak emerge as top performers, with complementary strengths in recall and precision. Ensemble calling with a $k \geq 2$ threshold consistently outperforms any individual tool. We recommend that bioinformatics pipelines for SV detection adopt ensemble strategies and that the community continue developing specialized methods for small SVs in repetitive regions, where current tools show the greatest room for improvement.

References

1. Bolognini, D. & Magi, A. (2020). VISOR: a versatile haplotype-aware structural variant simulator for short- and long-read sequencing. Bioinformatics, 36(4), 1267–1269.
2. Collins, R. L., et al. (2020). A structural variation reference for medical and population genetics. Nature, 581, 444–451.
3. English, A. C., et al. (2022). Truvari: refined structural variant comparison preserves allelic diversity. Genome Biology, 23, 271.
4. Heller, D. & Vingron, M. (2019). SVIM: structural variant identification using mapped long reads. Bioinformatics, 35(17), 2907–2915.
5. Jeffares, D. C., et al. (2017). Transient structural variations have strong effects on quantitative traits and reproductive isolation in fission yeast. Nature Communications, 8, 14061.
6. Jiang, T., et al. (2020). Long-read-based human genomic structural variation detection with cuteSV. Genome Biology, 21, 189.
7. Li, H. (2018). Minimap2: pairwise alignment for nucleotide sequences. Bioinformatics, 34(18), 3094–3100.
8. Logsdon, G. A., et al. (2020). Long-read human genome sequencing and its applications. Nature Reviews Genetics, 21, 597–614.
9. Ono, Y., et al. (2023). pbsim3: a simulator for all types of PacBio and ONT long reads. NAR Genomics and Bioinformatics, 4(4), lqac092.
10. Ren, J. & Bhatt, P. (2023). DeBreak: a local assembly-based structural variant caller for long reads. Nature Communications, 14, 4186.
11. Smolka, M., et al. (2024). Detection of mosaic and population-level structural variants with Sniffles2. Nature Biotechnology, 42, 1571–1580.
12. Sudmant, P. H., et al. (2015). An integrated map of structural variation in 2,504 human genomes. Nature, 526, 75–81.
13. Zook, J. M., et al. (2020). A robust benchmark for detection of germline large deletions and insertions. Nature Biotechnology, 38, 1347–1355.