Benchmarks

Every performance and equivalence figure in this documentation comes from bwa-mem3-bench, a public benchmarking suite that runs bwa-mem3 alongside bwa, bwa-mem2 v2.2.1, minibwa and bwameth on AWS instances across the x86 AVX2, x86 AVX-512 and Arm NEON SIMD tiers. Each release candidate is run through its full matrix before being blessed. Other pages cite this one instead of linking to the suite directly.

Published results

• Results for each release: the latest release’s same-host speed against bwa, bwa-mem2 and minibwa, then per release:
  • speed, memory and CPU efficiency on each dataset across six instance types;
  • agreement with bwa-mem2 (and bwameth for methylation), with the previous release, between Arm and x86, and between --fast and the default preset;
  • accuracy against simulated reads with known origins;
  • thread scaling from 1 to 64 threads.
• Methodology: how the numbers are produced, which host each measurement ran on, and which tags each agreement check ignores.

The headline speed table in the README comes from the same same-host runs.

Datasets

All are aligned to GRCh38 (the hg38 analysis set with decoys and ALTs) with 16 threads. Where to obtain each input is documented in the suite’s data-setup guide.

Release validation

Each release candidate runs the full matrix and must pass three gates before it is blessed as the new reference (“golden”) for the next release:

1. vs the baseline aligner: drift stays within each dataset’s budget in the expected-divergence registry (see Equivalence with bwa-mem2). DNA datasets are scored on concordance against bwa-mem2 v2.2.1; methylation datasets on confident relocation against bwameth (the share of primaries either aligner maps at MAPQ ≥ 20 that the two place at a different locus).
2. vs the previous release: at least 99.999% concordance on every cell that has a previous-release result. An intended alignment change is recorded in the release’s entry in the release ledger, which is required to promote the release; the entry does not relax this check.
3. Thread scaling: pipeline efficiency does not regress.

The suite’s release runbook is the full procedure; the release process page covers the bwa-mem3 side.

Earlier one-off measurements

• The ALT-aware --compat byte-identity figures on Equivalence with bwa-mem2 have their per-cell breakdown in bwa-mem3-bench#47.

Per-release concordance history

Per-(release, sample) primary-alignment concordance against upstream bwa-mem2 v2.2.1 (bwameth.py for the methylation samples), with supplementary-alignment counts, for the bwa-mem3 releases shown below. Concordance is the minimum vs-baseline value over reps and x86 architectures (deterministic per sample); supp_query/supp_baseline are total supplementary records emitted by bwa-mem3 and the baseline, and count_mismatch is the number of templates whose supplementary count differs. The divergence catalog explains what each kind of drift is and its budget. The suite’s methodology lists the instance types behind each architecture and the SIMD path each one runs.

This table and the divergence catalog are generated from the benchmark database; do not edit them by hand. After a new release is collected, regenerate both and replace the content between the FG-DIVERGENCE-CATALOG / FG-RELEASE-TABLE markers.

Links

• GitHub: https://github.com/fg-labs/bwa-mem3-bench
• License: MIT

See also: Performance Overview · SIMD dispatch matrix · Equivalence with bwa-mem2 · Release process