Skip to content

haplotypes: subchain kmers with copy numbers, and a count-model pair for haplotype sampling - #5112

Open
benedictpaten wants to merge 16 commits into
vgteam:masterfrom
benedictpaten:haplotypes-model-pair
Open

benedictpaten wants to merge 16 commits into
vgteam:masterfrom
benedictpaten:haplotypes-model-pair

Conversation

@benedictpaten

@benedictpaten benedictpaten commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

vg haplotypes samples a personal graph from a pangenome graph. The haplotypes in the graph are the panel haplotypes. The haplotype index divides each chain of the graph into subchains, which are short stretches between snarl boundaries, and it records which kmers each panel haplotype has in each subchain. To sample, vg haplotypes counts the kmers in the sample's reads. In each subchain, the greedy selection repeatedly takes the panel haplotype with the best kmer score, and the generated haplotypes are joined from the panel haplotypes taken in consecutive subchains. This pull request adds options for both the index and the sampling.

When the haplotype index is built, subchains can be represented by subchain kmers with their copy numbers. A subchain kmer is a minimizer whose every occurrence in the graph is on a node that the haplotypes crossing the subchain visit. Unlike a minimizer with a single occurrence in the graph, such a kmer can occur several times in a haplotype, so the index also stores how many times each kmer occurs in each haplotype (its copies, up to 5). This keeps kmers in repeats, where haplotypes differ by copy number.

When the haplotypes are sampled, the first two haplotypes in each subchain can come from a model of the kmer counts instead of the greedy selection. The count of a kmer is negative binomial given its copies in the sample's two haplotypes. The model is fitted once to the counts of all subchain kmers. In each subchain, the score of a pair of haplotypes is the log-likelihood of the counts given the copies the pair carries. A hidden Markov model over pairs follows each chain: each member of the pair keeps its haplotype into the next subchain, or moves to another haplotype at a fixed rate per bp. The Viterbi path gives the pair in each subchain. The greedy selection then adds the other haplotypes.

The greedy selection gets limits on the sequence a new haplotype adds, a bonus for continuing a generated haplotype, and a way of matching generated haplotypes with the selected ones by kmer similarity.

The pull request also makes building the haplotype information and sampling several times faster, and -t now sets the number of threads (see Speed). The output does not change: the haplotype information and the sampled graphs are the same as before.

The default behaviour of vg haplotypes does not change: an index built without --subchain-kmers holds the same kmers and gives the same sampled graphs as before. Every haplotype index is now written as version 8, which versions of vg without this change cannot read.

Recommended commands

Build the haplotype index with subchain kmers:

vg haplotypes -t 8 --kmer-length 29 --window-length 11 \
    -d graph.dist -r graph.ri --subchain-kmers -H graph.hapl graph.gbz

Sample 32 haplotypes with the model pair, keeping the CHM13 reference path:

vg haplotypes -t 8 --preset model --set-reference CHM13 \
    -i graph.hapl -k sample.kff -g sample.gbz graph.gbz

The model preset is the same as these options:

--include-reference --num-haplotypes 32 --continuity-bonus 2 --max-new-bp 1000 \
--bp-escape 300 --max-n-bp 0 --similar-threading --model-pair

Like the other presets, it replaces the sampling options given before it, and options after it override it. Like the haploid, diploid, hifi and ont presets, it includes the reference paths. To map reads to the sampled haplotypes without CHM13, so that CHM13 does not draw reads away from them, remove the CHM13 paths from the sampled graph:

vg gbwt -Z sample.gbz -o sample.gbwt
vg gbwt -x sample.gbz -R CHM13 -g sample.noref.gbz sample.gbwt

The first command copies out the GBWT of sample.gbz, because vg gbwt cannot remove paths from the GBWT inside a GBZ file. The second removes the paths whose sample name is CHM13 and writes the rest. sample.noref.gbz leaves out the nodes that only CHM13 visits and keeps the other nodes with their IDs, so vg call --read-likelihood can call reads mapped to it on sample.gbz. With --set-reference CHM13, CHM13 is the only reference sample, and sample.noref.gbz is the same graph that sampling a second time without --include-reference would make.

New options

In the descriptions below, a candidate is a panel haplotype in a subchain that the selection can take, and the new nodes of a candidate are the nodes on its path through the subchain that no haplotype already taken visits.

  • --subchain-kmers: build the index with subchain kmers and store their copies in each haplotype.
  • --continuity-bonus F: add F to the score of a candidate that a generated haplotype took in the previous subchain [0].
  • --max-new-bp N: after the first haplotype in a subchain, take a candidate only if its new nodes have at most N bp [no limit]. Reference paths do not count, so the selection is the same with and without --include-reference.
  • --bp-escape F: also take a candidate over --max-new-bp if it adds one kmer from the reads that the haplotypes already taken lack per F bp of new nodes (0: never) [0].
  • --max-n-bp N: take a candidate only if its nodes in the subchain have at most N unknown (N) bases [no limit].
  • --similar-threading: when a generated haplotype cannot continue with its panel haplotype, give it the unused selection whose kmers in the subchain differ least from that panel haplotype. Without it, the generated haplotype takes the next unused selection.
  • --model-pair: take the first two haplotypes in each subchain from the count model and the hidden Markov model over pairs.
  • --preset model: the options above, as listed under the recommended commands. It needs an index built with --subchain-kmers, and vg haplotypes stops with an error on an index without kmer copies.

When no candidate is within --max-new-bp or --max-n-bp, the selected haplotypes are repeated, as they are when a subchain has too few haplotypes. If no candidate for the first haplotype has few enough N bases, the best one is taken.

The model pair is exempt from --max-new-bp. A haplotype with more than --max-n-bp N bases is not a candidate for the pair, unless every haplotype that can be selected has that many. The pair is not used in chains sampled with the high- or half-coverage scoring model. The count model assumes two copies of the genome, so --model-pair cannot be combined with --haploid-scoring (or --preset haploid) or --diploid-sampling.

Speed

On a large node, vg haplotypes used few of its threads:

  • -t was capped at 16, and sampling used 0.85 times the default number of threads, whatever -t said. Now -t sets the number of threads.
  • The minimizer index was built under one lock, so it took the same time at any number of threads. It is now built without a lock, in buckets that each thread sorts on its own.
  • On a graph with more than one chromosome, each chromosome's subchains were built on one thread. Now the subchains of all chromosomes are one set of tasks, largest first, and a subchain larger than half of a thread's share, such as an unaligned centromere, uses all threads.

Building the haplotype information also does less work. Each haplotype is walked once through each subchain, and haplotypes with the same walk share their kmers. Kmer copies are counted in an array, with an open-addressing table for the kmers.

Sampling:

  • The model step no longer walks every haplotype through every subchain, because the index stores what the walks gave.
  • Without a continuity bonus (every preset but --preset model), the selections in all subchains are made in parallel.
  • Scoring decodes each subchain's kmer presence matrix once, instead of in every round of the greedy selection.
  • Diploid sampling scores pairs with popcounts.
  • The haplotype index loads on a second thread while the graph loads, and the KFF file is read without holding a lock while it is decoded.

On chr20 with 462 haplotypes and 8 threads, before this work and after:

Run Before After
Build, default 1067 s, 11.7 GB 105 s, 12.5 to 13.3 GB
Build, --subchain-kmers 573 s, 14.3 GB 119 s, 13.1 to 14.2 GB
Sample, default 100 s 39 s
Sample, --preset model 132 s 56 s
Sample, --preset diploid 84 s 13 s

On a graph of chr20 and chr6 with 60 haplotypes, building with --subchain-kmers takes 80 s instead of 671 s. Before, chr6 was built on one thread.

The sampled graphs were compared byte for byte with the previous code, and they matched:

  • on the test graph, on chr20 with 60 and 462 haplotypes, and on the chr20 and chr6 graph;
  • with the default options and the model and diploid presets;
  • with different numbers of threads;
  • when sampling from a version 7 index built by the previous code.

Haplotype index format

The haplotype index format becomes version 8. It adds two things to each subchain:

  • Its copies: three integer vectors that list, for each sequence, the kmers that occur more than once in it and their numbers of copies.
  • The median bases of the walks of its sequences through the subchain, and the N bases of each walk, which the model pair and --max-n-bp use.

Every index is written as version 8. In an index built without --subchain-kmers the two additions are empty, which adds 0.3% to the file on chr20 (253.4 MB against 252.6 MB).

Versions 5 to 7 can be read. Their subchains have no copies and no walk bases. Earlier commits of this pull request wrote a different version 8, with the copies but without the walk bases. Such a file is not recognized, and its index has to be rebuilt.

Loading checks that:

  • the copies list kmers of the subchain in increasing order, with 2 to 5 copies each;
  • the walk N bases match the sequences;
  • the kmer presence matrix matches the kmers and the sequences.

vg haplotypes --validate checks, against the haplotype sequences:

  • the presence, the copies and the walk bases of subchain kmers;
  • that some sequences have different copies of each kmer.

Code

  • src/recombinator_haplotypes.{hpp,cpp}:
    • the copies and walk bases in Haplotypes::Subchain, their serialization and checks, and the choice of file version;
    • presence_row(), which returns the kmer presence of a sequence as 64-bit words;
    • the scoring plan of KmerPresenceMatrix and score_sequence();
    • the KFF errors, which are now reported from the reading tasks.
  • src/recombinator.{hpp,cpp}:
    • HaplotypePartitioner::partition_haplotypes() collects the subchains and sequences of every chain, then builds all subchains as one set of tasks with build_subchain(). SubchainWalk walks a sequence through a subchain, and sequences with the same walk are grouped (SubchainWalks) so that their kmers are found once. With --subchain-kmers, build_subchain_kmers() keeps the minimizers whose occurrences are all on the subchain's nodes, counts their copies with OrientedKmerTable, OrientedKmerCounts and KmerCopyTally, and writes the informative ones with the walk bases. Without it, build_unique_kmers() keeps the minimizers with a single occurrence, as before.
    • select_haplotypes() applies the model pair, the limits and the continuity bonus. It is declared in the header with a Recombinator::SelectionContext that carries what it needs beyond the kmers, so that unit tests can call it. A SelectionLimits object in the source file checks a candidate against the new-bp and N-base limits; it walks a candidate only when it is first checked, and only until the candidate has too many new bases. Recombinator::select_in_parallel() makes the selections of all subchains in advance when there is no continuity bonus.
    • thread_by_similarity() implements --similar-threading.
    • model_candidates() finds the candidates for the pair in a subchain. Recombinator::model_pairs() fits the count model and runs the hidden Markov model on each chain before the haplotypes are generated.
  • artifacts/recombinator_model.{hpp,cpp}: the count model, its fit, the pair scores and the Viterbi algorithm, in namespace pair_model. These functions do not use the graph, so the tests can give them synthetic data.
  • artifacts/haplotype_minimizer_index.{hpp,cpp}: the minimizer index that vg haplotypes builds, without a lock.
  • artifacts/subchain_walk.{hpp,cpp}: SubchainWalk, and the OrientedKmerTable and OrientedKmerCounts that count the copies of kmers in a walk.
  • artifacts/diploid_pair_scorer.{hpp,cpp}: the pair scores of diploid sampling.
  • src/kff.{hpp,cpp}: ParallelKFFReader copies runs of single-kmer records under its lock and decodes them outside it.
  • src/subcommand/haplotypes_main.cpp:
    • the options and -t;
    • the check that the model preset has an index with kmer copies;
    • the validation of subchain kmers and walk bases;
    • loading the haplotype index while the graph loads.

Tests

  • Unit tests (vg test "[haplotypes]"):
    • the copies in a subchain, their serialization, reading a subchain from version 7, and rejecting copies or kmer presence that do not match the subchain;
    • presence_row() against the rows it was built from, on compressed matrices;
    • the fit of the count model recovering known parameters from an expected histogram;
    • pair scores against a direct sum over the kmers, the pair that produced the counts, sequences with the same kmers but different copies, and the choice of candidate groups;
    • the Viterbi path keeping a haplotype when switching is rare, switching when it is free, following a haplotype to another group, with different numbers of candidates in consecutive subchains, with different score scales, and rejecting a subchain without candidates;
    • the choice of sequences for the pair;
    • on a small graph: the candidates for the pair, and the new-bp limit, its escape, the N-base limit, the continuity bonus and the model pair in the selection of a subchain;
    • threading by similarity;
    • validation of the parameters, and the settings of the model preset;
    • the walk of a sequence through a subchain, across fragment breaks, for prefixes, suffixes and full haplotypes ([subchain_walk]);
    • the kmer table and the kmer counts against a map ([oriented_kmer_table]);
    • the minimizer index against gbwtgraph's ([haplotype_minimizer_index]), the diploid pair scores against a direct sum, and the KFF reader on sections of several kinds;
    • the stored walk bases, scoring with a plan and scoring one sequence with weights that round, and the N-base limit with stored N bases.
  • test/t/54_vg_haplotypes.t builds an index with subchain kmers on the micb-kir3dl1 test graph and samples with the recommended options. It checks that the model pair, threading by similarity, the continuity bonus, the new-bp limit and its escape each change the output; that the generated haplotypes do not depend on the reference paths; that the output does not depend on the number of threads, for the haplotype information, for subchains built on all threads, for selection without a continuity bonus and for diploid sampling; that the stored N bases of the walks validate on a graph with an N node; that a high-coverage contig is sampled the same with and without the model pair; that the model preset gives the same output as its options and stops on an index without subchain kmers; and that invalid combinations of options are errors with the right messages.

Results

HG002 was sampled from HPRC Minigraph-Cactus graphs of chr20 and chr6 that do not contain HG002. The options were developed on chr20, and chr6 was held out. The kmers were counted from HG002 Illumina reads. The two arms are:

  • default: vg haplotypes with its default options, 32 haplotypes, and CHM13 and GRCh38 as reference paths. Reads were mapped to and called on the sampled graph.
  • model preset: the recommended commands above. Reads were mapped to the same sampling without the CHM13 path and called on the graph with it.

Illumina and ONT reads were mapped with Giraffe and called with vg call, using the same build of vg for both arms, and scored against the HG002 Q100 benchmark. The differences are model preset minus default, with 95% intervals from a paired bootstrap over 1 Mb blocks. A * marks an interval that excludes 0. Switch errors are counted against the benchmark's phasing.

contig reads ALL F1 SNV F1 indel F1 SV F1 switch errors
chr20 Illumina 0.9765 → 0.9825, +0.0060* 0.9901 → 0.9952, +0.0051* 0.9281 → 0.9373, +0.0092* 0.700 → 0.743, +0.043* 1341 → 822*
chr20 ONT 0.9793 → 0.9808, +0.0016* 0.9961 → 0.9966, +0.0005* 0.9176 → 0.9231, +0.0055* 0.813 → 0.825, +0.013 253 → 221*
chr6 Illumina 0.9846 → 0.9850, +0.0004 0.9951 → 0.9951, +0.0000 0.9463 → 0.9481, +0.0019* 0.719 → 0.729, +0.009 2825 → 1695*
chr6 ONT 0.9830 → 0.9840, +0.0009* 0.9972 → 0.9970, −0.0002 0.9302 → 0.9353, +0.0051* 0.814 → 0.793, −0.020 455 → 386*

The intervals for the largest differences are: chr20 Illumina ALL [+0.0036, +0.0089], indel [+0.0051, +0.0135] and SV [+0.017, +0.082]; chr6 ONT indel [+0.0036, +0.0065] and SV [−0.056, +0.014]. Most of the chr6 ONT SV difference is in one block, at 169 Mb.

The kmers are canonical, so the sampling cannot tell an inversion from its direct orientation. On chr6, the model preset does not sample HG002's maternal allele of a heterozygous 48 kb inversion at 26.6 Mb, and the calls there give both haplotypes the direct orientation.

The chr20 centromere is one subchain of 7.2 Mb. There, no candidate after the model pair is within the new-bp limit, so the sampled graph has two haplotypes in the centromere: the panel walks closest to HG002's paternal and maternal haplotypes over that subchain, by kmer distance. The chr6 graph has no haplotypes through its centromere.

Fewer reads map ambiguously. On chr20, Illumina reads with MAPQ below 5 fall from 1,079,940 to 626,528 and unmapped ones from 126,747 to 62,351; unmapped ONT reads fall from 851 to 325. On chr6, Illumina reads with MAPQ below 5 fall from 1,307,526 to 1,257,688.

Cost, with 8 threads:

  • Index: on chr20, building with --subchain-kmers takes 119 s with a peak of 13.1 to 14.2 GB, against 105 s and 12.5 to 13.3 GB without. The peaks vary between runs. The file is 374 MB against 253 MB without subchain kmers. On chr6 it is 691 MB against 667 MB.
  • Sampling: on chr20, 56 s and 8.2 GB against 39 s and 7.4 GB for the default.
  • Calling: on chr20, Illumina calling takes 115 s and 12.3 GB against 260 s and 16.0 GB.

🤖 Generated with Claude Code

benedictpaten and others added 8 commits November 15, 2026 17:19
With --subchain-kmers, the haplotype index represents the haplotypes in a
subchain with subchain kmers instead of minimizers with a single occurrence
in the graph. A subchain kmer is a minimizer whose every occurrence in the
graph is on a node that the haplotypes crossing the subchain visit. Such a
kmer can occur several times in a haplotype, so the subchain also stores the
number of copies of each kmer in each sequence, up to 5. A kmer is kept only
if the sequences differ in their copies of it.

Minimizers with a single occurrence leave out repeats, where haplotypes
differ by copy number. Subchain kmers keep the kmers of a repeat that occur
only within the subchain.

The haplotype index format becomes version 8, which adds the copies to each
subchain. An index with subchain kmers is written as version 8. Any other
index is written as version 7, so that earlier versions of vg can still read
it. Versions 5 to 7 can be read, and their subchains have no copies.

--validate checks the presence and copies of subchain kmers against the
sequences, and that some sequences have different copies of each kmer.

build_subchains now collects the sequences crossing every subchain of a
chain before it converts them. Without --subchain-kmers, the conversion is
unchanged. With --subchain-kmers and a single construction job, the
subchains of a chain are built in parallel.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…uity bonus

Greedy selection takes the haplotype with the best kmer score in each round.
In a subchain where the panel has rare long insertions, the best candidate
after the first can bring thousands of bp of node sequence that the sample
does not carry, for a small gain in score. These options control that:

- --max-new-bp N: after the first haplotype, a candidate is taken only if at
  most N bp of the nodes on its path through the subchain are new, that is,
  not visited by the haplotypes already taken. When no candidate qualifies,
  the selected haplotypes are repeated. The limit does not depend on the
  reference paths, so the selection is the same with and without
  --include-reference.
- --bp-escape F: a candidate over the limit still qualifies if it adds one
  kmer from the reads per F bp of new node sequence.
- --max-n-bp N: a candidate is taken only if its nodes in the subchain have
  at most N unknown (N) bases, which mark assembly gaps.
- --continuity-bonus F: F is added to the score of a candidate that a
  generated haplotype took in the previous subchain, so that generated
  haplotypes switch less often between panel haplotypes.

The selection of a subchain is now declared in recombinator.hpp, with the
context it needs, so that unit tests can call it. It always tries the
candidates in order of score, which takes the same candidate as before when
no limit applies.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each generated haplotype takes one of the selected haplotypes in each
subchain. It continues with the GBWT sequence it took in the previous
subchain if that sequence was selected again. By default, the other
generated haplotypes take the remaining selections in order.

With --similar-threading, a generated haplotype whose sequence in the
previous subchain also crosses this subchain instead takes the remaining
selection whose kmer presence in this subchain differs least from that
sequence. A generated haplotype then switches to a similar haplotype, which
keeps its sequence closer to the one it followed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…odel

With --model-pair, the first two haplotypes in each subchain are the pair
that best explains the sample's kmer counts, and the greedy selection adds
the rest. The greedy selection is unchanged: it scores kmer presence, not
copies. The pair is meant to be the sample's two haplotypes, so it is chosen
from a model of the counts given the copies.

The model has three parts, in artifacts/recombinator_model.{hpp,cpp}:

- A count model: the count of a kmer is negative binomial given its number
  of copies in the sample's two haplotypes. Its parameters and the weights of
  the copy numbers are fitted once to the counts of all subchain kmers, by
  maximum likelihood with box-constrained BFGS.
- Pair scores: in each subchain, haplotypes with the same kmers and copies
  form a group, and the score of a pair of groups is the log-likelihood of
  the counts given the copies the pair carries.
- A hidden Markov model over pairs along each chain: each member keeps its
  haplotype into the next subchain or moves to another group, at a fixed
  rate per bp. Pair scores are divided by a fixed constant before they are
  combined with the transitions. The Viterbi path gives the pair in each
  subchain.

The model uses the copies that an index built with --subchain-kmers stores.
With another index, every kmer in a haplotype has one copy.

The pair is exempt from --max-new-bp. A haplotype over --max-n-bp is not a
candidate for the pair, unless every haplotype that can be selected is over
it. The pair is not used in chains with the high- or half-coverage scoring
model. The count model assumes two copies of the genome, so --model-pair
cannot be combined with --haploid-scoring or --diploid-sampling.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the model pair

Unit tests cover the kmer copies in a subchain and their serialization,
including the version written with and without copies and reading a
subchain from version 7; the presence rows of a compressed kmer presence
matrix; the count model fit recovering known parameters; pair scores
against a direct sum over the kmers, and the choice of candidate groups;
the Viterbi path with changing numbers of candidates and the effect of tau;
the choice of sequences for the pair; the candidates for the pair in a
subchain; the new-bp and N-base limits, the escape, the continuity bonus
and the model pair in the selection of a subchain; threading by
similarity; and parameter validation.

The command-line tests build an index with subchain kmers, check that each
new sampling option changes the output on the test data, that the output
does not depend on the number of threads, that a high-coverage contig gets
no model pairs, and that invalid combinations of options are errors.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
--preset model sets the options of model-pair sampling at once: 32 haplotypes, the reference paths, the
continuity bonus, the new-bp limit and its escape, the N-base limit, threading by similarity and the model pair.
As with the other presets, options given after it override it.

The preset is meant for an index built with --subchain-kmers, so vg haplotypes stops with an error when the
haplotype information has no kmer copies. Haplotypes::has_copies() reports this, and file_version() now uses it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…p the model pair

Changes from linting the branch with SonarQube and from a review of it:

- Split the functions SonarQube found too complex along their parts: the
  BFGS step, the likelihood of the count model and the moves of the Viterbi
  algorithm; a SelectionLimits class that checks the new-bp and N-base
  limits; and a KmerCopyTally that counts the copies of the subchain kmers.
- When loading a subchain, check its copies, and check that its kmer
  presence matches its kmers and sequences. A corrupt index now stops with
  an error instead of crashing the model pair.
- Give paths without a phase a haplotype id of their own sample in the model.
- Walk the candidates for the model pair with a cached GBWT and read the
  sequence of each node once. This cuts the CPU time of sampling chr20 with
  the model preset by a third.
- Take the best candidate without sorting when it is within the limits.
- score_pairs() returns complete candidates, viterbi() rejects a subchain
  without candidates, and the selection limits need a fragment map.
- Make model_pairs() a private member of Recombinator, and let the index
  builder and the validator share one kmer count.
- Record the kmer score of a selected haplotype without the continuity
  bonus, and give the model pair its kmer scores.
- Reject the model pair with a scoring model other than the standard one.
- Fix comments, and make the tests check outputs and error messages.

The sampled graphs and the haplotype index are unchanged on chr20, and the
sampled graph is unchanged on chr6.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pair

- Group sequences with a std::map instead of a hand-written hash, and drop
  SequenceGroups, whose representative was always the first member.
- Move both members of the pair with one Viterbi step function.
- Use std::inner_product, std::accumulate, vg::matrix_multiply and
  gbwt::removeDuplicates where the code spelled them out.
- Name the per-sequence copy list once, as Haplotypes::Subchain::
  kmer_copies_type (pair_model::KmerCopies is an alias), and the count table
  as CountTable, which is now indexed as table[count][copies].
- Walk a sequence through a subchain in one place. build_subchain_kmers()
  collects nodes with subchain_walk_nodes(), so generate_haplotype() is as on
  master again, and subchain_haplotype() sets up the walk for both index
  builders.
- Pack kmer copies with Subchain::set_copies(), in the index builder and in
  the tests. It stores the values as given, in fields wide enough for them.
- Ask Recombinator::Parameters whether a new-bp or N-base limit is set, and
  let SelectionLimits read the parameters instead of copying them.
- Sum node lengths and N bases with NodeBaseCounts::totals().
- State each rule and definition once in the comments.
- Tests: check presence_row() with the other matrix queries, share helpers
  for kmer rows, round trips and the best pair, build the selection fixture
  with get_gbwt(), name the repeated sampling options in the TAP test, and
  drop checks that other tests already make (the symmetry of one pair score,
  two option conflicts that the unit tests cover, and an index difference
  that the preset checks imply).

The sampled graphs and both haplotype indexes, with and without subchain
kmers, are unchanged on chr20, and the sampled graph is unchanged on chr6.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
benedictpaten and others added 8 commits November 19, 2026 01:26
…er counts in runs

Sampling used to load the GBZ, then the .hapl, then read the KFF file, all on
one thread at a time:

- vg haplotypes now loads the .hapl on a second thread while the GBZ loads,
  and reports the time both took.
- ParallelKFFReader read one KFF block per library call under its mutex, so
  the reading threads ran one at a time. A raw section of single-kmer blocks,
  which KMC writes, is a run of fixed-size records, so the reader now copies
  a run with one read under the mutex and decodes it after releasing it.
  Other sections are read one block at a time as before. The reader walks the
  sections with the KFF library's section API instead of Kff_reader, and
  throws if a later section changes the kmer length or count size or has
  longer blocks, which the old reader would have misread.

The kmer counts and the sampled graphs are unchanged. On chr20 with 462
haplotypes (8 threads) the prologue before GBWT construction falls from
7.8 s to 5.1-5.7 s with default sampling and from 8.6 s to 5.3 s with the
model preset: the .hapl load is hidden behind the GBZ load, and reading the
62M kmers takes 1.3-1.4 s instead of 2.9-3.2 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
select_diploid() scored each of the 528 pairs of 32 candidates by walking
all kmers of the subchain with two presence-matrix iterators, which decode
each child sequence's differences from its parent again for every pair. On
chr20 with 60 haplotypes this was about 10 of the 32 CPU-s of diploid-preset
sampling, all on the one thread that generates the chain.

DiploidPairScorer decodes the presence rows of the candidates once and keeps
a mask of the absent, heterozygous and present kmers. A pair then costs five
popcounts per 64 kmers, and the integer score equals
KmerPresenceMatrix::diploid_score(), so the chosen pairs and the sampled
graphs are unchanged. Diploid-preset sampling on that graph (8 threads) goes
from 25.2 s to 13.7 s, with GBWT construction from 18.7 s to 9.0 s; the rest
is the faster loading from the previous commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
vg haplotypes built a gbwtgraph::MinimizerIndex over all haplotype windows,
and gbwtgraph::index_haplotypes() inserts every batch of minimizers inside
one OpenMP critical section. On the unclipped chr20 graph with 462
haplotypes the index took 213-232 s at 8 threads, most of it waiting for
that lock, and its hash table was sized from graph length: 2^29 cells
(8.6 GB) for 14.6M keys.

HaplotypeMinimizerIndex (artifacts/haplotype_minimizer_index.*) replaces it
in vg haplotypes and in the autoindex recipe for haplotype information. It
visits the same windows (gbwtgraph::for_each_haplotype_window) with the
window logic of index_haplotypes(), and its k(), w(), minimizers(),
count(), find() and get_value() have the same signatures and answers. The
partitioner and the validators change only through the typedef.

Design: sort-based, in buckets by key hash. Each thread buffers (key,
position) pairs, removes the duplicates in each 64K-pair buffer, and moves
the rest into deques of its own, one per bucket. Collection takes no lock.
Then each bucket is gathered from the deques of all threads, sorted and
deduplicated on one thread, and stored as sorted keys with end offsets
and a flat array of positions. A lookup finds the bucket by hash and
binary-searches it. The storage is sized by the occurrences found (plan
item 3.2): 8 bytes per occurrence and 16 per key.

The bucket comes from the MurmurHash3 finalizer, not from the minimizer
hash: minimizers are the kmers with the smallest hashes, so the top bits
of their hashes are almost all zero and would fill a few buckets.

A collected pair takes 16 bytes, twice an indexed occurrence, so the graph
is traversed twice, each time for half of the buckets. On the 462-haplotype
graph (index phase only, side by side, 8 threads) the previous index took
212.7 s and peaked at 13.3 GB; one traversal took 39.1 s and 12.05 GB; two
take 73.1 s and 9.64 GB.

The alternative was a hash table split into shards with a mutex each. It
would keep gbwtgraph's insertion into a sorted vector per key, whose cost
grows with the number of positions of the key, and the many-copy satellite
keys of unclipped centromeres would serialize inside their shard. Sorting
a bucket handles any number of positions per key, needs no lock, and gives
the same result on any number of threads.

A unit test compares every key's count and positions with gbwtgraph's
index on a graph of 1500 bubbles, on 1 and 4 threads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Building the haplotype information:

- The subchains of all chains are built as one set of tasks, largest first.
  Before, a graph with more than one construction job built the subchains of
  each chain on one thread, so chr1 alone would set the time for a whole
  genome. The cost of a task is its number of sequences times the bases of
  the walk of one of them. A task larger than half of a thread's share runs
  by itself on all threads.
- Each sequence is walked once through its subchain. SubchainWalk replaces
  generate_haplotype(), subchain_haplotype() and subchain_walk_nodes(), and
  sequences with the same walk share their kmers, which are found once.
- With subchain kmers, the copies of a kmer in a sequence are counted in a
  dense array (OrientedKmerCounts), and kmers are looked up in an
  open-addressing table (OrientedKmerTable) instead of a sparsepp map.
- The minimizer index from the previous commit traverses the graph once, as
  the second traversal saved 2.4 GB but doubled its time, and it is looked
  up by key, so the builder keeps keys instead of minimizers.

The format becomes version 9. A subchain with copies also stores the median
bases of the walks of its sequences and the N bases of each walk, which the
model pair and the N-base limit used to walk the sequences again to find.
Version 8 can still be read; its sequences are walked as before. --validate
checks the stored values.

Sampling:

- The model step takes the walk bases from the index, finds the candidates
  of all chains as one set of tasks, keeps the pair scores of a subchain in a
  triangle, and keeps only the best pairs in a bounded heap.
- Without a continuity bonus, which is every preset but the model preset,
  the selection in a subchain depends only on the subchain, and the
  selections of all subchains are made in parallel before the construction
  jobs.
- A scoring plan decodes the differences of the child sequences from their
  parents once per subchain, and children with the same differences are
  scored once. The model pair scores only its two sequences.
- The new-bp limit stops walking a candidate once it has too many new bases,
  and the N-base limit reads the stored N bases.
- The haplotypes extended over a subchain share a cache of GBWT records.

-t now sets the number of threads. It was capped at 16, and sampling used
0.85 times the default number of threads whatever -t said.
set_thread_count() now caps at the maximum it is given.

KmerPresenceMatrix::diploid_score() had no caller left after
DiploidPairScorer, and it is removed. Subchain::get_copies() replaces
for_each_copy(), whose callers all built that list. The fragment map in
SelectionContext is a reference, as every caller had one.

The haplotype information (apart from the version 9 fields) and the sampled
graphs are byte-identical to those of the previous code, with 1 and 8
threads, on the test graph, on chr20 with 60 and 462 haplotypes, and on a
graph of chr20 and chr6. Sampling from a version 8 index gives the same
graph as before.

On chr20 with 462 haplotypes at 8 threads, before these four commits and
after:

| Run | Before | After |
|---|---|---|
| Build, default | 1067 s, 11.7 GB | 105 s, 12.5 to 13.3 GB |
| Build, subchain kmers | 573 s, 14.3 GB | 119 s, 13.1 to 14.2 GB |
| Sample, default | 100 s | 39 s |
| Sample, model preset | 132 s | 56 s |
| Sample, diploid preset | 84 s | 13 s |

Unit tests cover the walks (artifacts/unittest/subchain_walk.cpp), the kmer
table and counts (artifacts/unittest/oriented_kmer_table.cpp), the stored
walk bases and version 8, scoring with a plan and of one sequence with
weights that round, and the N-base limit with stored N bases. The TAP test
checks that the haplotype information, large subchains built on all threads,
parallel selection and diploid sampling do not depend on the number of
threads, and validates stored N bases on a graph with an N node.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
std::jthread is missing from the libc++ of Xcode 16, which the Mac CI job
uses, so the loader thread is a std::thread that is joined after the graph
loads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ed gain

The haplotype information and the sampled graphs are byte-identical to those
of the previous commit, and the run times are the same within noise.

Removed from the model step, because turning each off made no measurable
difference to the time of sampling on chr20 with 462 haplotypes: the
triangle of pair scores and the bounded heap of best pairs (which saved
memory, not time), the candidates of all chains as one set of tasks, and the
parallel histogram. recombinator_model.cpp and model_pairs() are as they
were before the optimizations, except that the walk bases come from the
index.

Simplified:

- The format has one new version. Version 9 stores the copies and the walk
  bases together. Version 8 was only ever written by development builds of
  this branch. It is no longer a valid version, so loading such a file
  fails with the usual version error, and the second path that walked its
  sequences is gone.
- HaplotypeMinimizerIndex keeps each bucket as one sorted vector of
  (key, position) occurrences, and occurrences(key) returns the positions of
  a key as a span.
- Building a subchain no longer passes a `parallel` flag down: the loops use
  if(!omp_in_parallel()), each thread fills its own containers, and each
  task returns its fragment counts, so there is no named critical section.
- Sequences are grouped by walk with a std::map on the walks. A function
  replaces the NodeBaseCounts class, and sort_permutation() from utility.hpp
  replaces a new largest-first sort.
- SubchainWalk, OrientedKmerTable and OrientedKmerCounts move from
  recombinator.hpp to artifacts/subchain_walk.hpp.
- The haplotype information is loaded on a plain std::thread around
  load_from(), and an error in a kmer counting task is passed on as a
  std::exception_ptr.

Kept, because removing each one cost time: the scoring plan with shared
lists of differences, scoring only the two sequences of the model pair, the
kmer rows and nodes of a candidate found when first needed, the cache shared
by the haplotypes extended over a subchain, and the open-addressing kmer
table.

Tests: the diploid pair scorer is checked in the presence matrix test
instead of a file of its own. The tests of the walks, the kmer counts and
the minimizer index use smaller fixtures. The TAP test names the N node
instead of searching for one.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The format this pull request adds is now version 8, and every index is
written in it, as master writes every index in its newest version. Before
this commit an index with subchain kmers was written as version 9 and any
other index as version 7, so that an earlier vg could still read an index
built without subchain kmers.

In an index built without subchain kmers, the copies and the walk bases of
each subchain are empty. That adds 0.3% to the file on chr20 with 462
haplotypes (253.4 MB against 252.6 MB) and 7% on the small test graph.

Versions 5 to 7 are read as before. Earlier commits of this pull request
wrote a version 8 with copies but without walk bases. Such a file is not
recognized, and its index has to be rebuilt.

Subchain, TopLevelChain and Haplotypes serialize with the signatures they
have on master, without a version argument, and Haplotypes::file_version()
is removed with its test. The test of reading an earlier version reads the
start of a version 8 subchain as version 7.

The sampled graphs are byte-identical to those of the previous commit on
the test graph, on chr20 with 60 and 462 haplotypes, and on the graph of
chr20 and chr6, both from an index built by this commit and from a version
7 index built by the earlier code. The index with subchain kmers differs
from the previous one only in the version number.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
- The score of a parent sequence and the changes a child sequence makes to
  it are each computed in one function again (parent_score() and
  add_child_differences()). The two commits before this one had written
  the first loop three times and the second twice, in functions whose
  scores have to be equal.
- A failure to load the haplotype information is reported by the main
  thread after it joins the loading thread. The loading thread had called
  Haplotypes::load_from(), which exits the program, while the main thread
  was still loading the graph.
- The comments of the serialization functions no longer mention a version
  argument.
- Tests: SubchainWalk::operator< is checked on two walks with the same
  nodes and different pieces, and a subchain without copies is serialized
  and loaded. The TAP test of the stored N bases fails if the N node is
  missing, instead of counting that as a test of its own. The headers of
  the walk and kmer table tests describe what they now check, and two
  helpers no test used are removed.

The haplotype information and the sampled graphs are byte-identical to
those of the previous commit.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

inference Theme/team for genome inference (calling, assembly, etc.)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants