Repository navigation
haplotypes: subchain kmers with copy numbers, and a count-model pair for haplotype sampling - #5112
Open
benedictpaten wants to merge 16 commits into
Open
benedictpaten wants to merge 16 commits into
benedictpaten wants to merge 16 commits into
Conversation
With --subchain-kmers, the haplotype index represents the haplotypes in a subchain with subchain kmers instead of minimizers with a single occurrence in the graph. A subchain kmer is a minimizer whose every occurrence in the graph is on a node that the haplotypes crossing the subchain visit. Such a kmer can occur several times in a haplotype, so the subchain also stores the number of copies of each kmer in each sequence, up to 5. A kmer is kept only if the sequences differ in their copies of it. Minimizers with a single occurrence leave out repeats, where haplotypes differ by copy number. Subchain kmers keep the kmers of a repeat that occur only within the subchain. The haplotype index format becomes version 8, which adds the copies to each subchain. An index with subchain kmers is written as version 8. Any other index is written as version 7, so that earlier versions of vg can still read it. Versions 5 to 7 can be read, and their subchains have no copies. --validate checks the presence and copies of subchain kmers against the sequences, and that some sequences have different copies of each kmer. build_subchains now collects the sequences crossing every subchain of a chain before it converts them. Without --subchain-kmers, the conversion is unchanged. With --subchain-kmers and a single construction job, the subchains of a chain are built in parallel. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…uity bonus Greedy selection takes the haplotype with the best kmer score in each round. In a subchain where the panel has rare long insertions, the best candidate after the first can bring thousands of bp of node sequence that the sample does not carry, for a small gain in score. These options control that: - --max-new-bp N: after the first haplotype, a candidate is taken only if at most N bp of the nodes on its path through the subchain are new, that is, not visited by the haplotypes already taken. When no candidate qualifies, the selected haplotypes are repeated. The limit does not depend on the reference paths, so the selection is the same with and without --include-reference. - --bp-escape F: a candidate over the limit still qualifies if it adds one kmer from the reads per F bp of new node sequence. - --max-n-bp N: a candidate is taken only if its nodes in the subchain have at most N unknown (N) bases, which mark assembly gaps. - --continuity-bonus F: F is added to the score of a candidate that a generated haplotype took in the previous subchain, so that generated haplotypes switch less often between panel haplotypes. The selection of a subchain is now declared in recombinator.hpp, with the context it needs, so that unit tests can call it. It always tries the candidates in order of score, which takes the same candidate as before when no limit applies. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each generated haplotype takes one of the selected haplotypes in each subchain. It continues with the GBWT sequence it took in the previous subchain if that sequence was selected again. By default, the other generated haplotypes take the remaining selections in order. With --similar-threading, a generated haplotype whose sequence in the previous subchain also crosses this subchain instead takes the remaining selection whose kmer presence in this subchain differs least from that sequence. A generated haplotype then switches to a similar haplotype, which keeps its sequence closer to the one it followed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…odel
With --model-pair, the first two haplotypes in each subchain are the pair
that best explains the sample's kmer counts, and the greedy selection adds
the rest. The greedy selection is unchanged: it scores kmer presence, not
copies. The pair is meant to be the sample's two haplotypes, so it is chosen
from a model of the counts given the copies.
The model has three parts, in artifacts/recombinator_model.{hpp,cpp}:
- A count model: the count of a kmer is negative binomial given its number
of copies in the sample's two haplotypes. Its parameters and the weights of
the copy numbers are fitted once to the counts of all subchain kmers, by
maximum likelihood with box-constrained BFGS.
- Pair scores: in each subchain, haplotypes with the same kmers and copies
form a group, and the score of a pair of groups is the log-likelihood of
the counts given the copies the pair carries.
- A hidden Markov model over pairs along each chain: each member keeps its
haplotype into the next subchain or moves to another group, at a fixed
rate per bp. Pair scores are divided by a fixed constant before they are
combined with the transitions. The Viterbi path gives the pair in each
subchain.
The model uses the copies that an index built with --subchain-kmers stores.
With another index, every kmer in a haplotype has one copy.
The pair is exempt from --max-new-bp. A haplotype over --max-n-bp is not a
candidate for the pair, unless every haplotype that can be selected is over
it. The pair is not used in chains with the high- or half-coverage scoring
model. The count model assumes two copies of the genome, so --model-pair
cannot be combined with --haploid-scoring or --diploid-sampling.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the model pair Unit tests cover the kmer copies in a subchain and their serialization, including the version written with and without copies and reading a subchain from version 7; the presence rows of a compressed kmer presence matrix; the count model fit recovering known parameters; pair scores against a direct sum over the kmers, and the choice of candidate groups; the Viterbi path with changing numbers of candidates and the effect of tau; the choice of sequences for the pair; the candidates for the pair in a subchain; the new-bp and N-base limits, the escape, the continuity bonus and the model pair in the selection of a subchain; threading by similarity; and parameter validation. The command-line tests build an index with subchain kmers, check that each new sampling option changes the output on the test data, that the output does not depend on the number of threads, that a high-coverage contig gets no model pairs, and that invalid combinations of options are errors. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
--preset model sets the options of model-pair sampling at once: 32 haplotypes, the reference paths, the continuity bonus, the new-bp limit and its escape, the N-base limit, threading by similarity and the model pair. As with the other presets, options given after it override it. The preset is meant for an index built with --subchain-kmers, so vg haplotypes stops with an error when the haplotype information has no kmer copies. Haplotypes::has_copies() reports this, and file_version() now uses it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…p the model pair Changes from linting the branch with SonarQube and from a review of it: - Split the functions SonarQube found too complex along their parts: the BFGS step, the likelihood of the count model and the moves of the Viterbi algorithm; a SelectionLimits class that checks the new-bp and N-base limits; and a KmerCopyTally that counts the copies of the subchain kmers. - When loading a subchain, check its copies, and check that its kmer presence matches its kmers and sequences. A corrupt index now stops with an error instead of crashing the model pair. - Give paths without a phase a haplotype id of their own sample in the model. - Walk the candidates for the model pair with a cached GBWT and read the sequence of each node once. This cuts the CPU time of sampling chr20 with the model preset by a third. - Take the best candidate without sorting when it is within the limits. - score_pairs() returns complete candidates, viterbi() rejects a subchain without candidates, and the selection limits need a fragment map. - Make model_pairs() a private member of Recombinator, and let the index builder and the validator share one kmer count. - Record the kmer score of a selected haplotype without the continuity bonus, and give the model pair its kmer scores. - Reject the model pair with a scoring model other than the standard one. - Fix comments, and make the tests check outputs and error messages. The sampled graphs and the haplotype index are unchanged on chr20, and the sampled graph is unchanged on chr6. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pair - Group sequences with a std::map instead of a hand-written hash, and drop SequenceGroups, whose representative was always the first member. - Move both members of the pair with one Viterbi step function. - Use std::inner_product, std::accumulate, vg::matrix_multiply and gbwt::removeDuplicates where the code spelled them out. - Name the per-sequence copy list once, as Haplotypes::Subchain:: kmer_copies_type (pair_model::KmerCopies is an alias), and the count table as CountTable, which is now indexed as table[count][copies]. - Walk a sequence through a subchain in one place. build_subchain_kmers() collects nodes with subchain_walk_nodes(), so generate_haplotype() is as on master again, and subchain_haplotype() sets up the walk for both index builders. - Pack kmer copies with Subchain::set_copies(), in the index builder and in the tests. It stores the values as given, in fields wide enough for them. - Ask Recombinator::Parameters whether a new-bp or N-base limit is set, and let SelectionLimits read the parameters instead of copying them. - Sum node lengths and N bases with NodeBaseCounts::totals(). - State each rule and definition once in the comments. - Tests: check presence_row() with the other matrix queries, share helpers for kmer rows, round trips and the best pair, build the selection fixture with get_gbwt(), name the repeated sampling options in the TAP test, and drop checks that other tests already make (the symmetry of one pair score, two option conflicts that the unit tests cover, and an index difference that the preset checks imply). The sampled graphs and both haplotype indexes, with and without subchain kmers, are unchanged on chr20, and the sampled graph is unchanged on chr6. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er counts in runs Sampling used to load the GBZ, then the .hapl, then read the KFF file, all on one thread at a time: - vg haplotypes now loads the .hapl on a second thread while the GBZ loads, and reports the time both took. - ParallelKFFReader read one KFF block per library call under its mutex, so the reading threads ran one at a time. A raw section of single-kmer blocks, which KMC writes, is a run of fixed-size records, so the reader now copies a run with one read under the mutex and decodes it after releasing it. Other sections are read one block at a time as before. The reader walks the sections with the KFF library's section API instead of Kff_reader, and throws if a later section changes the kmer length or count size or has longer blocks, which the old reader would have misread. The kmer counts and the sampled graphs are unchanged. On chr20 with 462 haplotypes (8 threads) the prologue before GBWT construction falls from 7.8 s to 5.1-5.7 s with default sampling and from 8.6 s to 5.3 s with the model preset: the .hapl load is hidden behind the GBZ load, and reading the 62M kmers takes 1.3-1.4 s instead of 2.9-3.2 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
select_diploid() scored each of the 528 pairs of 32 candidates by walking all kmers of the subchain with two presence-matrix iterators, which decode each child sequence's differences from its parent again for every pair. On chr20 with 60 haplotypes this was about 10 of the 32 CPU-s of diploid-preset sampling, all on the one thread that generates the chain. DiploidPairScorer decodes the presence rows of the candidates once and keeps a mask of the absent, heterozygous and present kmers. A pair then costs five popcounts per 64 kmers, and the integer score equals KmerPresenceMatrix::diploid_score(), so the chosen pairs and the sampled graphs are unchanged. Diploid-preset sampling on that graph (8 threads) goes from 25.2 s to 13.7 s, with GBWT construction from 18.7 s to 9.0 s; the rest is the faster loading from the previous commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
vg haplotypes built a gbwtgraph::MinimizerIndex over all haplotype windows, and gbwtgraph::index_haplotypes() inserts every batch of minimizers inside one OpenMP critical section. On the unclipped chr20 graph with 462 haplotypes the index took 213-232 s at 8 threads, most of it waiting for that lock, and its hash table was sized from graph length: 2^29 cells (8.6 GB) for 14.6M keys. HaplotypeMinimizerIndex (artifacts/haplotype_minimizer_index.*) replaces it in vg haplotypes and in the autoindex recipe for haplotype information. It visits the same windows (gbwtgraph::for_each_haplotype_window) with the window logic of index_haplotypes(), and its k(), w(), minimizers(), count(), find() and get_value() have the same signatures and answers. The partitioner and the validators change only through the typedef. Design: sort-based, in buckets by key hash. Each thread buffers (key, position) pairs, removes the duplicates in each 64K-pair buffer, and moves the rest into deques of its own, one per bucket. Collection takes no lock. Then each bucket is gathered from the deques of all threads, sorted and deduplicated on one thread, and stored as sorted keys with end offsets and a flat array of positions. A lookup finds the bucket by hash and binary-searches it. The storage is sized by the occurrences found (plan item 3.2): 8 bytes per occurrence and 16 per key. The bucket comes from the MurmurHash3 finalizer, not from the minimizer hash: minimizers are the kmers with the smallest hashes, so the top bits of their hashes are almost all zero and would fill a few buckets. A collected pair takes 16 bytes, twice an indexed occurrence, so the graph is traversed twice, each time for half of the buckets. On the 462-haplotype graph (index phase only, side by side, 8 threads) the previous index took 212.7 s and peaked at 13.3 GB; one traversal took 39.1 s and 12.05 GB; two take 73.1 s and 9.64 GB. The alternative was a hash table split into shards with a mutex each. It would keep gbwtgraph's insertion into a sorted vector per key, whose cost grows with the number of positions of the key, and the many-copy satellite keys of unclipped centromeres would serialize inside their shard. Sorting a bucket handles any number of positions per key, needs no lock, and gives the same result on any number of threads. A unit test compares every key's count and positions with gbwtgraph's index on a graph of 1500 bubbles, on 1 and 4 threads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Building the haplotype information: - The subchains of all chains are built as one set of tasks, largest first. Before, a graph with more than one construction job built the subchains of each chain on one thread, so chr1 alone would set the time for a whole genome. The cost of a task is its number of sequences times the bases of the walk of one of them. A task larger than half of a thread's share runs by itself on all threads. - Each sequence is walked once through its subchain. SubchainWalk replaces generate_haplotype(), subchain_haplotype() and subchain_walk_nodes(), and sequences with the same walk share their kmers, which are found once. - With subchain kmers, the copies of a kmer in a sequence are counted in a dense array (OrientedKmerCounts), and kmers are looked up in an open-addressing table (OrientedKmerTable) instead of a sparsepp map. - The minimizer index from the previous commit traverses the graph once, as the second traversal saved 2.4 GB but doubled its time, and it is looked up by key, so the builder keeps keys instead of minimizers. The format becomes version 9. A subchain with copies also stores the median bases of the walks of its sequences and the N bases of each walk, which the model pair and the N-base limit used to walk the sequences again to find. Version 8 can still be read; its sequences are walked as before. --validate checks the stored values. Sampling: - The model step takes the walk bases from the index, finds the candidates of all chains as one set of tasks, keeps the pair scores of a subchain in a triangle, and keeps only the best pairs in a bounded heap. - Without a continuity bonus, which is every preset but the model preset, the selection in a subchain depends only on the subchain, and the selections of all subchains are made in parallel before the construction jobs. - A scoring plan decodes the differences of the child sequences from their parents once per subchain, and children with the same differences are scored once. The model pair scores only its two sequences. - The new-bp limit stops walking a candidate once it has too many new bases, and the N-base limit reads the stored N bases. - The haplotypes extended over a subchain share a cache of GBWT records. -t now sets the number of threads. It was capped at 16, and sampling used 0.85 times the default number of threads whatever -t said. set_thread_count() now caps at the maximum it is given. KmerPresenceMatrix::diploid_score() had no caller left after DiploidPairScorer, and it is removed. Subchain::get_copies() replaces for_each_copy(), whose callers all built that list. The fragment map in SelectionContext is a reference, as every caller had one. The haplotype information (apart from the version 9 fields) and the sampled graphs are byte-identical to those of the previous code, with 1 and 8 threads, on the test graph, on chr20 with 60 and 462 haplotypes, and on a graph of chr20 and chr6. Sampling from a version 8 index gives the same graph as before. On chr20 with 462 haplotypes at 8 threads, before these four commits and after: | Run | Before | After | |---|---|---| | Build, default | 1067 s, 11.7 GB | 105 s, 12.5 to 13.3 GB | | Build, subchain kmers | 573 s, 14.3 GB | 119 s, 13.1 to 14.2 GB | | Sample, default | 100 s | 39 s | | Sample, model preset | 132 s | 56 s | | Sample, diploid preset | 84 s | 13 s | Unit tests cover the walks (artifacts/unittest/subchain_walk.cpp), the kmer table and counts (artifacts/unittest/oriented_kmer_table.cpp), the stored walk bases and version 8, scoring with a plan and of one sequence with weights that round, and the N-base limit with stored N bases. The TAP test checks that the haplotype information, large subchains built on all threads, parallel selection and diploid sampling do not depend on the number of threads, and validates stored N bases on a graph with an N node. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
std::jthread is missing from the libc++ of Xcode 16, which the Mac CI job uses, so the loader thread is a std::thread that is joined after the graph loads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ed gain The haplotype information and the sampled graphs are byte-identical to those of the previous commit, and the run times are the same within noise. Removed from the model step, because turning each off made no measurable difference to the time of sampling on chr20 with 462 haplotypes: the triangle of pair scores and the bounded heap of best pairs (which saved memory, not time), the candidates of all chains as one set of tasks, and the parallel histogram. recombinator_model.cpp and model_pairs() are as they were before the optimizations, except that the walk bases come from the index. Simplified: - The format has one new version. Version 9 stores the copies and the walk bases together. Version 8 was only ever written by development builds of this branch. It is no longer a valid version, so loading such a file fails with the usual version error, and the second path that walked its sequences is gone. - HaplotypeMinimizerIndex keeps each bucket as one sorted vector of (key, position) occurrences, and occurrences(key) returns the positions of a key as a span. - Building a subchain no longer passes a `parallel` flag down: the loops use if(!omp_in_parallel()), each thread fills its own containers, and each task returns its fragment counts, so there is no named critical section. - Sequences are grouped by walk with a std::map on the walks. A function replaces the NodeBaseCounts class, and sort_permutation() from utility.hpp replaces a new largest-first sort. - SubchainWalk, OrientedKmerTable and OrientedKmerCounts move from recombinator.hpp to artifacts/subchain_walk.hpp. - The haplotype information is loaded on a plain std::thread around load_from(), and an error in a kmer counting task is passed on as a std::exception_ptr. Kept, because removing each one cost time: the scoring plan with shared lists of differences, scoring only the two sequences of the model pair, the kmer rows and nodes of a candidate found when first needed, the cache shared by the haplotypes extended over a subchain, and the open-addressing kmer table. Tests: the diploid pair scorer is checked in the presence matrix test instead of a file of its own. The tests of the walks, the kmer counts and the minimizer index use smaller fixtures. The TAP test names the N node instead of searching for one. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The format this pull request adds is now version 8, and every index is written in it, as master writes every index in its newest version. Before this commit an index with subchain kmers was written as version 9 and any other index as version 7, so that an earlier vg could still read an index built without subchain kmers. In an index built without subchain kmers, the copies and the walk bases of each subchain are empty. That adds 0.3% to the file on chr20 with 462 haplotypes (253.4 MB against 252.6 MB) and 7% on the small test graph. Versions 5 to 7 are read as before. Earlier commits of this pull request wrote a version 8 with copies but without walk bases. Such a file is not recognized, and its index has to be rebuilt. Subchain, TopLevelChain and Haplotypes serialize with the signatures they have on master, without a version argument, and Haplotypes::file_version() is removed with its test. The test of reading an earlier version reads the start of a version 8 subchain as version 7. The sampled graphs are byte-identical to those of the previous commit on the test graph, on chr20 with 60 and 462 haplotypes, and on the graph of chr20 and chr6, both from an index built by this commit and from a version 7 index built by the earlier code. The index with subchain kmers differs from the previous one only in the version number. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
- The score of a parent sequence and the changes a child sequence makes to it are each computed in one function again (parent_score() and add_child_differences()). The two commits before this one had written the first loop three times and the second twice, in functions whose scores have to be equal. - A failure to load the haplotype information is reported by the main thread after it joins the loading thread. The loading thread had called Haplotypes::load_from(), which exits the program, while the main thread was still loading the graph. - The comments of the serialization functions no longer mention a version argument. - Tests: SubchainWalk::operator< is checked on two walks with the same nodes and different pieces, and a subchain without copies is serialized and loaded. The TAP test of the stored N bases fails if the N node is missing, instead of counting that as a test of its own. The headers of the walk and kmer table tests describe what they now check, and two helpers no test used are removed. The haplotype information and the sampled graphs are byte-identical to those of the previous commit. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
vg haplotypessamples a personal graph from a pangenome graph. The haplotypes in the graph are the panel haplotypes. The haplotype index divides each chain of the graph into subchains, which are short stretches between snarl boundaries, and it records which kmers each panel haplotype has in each subchain. To sample,vg haplotypescounts the kmers in the sample's reads. In each subchain, the greedy selection repeatedly takes the panel haplotype with the best kmer score, and the generated haplotypes are joined from the panel haplotypes taken in consecutive subchains. This pull request adds options for both the index and the sampling.When the haplotype index is built, subchains can be represented by subchain kmers with their copy numbers. A subchain kmer is a minimizer whose every occurrence in the graph is on a node that the haplotypes crossing the subchain visit. Unlike a minimizer with a single occurrence in the graph, such a kmer can occur several times in a haplotype, so the index also stores how many times each kmer occurs in each haplotype (its copies, up to 5). This keeps kmers in repeats, where haplotypes differ by copy number.
When the haplotypes are sampled, the first two haplotypes in each subchain can come from a model of the kmer counts instead of the greedy selection. The count of a kmer is negative binomial given its copies in the sample's two haplotypes. The model is fitted once to the counts of all subchain kmers. In each subchain, the score of a pair of haplotypes is the log-likelihood of the counts given the copies the pair carries. A hidden Markov model over pairs follows each chain: each member of the pair keeps its haplotype into the next subchain, or moves to another haplotype at a fixed rate per bp. The Viterbi path gives the pair in each subchain. The greedy selection then adds the other haplotypes.
The greedy selection gets limits on the sequence a new haplotype adds, a bonus for continuing a generated haplotype, and a way of matching generated haplotypes with the selected ones by kmer similarity.
The pull request also makes building the haplotype information and sampling several times faster, and
-tnow sets the number of threads (see Speed). The output does not change: the haplotype information and the sampled graphs are the same as before.The default behaviour of
vg haplotypesdoes not change: an index built without--subchain-kmersholds the same kmers and gives the same sampled graphs as before. Every haplotype index is now written as version 8, which versions of vg without this change cannot read.Recommended commands
Build the haplotype index with subchain kmers:
Sample 32 haplotypes with the model pair, keeping the CHM13 reference path:
The model preset is the same as these options:
Like the other presets, it replaces the sampling options given before it, and options after it override it. Like the haploid, diploid, hifi and ont presets, it includes the reference paths. To map reads to the sampled haplotypes without CHM13, so that CHM13 does not draw reads away from them, remove the CHM13 paths from the sampled graph:
The first command copies out the GBWT of
sample.gbz, becausevg gbwtcannot remove paths from the GBWT inside a GBZ file. The second removes the paths whose sample name is CHM13 and writes the rest.sample.noref.gbzleaves out the nodes that only CHM13 visits and keeps the other nodes with their IDs, sovg call --read-likelihoodcan call reads mapped to it onsample.gbz. With--set-reference CHM13, CHM13 is the only reference sample, andsample.noref.gbzis the same graph that sampling a second time without--include-referencewould make.New options
In the descriptions below, a candidate is a panel haplotype in a subchain that the selection can take, and the new nodes of a candidate are the nodes on its path through the subchain that no haplotype already taken visits.
--subchain-kmers: build the index with subchain kmers and store their copies in each haplotype.--continuity-bonus F: add F to the score of a candidate that a generated haplotype took in the previous subchain [0].--max-new-bp N: after the first haplotype in a subchain, take a candidate only if its new nodes have at most N bp [no limit]. Reference paths do not count, so the selection is the same with and without--include-reference.--bp-escape F: also take a candidate over--max-new-bpif it adds one kmer from the reads that the haplotypes already taken lack per F bp of new nodes (0: never) [0].--max-n-bp N: take a candidate only if its nodes in the subchain have at most N unknown (N) bases [no limit].--similar-threading: when a generated haplotype cannot continue with its panel haplotype, give it the unused selection whose kmers in the subchain differ least from that panel haplotype. Without it, the generated haplotype takes the next unused selection.--model-pair: take the first two haplotypes in each subchain from the count model and the hidden Markov model over pairs.--preset model: the options above, as listed under the recommended commands. It needs an index built with--subchain-kmers, andvg haplotypesstops with an error on an index without kmer copies.When no candidate is within
--max-new-bpor--max-n-bp, the selected haplotypes are repeated, as they are when a subchain has too few haplotypes. If no candidate for the first haplotype has few enough N bases, the best one is taken.The model pair is exempt from
--max-new-bp. A haplotype with more than--max-n-bpN bases is not a candidate for the pair, unless every haplotype that can be selected has that many. The pair is not used in chains sampled with the high- or half-coverage scoring model. The count model assumes two copies of the genome, so--model-paircannot be combined with--haploid-scoring(or--preset haploid) or--diploid-sampling.Speed
On a large node,
vg haplotypesused few of its threads:-twas capped at 16, and sampling used 0.85 times the default number of threads, whatever-tsaid. Now-tsets the number of threads.Building the haplotype information also does less work. Each haplotype is walked once through each subchain, and haplotypes with the same walk share their kmers. Kmer copies are counted in an array, with an open-addressing table for the kmers.
Sampling:
--preset model), the selections in all subchains are made in parallel.On chr20 with 462 haplotypes and 8 threads, before this work and after:
--subchain-kmers--preset model--preset diploidOn a graph of chr20 and chr6 with 60 haplotypes, building with
--subchain-kmerstakes 80 s instead of 671 s. Before, chr6 was built on one thread.The sampled graphs were compared byte for byte with the previous code, and they matched:
Haplotype index format
The haplotype index format becomes version 8. It adds two things to each subchain:
--max-n-bpuse.Every index is written as version 8. In an index built without
--subchain-kmersthe two additions are empty, which adds 0.3% to the file on chr20 (253.4 MB against 252.6 MB).Versions 5 to 7 can be read. Their subchains have no copies and no walk bases. Earlier commits of this pull request wrote a different version 8, with the copies but without the walk bases. Such a file is not recognized, and its index has to be rebuilt.
Loading checks that:
vg haplotypes --validatechecks, against the haplotype sequences:Code
src/recombinator_haplotypes.{hpp,cpp}:Haplotypes::Subchain, their serialization and checks, and the choice of file version;presence_row(), which returns the kmer presence of a sequence as 64-bit words;KmerPresenceMatrixandscore_sequence();src/recombinator.{hpp,cpp}:HaplotypePartitioner::partition_haplotypes()collects the subchains and sequences of every chain, then builds all subchains as one set of tasks withbuild_subchain().SubchainWalkwalks a sequence through a subchain, and sequences with the same walk are grouped (SubchainWalks) so that their kmers are found once. With--subchain-kmers,build_subchain_kmers()keeps the minimizers whose occurrences are all on the subchain's nodes, counts their copies withOrientedKmerTable,OrientedKmerCountsandKmerCopyTally, and writes the informative ones with the walk bases. Without it,build_unique_kmers()keeps the minimizers with a single occurrence, as before.select_haplotypes()applies the model pair, the limits and the continuity bonus. It is declared in the header with aRecombinator::SelectionContextthat carries what it needs beyond the kmers, so that unit tests can call it. ASelectionLimitsobject in the source file checks a candidate against the new-bp and N-base limits; it walks a candidate only when it is first checked, and only until the candidate has too many new bases.Recombinator::select_in_parallel()makes the selections of all subchains in advance when there is no continuity bonus.thread_by_similarity()implements--similar-threading.model_candidates()finds the candidates for the pair in a subchain.Recombinator::model_pairs()fits the count model and runs the hidden Markov model on each chain before the haplotypes are generated.artifacts/recombinator_model.{hpp,cpp}: the count model, its fit, the pair scores and the Viterbi algorithm, innamespace pair_model. These functions do not use the graph, so the tests can give them synthetic data.artifacts/haplotype_minimizer_index.{hpp,cpp}: the minimizer index thatvg haplotypesbuilds, without a lock.artifacts/subchain_walk.{hpp,cpp}:SubchainWalk, and theOrientedKmerTableandOrientedKmerCountsthat count the copies of kmers in a walk.artifacts/diploid_pair_scorer.{hpp,cpp}: the pair scores of diploid sampling.src/kff.{hpp,cpp}:ParallelKFFReadercopies runs of single-kmer records under its lock and decodes them outside it.src/subcommand/haplotypes_main.cpp:-t;Tests
vg test "[haplotypes]"):presence_row()against the rows it was built from, on compressed matrices;[subchain_walk]);[oriented_kmer_table]);[haplotype_minimizer_index]), the diploid pair scores against a direct sum, and the KFF reader on sections of several kinds;test/t/54_vg_haplotypes.tbuilds an index with subchain kmers on the micb-kir3dl1 test graph and samples with the recommended options. It checks that the model pair, threading by similarity, the continuity bonus, the new-bp limit and its escape each change the output; that the generated haplotypes do not depend on the reference paths; that the output does not depend on the number of threads, for the haplotype information, for subchains built on all threads, for selection without a continuity bonus and for diploid sampling; that the stored N bases of the walks validate on a graph with an N node; that a high-coverage contig is sampled the same with and without the model pair; that the model preset gives the same output as its options and stops on an index without subchain kmers; and that invalid combinations of options are errors with the right messages.Results
HG002 was sampled from HPRC Minigraph-Cactus graphs of chr20 and chr6 that do not contain HG002. The options were developed on chr20, and chr6 was held out. The kmers were counted from HG002 Illumina reads. The two arms are:
vg haplotypeswith its default options, 32 haplotypes, and CHM13 and GRCh38 as reference paths. Reads were mapped to and called on the sampled graph.Illumina and ONT reads were mapped with Giraffe and called with
vg call, using the same build of vg for both arms, and scored against the HG002 Q100 benchmark. The differences are model preset minus default, with 95% intervals from a paired bootstrap over 1 Mb blocks. A*marks an interval that excludes 0. Switch errors are counted against the benchmark's phasing.The intervals for the largest differences are: chr20 Illumina ALL [+0.0036, +0.0089], indel [+0.0051, +0.0135] and SV [+0.017, +0.082]; chr6 ONT indel [+0.0036, +0.0065] and SV [−0.056, +0.014]. Most of the chr6 ONT SV difference is in one block, at 169 Mb.
The kmers are canonical, so the sampling cannot tell an inversion from its direct orientation. On chr6, the model preset does not sample HG002's maternal allele of a heterozygous 48 kb inversion at 26.6 Mb, and the calls there give both haplotypes the direct orientation.
The chr20 centromere is one subchain of 7.2 Mb. There, no candidate after the model pair is within the new-bp limit, so the sampled graph has two haplotypes in the centromere: the panel walks closest to HG002's paternal and maternal haplotypes over that subchain, by kmer distance. The chr6 graph has no haplotypes through its centromere.
Fewer reads map ambiguously. On chr20, Illumina reads with MAPQ below 5 fall from 1,079,940 to 626,528 and unmapped ones from 126,747 to 62,351; unmapped ONT reads fall from 851 to 325. On chr6, Illumina reads with MAPQ below 5 fall from 1,307,526 to 1,257,688.
Cost, with 8 threads:
--subchain-kmerstakes 119 s with a peak of 13.1 to 14.2 GB, against 105 s and 12.5 to 13.3 GB without. The peaks vary between runs. The file is 374 MB against 253 MB without subchain kmers. On chr6 it is 691 MB against 667 MB.🤖 Generated with Claude Code