Repository navigation
Giraffe server engine - #4879
Conversation
Refactor Giraffe usage into a reusable in-process engine that loads indexes once and maps batched reads repeatedly, and add a giraffe-server subcommand with framed output/flush control for stable middleware request-response behavior.
adamnovak
left a comment
There was a problem hiding this comment.
@parsaeskandar You wanted me to check this for overly tight coupling to what Giraffe does. I don't think it should have to change too much when Giraffe changes. We won't be able to avoid touching it if we e.g. add a new thing MininizerMapper needs to have loaded in order to work, but I'm not sure there's a great way to avoid that.
But I think you might not connect to the Giraffe machinery quite enough, because there's no way here to e.g. set the mapping parameters that need to change between HiFi and R10 reads (and there's no code to set them to good values for pretending to be BLAT). I'm not sure we're even turning on the chain-based mapping. The Giraffe preset system is all tied in with its option parsing system, which makes using it from here difficult, but maybe it needs to be broken out somehow?
There also seems to be a profusion of little classes defined mostly in relation to each other, like HaplotypeSurjectionResult and GiraffeEnginePaths, which could do with more explanation of what they really represent or mean for someone who does not, to begin with, care about a HaplotypeSurjector or a GiraffeEngine.
# Conflicts: # autocomp.bash # autocomplete.sh
scripts/lint.py (formerly check_options.py) now also requires every subcommand in doc/vgmanmd.desc.md and doc/vgmanmd.py. That check arrived with the merge of master, so make test has failed on this branch since then. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolves the doc/vgmanmd.py conflict (upstream deprecated genotype and viz; keep giraffe-server in the list) and follows upstream's rename of min_chains -> target_alignment_attempts in the engine's hifi and blat presets, without which the engine no longer compiles. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d make the blat preset report near-best copies with length-independent MAPQ
…framed-output is gone
…g the whole batch
…- old thread pools junk gone
…stead of hanging or aborting
…nd the server names the haplotypes when it writes the tags
…rl boundaries in a method of its own
|
Thanks @adamnovak. I went through all of the code by hand this time. Here is a summary of what changed:
|
|
I also hand copied this into the giraffe-server-engine branch so the tests can run. |
adamnovak
left a comment
There was a problem hiding this comment.
I think this is almost ready. The only thing I found that I think really needs to change is that we're adding cigar_to_string() when we already have a cigar_string() that does almost the same thing. We should only have one of those.
| /// The result of surjecting an alignment onto a haplotype path: where it | ||
| /// lands, or why it could not be placed. Fields other than status are set | ||
| /// only when status is OK. | ||
| struct HaplotypeSurjectionResult { | ||
| enum class Status { | ||
| OK, ///< The alignment was surjected onto the target path. | ||
| EMPTY_INPUT, ///< The alignment has no graph path. | ||
| UNKNOWN_PATH, ///< The graph has no path with the target's name. | ||
| PATH_NOT_INDEXED, ///< The graph has the target path, but its positions are not indexed. | ||
| INCOMPATIBLE, ///< The target path does not carry the alignment's path. | ||
| SURJECTION_FAILED, ///< The surjector found no alignment on the target path. | ||
| }; | ||
|
|
||
| Status status = Status::OK; | ||
|
|
||
| /// Name of the target path. | ||
| std::string path_name; | ||
| /// 0-based start of the alignment on the path's forward strand. | ||
| int64_t path_position = -1; | ||
| /// True if the alignment is on the path's reverse strand. | ||
| bool path_reverse = false; | ||
| /// Score of the surjected alignment. | ||
| int32_t score = 0; | ||
| /// MAPQ of the surjected alignment, which the surjector copies from the | ||
| /// graph alignment. | ||
| int32_t mapping_quality = 0; | ||
| /// CIGAR of the surjected alignment. | ||
| std::string cigar; | ||
|
|
||
| /// Bases in each kind of edit of the surjected alignment. The CIGAR writes | ||
| /// matches and mismatches both as M, so these are counted from the edits. | ||
| uint64_t matches = 0; | ||
| uint64_t mismatches = 0; | ||
| uint64_t inserted = 0; ///< Read bases missing from the path. | ||
| uint64_t deleted = 0; ///< Path bases missing from the read. | ||
| }; | ||
|
|
||
| /// What GiraffeEngine::load() loads, and how the engine maps. | ||
| struct GiraffeEngineConfig { | ||
| /// The GBZ graph and Giraffe's minimizer, distance and zipcode indexes. | ||
| std::string gbz_path; | ||
| std::string minimizer_path; | ||
| std::string distance_path; | ||
| std::string zipcode_path; | ||
|
|
||
| /// Number of mapping threads. | ||
| size_t threads = 1; | ||
|
|
||
| /// Page the whole distance index into memory at load time. | ||
| bool preload_distance_index = true; | ||
|
|
||
| /// Names of the paths that map_reads() can surject onto. If empty, | ||
| /// map_reads() surjects nothing. | ||
| std::vector<std::string> surjection_target_paths; | ||
|
|
||
| /// Have the surjectors drop low-complexity anchors. vg giraffe turns this | ||
| /// on with --prune-low-cplx. | ||
| bool prune_suspicious_anchors = false; | ||
|
|
||
| /// Find the haplotypes that follow each alignment's path. | ||
| bool assign_haplotypes = true; | ||
| /// When assigning haplotypes, also find the boundary nodes of the | ||
| /// smallest snarl around each alignment's path. | ||
| bool find_snarl_boundaries = true; | ||
| }; | ||
|
|
||
| /// A read to map, and the path to surject its alignments onto. | ||
| struct GiraffeServerRequest { | ||
| /// The read's name, sequence and base qualities. The qualities are raw | ||
| /// Phred scores. If they are empty, base quality does not cap MAPQ. | ||
| Alignment read; | ||
| /// One of the surjection target paths, or empty to surject nothing. | ||
| std::string surjection_target; | ||
| }; | ||
|
|
||
| /// An alignment of a read, with the haplotypes that follow its path and its | ||
| /// surjection. A read can have several. | ||
| struct MappedAlignment { | ||
| Alignment alignment; | ||
| /// The runs of the alignment's path that haplotypes follow, in path | ||
| /// order. Unset if haplotype assignment is off. | ||
| std::optional<std::vector<HaplotypeSegment>> haplotype_segments; | ||
| /// Where the haplotypes that follow the whole path leave the snarl around | ||
| /// it. Both sides are unset if finding the boundaries is off. | ||
| SnarlBoundaries snarl_boundaries; | ||
| /// Where the alignment lands on the read's surjection target. Unset if | ||
| /// the read names no target. | ||
| std::optional<HaplotypeSurjectionResult> surjection; | ||
| }; |
There was a problem hiding this comment.
These still aren't great descriptions of what the classes are for, but I think that's because these aren't great classes. They're classes of convenience, with whatever fields some other thing seemed to need, rather than representing things in the domain model. I guess we can keep them, but we might end up refactoring this part eventually.
There was a problem hiding this comment.
Right, I couldn't figure a better structure other than this for now.
| /// The alignments of a read, best first. An unmapped read has one unmapped | ||
| /// alignment. | ||
| struct MappedRead { | ||
| std::vector<MappedAlignment> alignments; | ||
| /// Why the read could not be mapped. If set, there are no alignments. | ||
| std::optional<std::string> error; | ||
| }; |
There was a problem hiding this comment.
This is OK the way it is.
But this really wants to be a Rust-style "result", which is either the data or an error.
We don't get std::expected until C++23, but we could use std::variant from C++17 to express that we want either alignments or an error.
There was a problem hiding this comment.
Agreed, a std::variant is better. I left it as it is for now.
| /// Count the matched, mismatched, inserted and deleted bases in an alignment's | ||
| /// edits. | ||
| void count_edits(const Alignment& aln, uint64_t& matches, uint64_t& mismatches, |
There was a problem hiding this comment.
There's nothing like this in alignment.hpp? And this doesn't belong in alignment.hpp?
There was a problem hiding this comment.
Couldn't find the same functionality anywhere. The closest thing is what we do in the vg stats, but it is inline. We had a function called edit_count so I changed my function name to count_edit_bases to not be confusing.
Also the edit_count seems like a dead code. No one uses it and the code itself is wrong as it always returns 0.
Line 3606 in 9dd4512
Do you want me to remove that from codebase?
There was a problem hiding this comment.
Hmm. If you found code that's never called and also broken, we probably do want to delete it.
| /// Return a CIGAR as a string such as "55M", or "*" if it is empty. | ||
| string cigar_to_string(const vector<pair<int, char>>& cigar) { | ||
| if (cigar.empty()) { | ||
| return "*"; | ||
| } | ||
| stringstream ss; | ||
| for (const auto& op : cigar) { | ||
| ss << op.first << op.second; | ||
| } | ||
| return ss.str(); | ||
| } |
There was a problem hiding this comment.
We already have this, without the * fallback:
Line 99 in 9dd4512
It needs a doc comment though.
I found this with: semble search cigar_to_string.
We shouldn't add another copy of the logic. We might use a wrapper that returns * when the CIGAR is empty.
There was a problem hiding this comment.
Cool, I liked the semble search. I use the cigar_string now and also added a comment for it. I check if it's empty where I use it as I only have that in 1 place.
Changelog Entry
To be copied to the draft changelog by merger:
vg giraffe-serversubcommand that loads the Giraffe indexes once and maps queries streamed on stdin, with optional surjection onto any haplotype path.vg giraffepresetblat, which reports up to 20 near-best alignments with MAPQ independent of query length.Description
This PR adds a server-oriented in-process Giraffe mapping API and extends giraffe-server to emit structured graph-path payloads that middleware can forward to coordinate-translation code.