Skip to content

Giraffe server engine - #4879

Merged
parsaeskandar merged 63 commits into
vgteam:masterfrom
parsaeskandar:giraffe-server-engine
Nov 21, 2026
Merged

parsaeskandar merged 63 commits into
vgteam:masterfrom
parsaeskandar:giraffe-server-engine

Conversation

@parsaeskandar

@parsaeskandar parsaeskandar commented Apr 20, 2026 •

Copy link
Copy Markdown
Contributor

Changelog Entry

To be copied to the draft changelog by merger:

  • New vg giraffe-server subcommand that loads the Giraffe indexes once and maps queries streamed on stdin, with optional surjection onto any haplotype path.
  • vg giraffe preset blat, which reports up to 20 near-best alignments with MAPQ independent of query length.

Description

This PR adds a server-oriented in-process Giraffe mapping API and extends giraffe-server to emit structured graph-path payloads that middleware can forward to coordinate-translation code.

parsaeskandar and others added 3 commits April 16, 2026 09:03
Refactor Giraffe usage into a reusable in-process engine that loads indexes once and maps batched reads repeatedly, and add a giraffe-server subcommand with framed output/flush control for stable middleware request-response behavior.
@parsaeskandar
parsaeskandar requested a review from adamnovak April 20, 2026 17:53

@adamnovak adamnovak left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@parsaeskandar You wanted me to check this for overly tight coupling to what Giraffe does. I don't think it should have to change too much when Giraffe changes. We won't be able to avoid touching it if we e.g. add a new thing MininizerMapper needs to have loaded in order to work, but I'm not sure there's a great way to avoid that.

But I think you might not connect to the Giraffe machinery quite enough, because there's no way here to e.g. set the mapping parameters that need to change between HiFi and R10 reads (and there's no code to set them to good values for pretending to be BLAT). I'm not sure we're even turning on the chain-based mapping. The Giraffe preset system is all tied in with its option parsing system, which makes using it from here difficult, but maybe it needs to be broken out somehow?

There also seems to be a profusion of little classes defined mostly in relation to each other, like HaplotypeSurjectionResult and GiraffeEnginePaths, which could do with more explanation of what they really represent or mean for someone who does not, to begin with, care about a HaplotypeSurjector or a GiraffeEngine.

Comment thread src/subcommand/giraffe_server_main.cpp Resolved
Comment thread src/subcommand/giraffe_server_main.cpp Resolved
Comment thread src/subcommand/giraffe_server_main.cpp Resolved
Comment thread src/subcommand/giraffe_server_main.cpp Resolved
Comment thread src/subcommand/giraffe_server_main.cpp Resolved
Comment thread src/haplotype_assignment.hpp Resolved
Comment thread src/giraffe_engine.hpp Resolved
Comment thread src/giraffe_engine.hpp Resolved
Comment thread src/giraffe_engine.hpp Resolved
Comment thread src/giraffe_engine.hpp Resolved
@parsaeskandar parsaeskandar linked an issue Sep 21, 2026 that may be closed by this pull request
parsaeskandar and others added 4 commits October 3, 2026 12:32
scripts/lint.py (formerly check_options.py) now also requires every
subcommand in doc/vgmanmd.desc.md and doc/vgmanmd.py. That check arrived with
the merge of master, so make test has failed on this branch since then.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolves the doc/vgmanmd.py conflict (upstream deprecated genotype and viz;
keep giraffe-server in the list) and follows upstream's rename of
min_chains -> target_alignment_attempts in the engine's hifi and blat
presets, without which the engine no longer compiles.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@adamnovak adamnovak added the mapping Theme/team for mapping to pangenomes label Nov 2, 2026
…nd the server names the haplotypes when it writes the tags
@parsaeskandar

Copy link
Copy Markdown
Contributor Author

Thanks @adamnovak. I went through all of the code by hand this time. Here is a summary of what changed:

  • vg giraffe and vg giraffe-server both fill giraffe option parser now. Both call set_up_giraffe_mapper() so all presetts, options, rec mode, ... are the same code in both.
  • The PANGENOME_* environment variables are all gone.
  • In haplotype_assignment, the assigner now answers two separate questions about an alignment’s path.
    • Which haplotypes follow the path? In HaplotypeSegment
    • Where do those haplotypes leave the snarl around the path? snarl_boundaries().
  • in the vg giraffe-server I put the rule for requests and errors at the top of the giraffe_server_main.cpp. Now every request get exactly one frame (a result, or an error) and the frames are the only output format. The nonsense threadpool is not gone and I am using OpenMP threading.
  • Changed Engine to know nothing about GAF now and the server now writes GAF lines and tags.
  • I added a wiki that should be enough to for using vg giraffe-server: https://github.com/vgteam/vg/wiki/Giraffe-server-protocol
  • The timings, and FLUSH_NOW are gone.
  • Also the client can see the relevant errors on what went wrong, but one error or bad input won't break the whole process.

@parsaeskandar

Copy link
Copy Markdown
Contributor Author

I also hand copied this into the giraffe-server-engine branch so the tests can run.

@adamnovak adamnovak left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is almost ready. The only thing I found that I think really needs to change is that we're adding cigar_to_string() when we already have a cigar_string() that does almost the same thing. We should only have one of those.

Comment thread src/giraffe_engine.hpp
Comment on lines +25 to +113
/// The result of surjecting an alignment onto a haplotype path: where it
/// lands, or why it could not be placed. Fields other than status are set
/// only when status is OK.
struct HaplotypeSurjectionResult {
enum class Status {
OK, ///< The alignment was surjected onto the target path.
EMPTY_INPUT, ///< The alignment has no graph path.
UNKNOWN_PATH, ///< The graph has no path with the target's name.
PATH_NOT_INDEXED, ///< The graph has the target path, but its positions are not indexed.
INCOMPATIBLE, ///< The target path does not carry the alignment's path.
SURJECTION_FAILED, ///< The surjector found no alignment on the target path.
};

Status status = Status::OK;

/// Name of the target path.
std::string path_name;
/// 0-based start of the alignment on the path's forward strand.
int64_t path_position = -1;
/// True if the alignment is on the path's reverse strand.
bool path_reverse = false;
/// Score of the surjected alignment.
int32_t score = 0;
/// MAPQ of the surjected alignment, which the surjector copies from the
/// graph alignment.
int32_t mapping_quality = 0;
/// CIGAR of the surjected alignment.
std::string cigar;

/// Bases in each kind of edit of the surjected alignment. The CIGAR writes
/// matches and mismatches both as M, so these are counted from the edits.
uint64_t matches = 0;
uint64_t mismatches = 0;
uint64_t inserted = 0; ///< Read bases missing from the path.
uint64_t deleted = 0; ///< Path bases missing from the read.
};

/// What GiraffeEngine::load() loads, and how the engine maps.
struct GiraffeEngineConfig {
/// The GBZ graph and Giraffe's minimizer, distance and zipcode indexes.
std::string gbz_path;
std::string minimizer_path;
std::string distance_path;
std::string zipcode_path;

/// Number of mapping threads.
size_t threads = 1;

/// Page the whole distance index into memory at load time.
bool preload_distance_index = true;

/// Names of the paths that map_reads() can surject onto. If empty,
/// map_reads() surjects nothing.
std::vector<std::string> surjection_target_paths;

/// Have the surjectors drop low-complexity anchors. vg giraffe turns this
/// on with --prune-low-cplx.
bool prune_suspicious_anchors = false;

/// Find the haplotypes that follow each alignment's path.
bool assign_haplotypes = true;
/// When assigning haplotypes, also find the boundary nodes of the
/// smallest snarl around each alignment's path.
bool find_snarl_boundaries = true;
};

/// A read to map, and the path to surject its alignments onto.
struct GiraffeServerRequest {
/// The read's name, sequence and base qualities. The qualities are raw
/// Phred scores. If they are empty, base quality does not cap MAPQ.
Alignment read;
/// One of the surjection target paths, or empty to surject nothing.
std::string surjection_target;
};

/// An alignment of a read, with the haplotypes that follow its path and its
/// surjection. A read can have several.
struct MappedAlignment {
Alignment alignment;
/// The runs of the alignment's path that haplotypes follow, in path
/// order. Unset if haplotype assignment is off.
std::optional<std::vector<HaplotypeSegment>> haplotype_segments;
/// Where the haplotypes that follow the whole path leave the snarl around
/// it. Both sides are unset if finding the boundaries is off.
SnarlBoundaries snarl_boundaries;
/// Where the alignment lands on the read's surjection target. Unset if
/// the read names no target.
std::optional<HaplotypeSurjectionResult> surjection;
};

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These still aren't great descriptions of what the classes are for, but I think that's because these aren't great classes. They're classes of convenience, with whatever fields some other thing seemed to need, rather than representing things in the domain model. I guess we can keep them, but we might end up refactoring this part eventually.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, I couldn't figure a better structure other than this for now.

Comment thread src/giraffe_engine.hpp
Comment on lines +115 to +121
/// The alignments of a read, best first. An unmapped read has one unmapped
/// alignment.
struct MappedRead {
std::vector<MappedAlignment> alignments;
/// Why the read could not be mapped. If set, there are no alignments.
std::optional<std::string> error;
};

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is OK the way it is.

But this really wants to be a Rust-style "result", which is either the data or an error.

We don't get std::expected until C++23, but we could use std::variant from C++17 to express that we want either alignments or an error.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, a std::variant is better. I left it as it is for now.

Comment thread src/giraffe_engine.cpp Outdated
Comment on lines +32 to +34
/// Count the matched, mismatched, inserted and deleted bases in an alignment's
/// edits.
void count_edits(const Alignment& aln, uint64_t& matches, uint64_t& mismatches,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's nothing like this in alignment.hpp? And this doesn't belong in alignment.hpp?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couldn't find the same functionality anywhere. The closest thing is what we do in the vg stats, but it is inline. We had a function called edit_count so I changed my function name to count_edit_bases to not be confusing.

Also the edit_count seems like a dead code. No one uses it and the code itself is wrong as it always returns 0.

for (int j = path.mapping_size(); j < path.mapping_size(); ++j) {

Do you want me to remove that from codebase?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm. If you found code that's never called and also broken, we probably do want to delete it.

Comment thread src/giraffe_engine.cpp Outdated
Comment on lines +58 to +68
/// Return a CIGAR as a string such as "55M", or "*" if it is empty.
string cigar_to_string(const vector<pair<int, char>>& cigar) {
if (cigar.empty()) {
return "*";
}
stringstream ss;
for (const auto& op : cigar) {
ss << op.first << op.second;
}
return ss.str();
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already have this, without the * fallback:

string cigar_string(const vector<pair<int, char> >& cigar);

It needs a doc comment though.

I found this with: semble search cigar_to_string.

We shouldn't add another copy of the logic. We might use a wrapper that returns * when the CIGAR is empty.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool, I liked the semble search. I use the cigar_string now and also added a comment for it. I check if it's empty where I use it as I only have that in 1 place.

@parsaeskandar
parsaeskandar merged commit 08ed905 into vgteam:master Nov 21, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

mapping Theme/team for mapping to pangenomes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Genome Browser Intergration

2 participants