Skip to content

[DO NOT MERGE] Reproduce flaky AnnCagraMultiPartitionTestF16_U32.Search/0 recall - #2838

Closed
bdice wants to merge 3 commits into
NVIDIA:mainfrom
bdice:flaky-cagra-mp-fp16-recall
Closed

bdice wants to merge 3 commits into
NVIDIA:mainfrom
bdice:flaky-cagra-mp-fp16-recall

Conversation

@bdice

@bdice bdice commented Oct 11, 2026

Copy link
Copy Markdown
Contributor

Reproducer for a flaky recall failure in NEIGHBORS_ANN_CAGRA_HALF_UINT32_TEST AnnCagraMultiPartitionTest/AnnCagraMultiPartitionTestF16_U32.Search/0 (recall 0.966 vs a minimum of 0.97), found by the stress run in #2793. Part of #2783 (workstream Q).

CI is reduced to the arm64 CUDA 13.3 build and the nightly gb300 and a100 C++ test configurations (two copies each). The test script repeats the single-partition multi-partition cases and logs the recall of every repetition, to get the recall distribution.

@bdice

bdice commented Oct 11, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 7c532cb

@copy-pr-bot

copy-pr-bot Bot commented Oct 11, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@bdice

bdice commented Oct 11, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test efaba68

1 similar comment
@bdice

bdice commented Oct 11, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test efaba68

@bdice

bdice commented Oct 11, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test bf29166

@bdice
bdice force-pushed the flaky-cagra-mp-fp16-recall branch from bf29166 to eaa8e39 Compare October 11, 2026 23:30
@bdice

bdice commented Oct 11, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test eaa8e39

@bdice

bdice commented Oct 12, 2026

Copy link
Copy Markdown
Contributor Author

Closing this reproducer. It showed the recall misses are the tail of a stable distribution at itopk_size=64 with num_partitions=1, not a library bug. The fix is in #2839 (52,000 recall evaluations, 0 failures).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant