[CI] Flaky test sweep (Sep–Oct 2026): root causes and fix PRs #2783

Activity

1. bdice

   added

   flaky

   on Oct 10, 2026
2. github-project-automation

   added this to Unstructured Data Processingon Oct 10, 2026
3. bdice commented on Oct 10, 2026

   

   bdice

   on Oct 10, 2026

   ContributorAuthor

   More actions

   Workstream B (H100 CAGRA hang): Opened draft #2784. It runs only the H100 conda Python test configs (PR 12.2.2 rockylinux8 oldest-deps and nightly 12.9.2 ubuntu24.04 latest-deps, 2 copies each) against nightly packages, loops test_cagra.py (alternating with the preceding test files) and cuvs_bench test_cli.py, and on a stall dumps py-spy --native, gdb all-thread backtraces, cuda-gdb kernel info, and nvidia-smi utilization.

   Log analysis so far: in 4 of 6 hung jobs, test_cagra.py stalls at test 35 or 36 of the file (test_cagra_dataset_dtype_host_device with ivf_pq, cosine, float16); the other 2 stall later, in test_cagra_build_from_dataset_handle. The bench hang is inside cagra::build (NN-descent) called from hnsw::build. pytest ignores the SIGINT from timeout, so it is stuck in native code.
4. bdice commented on Oct 10, 2026

   

   bdice

   on Oct 10, 2026

   ContributorAuthor

   More actions

   Workstream C (lucene missing results): Opened draft PR #2785 (CI reduced to CUDA 13 cpp build -> java/lucene build -> 4 lucene test jobs, amd64+arm64). Triage of the logs so far: all TestCuVSGaps.testVectorSearchWithFilterAndAlternatingDocuments / TestCuVSDeletedDocuments hits predate #2612 (exact fallback for selective filters, merged 2026-09-17); none since. The active one is TestMultiSegmentGPUFilterConcurrency.filteredSearchesTakeTheMultiPartitionGpuPath (24 vs 23, 5 hits since 09-26), where a filtered multi-partition CAGRA search returns < k hits and the query falls back to the per-segment path. Suspect: cagra::detail::search_multi_partition never derives filtering_rate from the partition bitsets, so MULTI_CTA itopk is not enlarged for selective filters the way single-partition search does it. CI run measures the shortfall rate vs itopk/algo and replays the failing seeds.
5. bdice commented on Oct 10, 2026

   

   bdice

   on Oct 10, 2026

   ContributorAuthor

   More actions

   Workstream D (lucene index not found): Draft PR #2786 is open. CI is reduced to the CUDA 12 cpp/java/lucene jobs, with 3 copies of conda-lucene-tests on amd64 and arm64. Each copy runs the full suite, then repeats TestCuVSGaps, TestCuVSRandomizedVectorSearch, TestCuVSVectorsFormat and TestCuVSDeletedDocuments for 40 minutes.

   Root cause (reproduced locally, every run): CuVS2510GPUVectorsReader skips loading the GPU indexes when it is opened with IOContext.Context.MERGE, so cuvsIndices stays null. Lucene's ReadersAndUpdates.getReaderForMerge(context) caches the SegmentReader it opens for a merge in the writer's reader pool. If an NRT reader (writer.getReader() / DirectoryReader.open(writer)) is opened while that merge is still running, getReadOnlyClone returns that pooled reader, or a clone that shares its core. Searching it then fails:

   • GPUKnnFloatVectorQuery.rewrite sees getCagraIndexForField == null and falls back to the per-leaf path.
   • That path calls CuVS2510GPUVectorsReader.search, which throws IllegalStateException: Index not found for field:....

   This is a race between NRT reopen and a background merge, which is why the failure is intermittent and can't be reproduced by seed. It is also a real production bug, not only a test problem.

   New test TestSearchSegmentsOpenedForMerge reproduces it every time: it blocks a forced merge in OneMerge.wrapForMerge, after the merge readers are open, then opens an NRT reader and searches. It fails with the same stack as CI (CuVS2510GPUVectorsReader.search:467 <- GPUKnnFloatVectorQuery.approximateSearch).

   Fix (local, passing): a reader opened for a merge loads its GPU indexes on first search (thread-safe lazy load) instead of never loading them. The memory saving for merge-only readers is unchanged.

   testRandomExceptions UNEXPECTED: exception type: the 3 hits are all from before #2744 (merged 2026-10-07), which unwraps the InvocationTargetException in LuceneProvider so the injected IOException surfaces correctly. I found no hits after #2744, so I'm treating it as already fixed.

   The cause is not shared with workstream C. It only produces exceptions, not missing hits.
6. bdice commented on Oct 10, 2026

   

   bdice

   on Oct 10, 2026

   ContributorAuthor

   More actions

   Workstream A (spectral clustering): Draft PR #2788 opened. CI is reduced to 4 copies of the arm64 / CUDA 12.2.2 / A100 C++ test job. Each copy runs CLUSTER_TEST 5 times, SpectralClustering* 300 times in one process, and SpectralClusteringTestF.Result/7 200 times in fresh processes, plus a diagnostic test that checks which stage is nondeterministic. All 17 historical failures are Result/7 (ARI 0.58-0.64). Locally on an RTX 6000 Ada the kNN connectivity graph already differs between runs on identical input (3-4 variants, presumably from distance ties in kNN selection), but Lanczos always finds the 5-dim null space and ARI stays at 1.0.
7. bdice commented on Oct 10, 2026

   

   bdice

   on Oct 10, 2026

   ContributorAuthor

   More actions

   Workstream E (kmeans batched fit): draft PR #2787 (CI reduced to CLUSTER_TEST --gtest_filter=KmeansFitBatchedTests/* --gtest_repeat=100, 4x H100 / CUDA 13.3.0; first run: https://github.com/NVIDIA/cuvs/actions/runs/38063498592).

   Notes so far:

   • Every KmeansFitBatchedTestF hit in the window is from release/26.10 nightlies or from main before Use new top_1_nn primitive in KMeans #2249 (2026-09-23). Use new top_1_nn primitive in KMeans #2249 removed ASSERT_TRUE(centroids_match) from the float test on main, so main hides the symptom but not the cause. The 2026-10-06 nightly failure was Result/3 (1.959 != 2.001 @13,2), not Result/4.
   • Reproduced locally (RTX 6000 Ada, nightly libcuvs 24b0b170): 30 identical in-memory kmeans::fit calls (same data, init=Array) give 30 different centroid sets, even with max_iter=1. With the float centroid check restored, KmeansFitBatchedTestF.Result/5 fails 5/20 locally.
   • Root cause: the centroid update accumulates with atomics. raft::linalg::reduce_rows_by_key (per-cluster row sums) and reduce_cols_by_key (per-cluster weights) do this, so the float summation order changes from run to run. That noise either flips the iteration at which the convergence check stops ([BUG] kmeans::fit is not run-to-run reproducible; KmeansFitBatchedTestF.Result/4 flakes ~50% on GB10 and should not gate CI #2657) or, when cuTile's TF32 assignment is used, moves borderline points. mapThenSumReduce (centroid shift, weight sum) also sums with atomics.
   • Candidate fix (in the PR next): deterministic per-cluster sums (stable radix sort by label, fixed-size pieces, fixed-order combine), plus cub::DeviceReduce for the shift and weight sum. Locally all fits are bitwise identical (1 distinct result out of 30, float and double, device and host-batched input), and the restored assertions pass 20/20. Kernel cost is within about 15% of the atomic version for large d, faster for small d with heavy contention, plus ~20 us fixed overhead per iteration on tiny inputs.
8. bdice commented on Oct 10, 2026

   

   bdice

   on Oct 10, 2026

   ContributorAuthor

   More actions

   Workstream A (spectral clustering): Root cause found and reproduced locally. CI confirmation on A100 is still running.

   The embedding is fine. The k-means assignment step is wrong. For a graph with 5 components, the normalized-Laplacian embedding puts each cluster at a single point, and all 5 points have the same squared norm (1/vol = 1/3800 ≈ 2.6e-4). The squared distance between them is 5.3e-4.

   The L2-expanded epilogue in cpp/src/distance/detail/distance_ops/l2_exp.cuh zeroes any distance with d^2 < 1e-6 when ||x||^2 == ||y||^2. That clamp is meant to remove self-distance round-off, but it uses an absolute tolerance. Here d^2 = 2.8e-7, and many rows' norms are bitwise equal to another centroid's norm, so wrong centroids get distance 0 and win the argmin. k-means then returns ARI ≈ 0.6. Every mismatched row I checked has a norm bitwise equal to a wrong centroid's norm.

   Only the CUTLASS/SIMT fused 1-NN path has this clamp. The unfused path does not. With CUDA 12, the CUTLASS path is chosen for compute capability 8.x, so A100 is the only GPU in the PR matrix that hits it. The double-precision test passes because its clamp threshold is 1e-15.

   Run-to-run flakiness comes from small rounding differences that decide whether norms collide. The kNN graph is not deterministic: on identical input I see 3-4 graph variants, presumably from distance ties.

   Smallest reproducer: pairwise_distance(L2Expanded) on 0.01 * I returns 0 for every off-diagonal entry, where 2e-4 is expected. Fix in progress: make the clamp tolerance relative to the norms when the norms are below 1.
9. bdice commented on Oct 10, 2026

   

   bdice

   on Oct 10, 2026

   ContributorAuthor

   More actions

   Workstream E (kmeans batched fit): reproduced in CI. Baseline run on main + diagnostics, 4x H100 / CUDA 13.3.0, 100 repetitions each: https://github.com/NVIDIA/cuvs/actions/runs/38063498592

   • New KmeansFitBatchedTest{F,D}.Reproducible/* test (3 identical fits from the same initial centroids, device and host input, compared bitwise): fails 400/400 for all 12 configs. Inertia and centroids differ on every run, both float and double. The run-to-run nondeterminism is present on H100.
   • With ASSERT_TRUE(centroids_match) restored for float, Result/* passed 400/400 on H100. On current main, H100 converges Result/4 in 4 iterations (11 or 14 before Use new top_1_nn primitive in KMeans #2249), so the noise does not currently change the stopping iteration there. It still does on other GPUs: locally on Ada, Result/5 mismatched 5/20.
     Fix pushed to [DO NOT MERGE] Reproduce flaky KmeansFitBatched test #2787 (deterministic per-cluster sums); CI is running.
10. bdice commented on Oct 10, 2026

    

    bdice

    on Oct 10, 2026

    ContributorAuthor

    More actions

    Workstream D (lucene index not found): For workstream C: I found a seed that reproduces C's missing-results failure locally every time, with the D fix applied as well. mvn test -Dtest=TestCuVSGaps -Dtests.seed=65F332205DAF280D fails testVectorSearchWithAlternatingDocuments with Should return exactly 56 results expected:<56> but was:<54> in 3 of 3 runs. Setup: RTX 6000 Ada, nightly libcuvs 26.12.00a108 (24b0b17, cuda12), and the cuvs-java jar from build run 38028687031. It is unrelated to the 'Index not found' bug.

    Local loop of the 4 affected classes before the D fix: 2/64 iterations failed. One was Index not found (TestCuVSVectorsFormat.testRandomWithUpdatesAndGraph, seed 971AE5B7CFADB729; timing-dependent, so the seed does not reproduce it). The other was the C failure above.
11. bdice commented on Oct 10, 2026

    

    bdice

    on Oct 10, 2026

    ContributorAuthor

    More actions

    Workstream C (lucene missing results): Reproducer confirmed in CI on #2785 (run https://github.com/NVIDIA/cuvs/actions/runs/38062855918, amd64 jobs 114249302243, 114249302320):

    • Replaying the CI seeds of TestMultiSegmentGPUFilterConcurrency.filteredSearchesTakeTheMultiPartitionGpuPath fails 9/36 runs with the same expected:<24> but was:<23> (random seeds: 1/80). It's not deterministic per seed, because the CAGRA graph build isn't deterministic.
    • Diagnostic test, same index shape (4 segments x 500 docs, filter keeps ~4% of each), 18k filtered queries: multi-partition search returns < k hits in 0.04-0.05% of queries at itopk 10/32 (AUTO -> MULTI_CTA), and 0 at itopk >= 64. SINGLE_CTA: 50%.
    • Root cause: cagra::search multi-partition overload (cpp/src/neighbors/cagra.cuh) never derives filtering_rate from the partition bitsets. Single-partition search counts the bitset and MULTI_CTA's adjust_search_params then enlarges itopk_size (here 32 -> ~352). Multi-partition leaves it at -1, so a selective filter runs with itopk 32, sometimes finds < k valid neighbors, and the Lucene query falls back to per-segment search. Fix: compute the rate per partition (max over partitions that accept >= 1 row). Pushing next with a C++ test.
    • Separately, TestCuVSGaps.testVectorSearchWithAlternatingDocuments 59 vs 53 (seed BBE2A1CCC7772A7F) fails deterministically: it's a test bug. With that seed the index has only 53 docs with vectors, and topK=59. Fix flaky accelerated HNSW gaps test #2709 fixed the same issue in TestCuVSAcceleratedHNSWGaps. The older TestCuVSGaps...FilterAndAlternatingDocuments / TestCuVSDeletedDocuments hits all predate Lucene: add exact search fallback for very selective filters #2612 (2026-09-17) and haven't recurred. No overlap with workstream D's Index not found.
12. bdice commented on Oct 10, 2026

    

    bdice

    on Oct 10, 2026

    ContributorAuthor

    More actions

    Workstream A (spectral clustering): Reproducer confirmed in CI (run 38063875977, 4 copies of the arm64 / 12.2.2 / A100 job, unmodified library):

    • Full CLUSTER_TEST: 6/20 runs failed.
    • SpectralClusteringTestF.Result/7 in fresh processes: 126/800 failed (4, 12, 95, 15 per copy).
    • SpectralClustering* with --gtest_repeat=300: 35 Result/7 failures out of 1200.
    • Diagnostic test: 118/1200 repeats had ARI < 0.7. In every failing repeat, Lanczos found all 5 null vectors (eigenvalues ~1e-7) and the embedding was exact (each cluster collapses to a point, max deviation ~1e-8). The error is only in the k-means assignment, which matches the norm-clamp root cause above.

    Fix (relative self-distance clamp in l2_exp.cuh, plus regression tests for pairwise_distance and top_1_nn) is pushed to #2788 and CI is running.
13. bdice commented on Oct 10, 2026

    

    bdice

    on Oct 10, 2026

    ContributorAuthor

    More actions

    Workstream D (lucene index not found): CI reproducer confirmed without the fix (run 38063064947, 6 lucene test jobs: 3 on amd64 rtxpro6000, 3 on arm64 l4):

    • The new deterministic TestSearchSegmentsOpenedForMerge failed in 6/6 jobs with Index not found for field:vector.
    • The repeat loop hit Index not found in 17 of 853 iterations (about 2%), spread across all four affected classes and both arches: TestCuVSVectorsFormat.testRandomWithUpdatesAndGraph, TestCuVSRandomizedVectorSearch.testVectorSearch and testVectorSearchWithFilter, and TestCuVSGaps.testVectorSearchWithAlternatingDocuments.
    • The loop also hit 3 of workstream C's Should return exactly N results failures (TestCuVSGaps).

    Pushed the fix (37c16e4) to #2786. Locally, the same loop went from 2/64 iterations failing before the fix to 0/106 after, and the full cuvs-lucene suite passes (375 tests).
14. bdice commented on Oct 10, 2026

    

    bdice

    on Oct 10, 2026

    ContributorAuthor

    More actions

    Workstream I (silhouetteScore): Opened draft #2789.

    Triage: both known hits are silhouetteScore.BatchedStreamPoolOrdering on arm64. One is PR job 86496273362 (2026-07-11, A100, CUDA 12.2.2), and the other is nightly job 91625948741 (2026-08-03, GB300, CUDA 13.3.0). The expected value is the same in both (-4.86e-4), but actual varies (3.25e-3 and 8.4e-4).

    Both hits predate #2422 (merged 2026-08-17). Before that change, the batched path wrote the row-wise min of b (n_rows x n_labels) in place to b[0..n_rows). With n_labels=2, RAFT's thin coalesced reduction (512 rows per block) has block k overwrite inputs that block k/2 may not have read yet. That is an inter-block race, so it only shows up under GPU contention (ctest -j8). Working hypothesis: #2422 already fixed #2406.

    #2789 checks this on 4x arm64 A100. Each job loops silhouetteScore.Batched* in 8 concurrent processes, both with a temporary env switch that restores the in-place reduction (negative control) and with current main. I'm also scanning the failed conda-cpp-tests jobs since 2026-08-01 for later hits.
15. bdice commented on Oct 10, 2026

    

    bdice

    on Oct 10, 2026

    ContributorAuthor

    More actions

    Workstream H (Python IVF result flakes): Opened draft #2790. CI is reduced to the nightly-matrix A100 conda Python test configs (amd64 12.9.2, arm64 13.0.3, arm64 13.3.0) plus rtxpro6000 13.3.0, 2 copies each. It runs against nightly packages with this branch's test files, runs tie/recall diagnostics, and repeats the three tests 200 times each with pytest-repeat.

    Local findings (RTX 6000 Ada, nightly cuvs 26.12.00a108):

    • test_build_precomputed: I reproduced the swap locally by using 10000 queries instead of 100 (2 of 10 runs swapped one pair). The regular and precomputed indexes have identical labels and PQ codes, and their distances are bitwise identical. Only the in-list storage order differs: about 2% of rows are at the same position, because process_and_fill_codes assigns slots with atomicAdd. Every swap I saw is two neighbors from the same list whose float32 distances are bitwise equal (for example 0x1.6efa14p+1), even though their codes differ. The tie is resolved by storage position, so the test's exact index comparison is wrong when distances tie. Add a stable flag to the IVF-PQ and IVF-SQ search #2696's stable flag breaks ties on the in-list candidate position, not the dataset id, so it would not make these two indexes agree.
    • test_ivf_pq[inner_product]: generate_data uses the unseeded global numpy RNG. Over 1000 seeds, recall has mean 0.741, std 0.011 and min 0.703, versus about 0.91 for L2. On the same normalized problem with identical ground truth, IVF-PQ gets 0.92 recall with sqeuclidean and 0.74 with inner_product (pq_dim auto = 8 for dim 10). The IP score error is q·(r - r̂), while the L2 error scales with |q - x|, so the gap is expected. With the data seed fixed, recall still varies by about ±0.005 between runs because of build nondeterminism.
    • test_filtered_ivf_flat[0.5]: Fix test params for filtered ivf_flat #1463 retuned n_lists and n_probes after [BUG] Intermittent test failure: test_ivf_flat.py::test_filtered_ivf_flat[0.5] #1445 was filed. Mean recall is about 0.96 and there have been no hits in the window. I'll confirm this in CI.
16. 78 remaining items

    Load more

17. bdice commented on Oct 11, 2026

    

    bdice

    on Oct 11, 2026

    ContributorAuthor

    More actions

    Workstream S (stress run) summary #4 (2026-10-11 18:05 UTC), PR #2793

    Current head 23d182e3, building now. I rebuilt the branch from 24b0b170 with:

    • E as Make k-means fit deterministic #2801 (7dcc920a) plus the narrowed Make balanced k-means donor selection and IVF-PQ codebook grouping independent of scheduling #2804 (fe6517c6)
    • J as Re-enable CagraRandomizedIT with a recall-based check #2830 (9af2f6ba), which also includes the single-row fix
    • B at 696e4612 (refactor)
    • P (0a27e5fa)
    • A, C, D, F, G, H, K, L ×2, N and O unchanged

    Since summary #3: attempts 4-7 of 38117245263, the full-fix build, were all clean, making 7/7 clean passes there (847/847 test jobs). The rebuild run 38149049954 failed deterministically in 19 jobs because of the first narrowed #2804 test (reported above, fixed in fe6517c6). Its other 102 jobs passed.

    Cumulative: 16 passes, 1936 test-job executions. Excluding those 19 deterministic failures, 3 failed, all reported earlier (O, P, ScaNN #2549); there have been no new flakes since 03:00 UTC. Per flake (failed / executions with the fix):

    • A spectral: 0/304 CLUSTER_TEST jobs, 0/16 on arm64 12.2.2 A100
    • B H100 hang: 0/24 H100 conda Python jobs with the fix (also 0/24 before it was included)
    • C / D lucene: 0/64 lucene jobs each
    • E kmeans: 0/304 CLUSTER_TEST, 0/48 on H100
    • F AnnMG: 0/285
    • G RaBitQ: 0/285
    • H Python IVF: 0/528 Python jobs
    • I silhouette: 0/304 STATS_TEST jobs
    • J java: 0/64
    • O lucene randomized HNSW: 0/40 lucene jobs since the fix (1 hit before it)
    • P: 4 lucene jobs so far
    • ScaNN [CI] SCANN Test flaky #2549 (not in scope here): 1 hit in ~300 shard-3 jobs, on GB300.

    Plan until 02:00 UTC: one full run on 23d182e3, then test-only re-runs, merging any further FIX READY branches.
18. bdice commented on Oct 11, 2026

    

    bdice

    on Oct 11, 2026

    ContributorAuthor

    More actions

    Workstream E2 (k-means perf evidence): benchmarks are done, and #2801 and #2804 now have Performance/Accuracy sections in their descriptions. Method: main, #2801 and narrowed #2804 built identically in one run, interleaved, 10 samples per config, on H100, A100, GH200, GB300, RTX PRO 6000 and L4 (#2803, https://github.com/NVIDIA/cuvs/actions/runs/38154672010).

    • Make k-means fit deterministic #2801: fits are bitwise reproducible in every config (main differs run to run in 52 of 53 configs on A100). Per-iteration time is up to 30% faster for small k and large n. There is a regression at n≈100K with k≥100 or d=1024: +5–11% per iteration (0.05–0.1 ms). Nsight Systems attributes it to the fixed cost of the radix sort by label (124 → 195 µs per centroid update on A100). Centers are closer to a float64 fit (up to 10–100x). It does not touch the IVF build paths (IVF builds within ±2%).
    • Make balanced k-means donor selection and IVF-PQ codebook grouping independent of scheduling #2804 (narrowed, fe6517c6): IVF-Flat/PQ/SQ/RaBitQ/CAGRA builds are within -2.2%..+1.5% on H100 and -3.2%..+6.3% on A100 (only the 1M SQ/RaBitQ builds are above +3%, with overlapping ranges). Recall is unchanged. In a 64-fit microbenchmark of tiny balanced fits, H100 shows +12–16% in 2 of 5 configs; nsys shows more EM iterations (a different donor trajectory), not slower kernels.
19. bdice commented on Oct 11, 2026

    

    bdice

    on Oct 11, 2026

    ContributorAuthor

    More actions

    NEW FLAKE: C++ NEIGHBORS_ANN_CAGRA_HALF_UINT32_TEST AnnCagraMultiPartitionTest/AnnCagraMultiPartitionTestF16_U32.Search/0, with {n_queries=100, dataset shape=10000x64, k=10, num_partitions=1, split=even, single-cta, itopk_size=64, metric=L2}:

    Recall = 0.966000 (966/1000), the error is 13.3% above the threshold (eps = 0.003000).
    ann_cagra.cuh:2294: Failure
    Value of: eval_neighbours(indices_naive, indices_mp, ...)
      Actual: false (actual recall (0.966) is lower than the minimum expected recall (0.97); eps = 0.003. )
    

    • Config: conda-cpp-tests (3, 4) / 13.3.0, 3.14, arm64, ubuntu26.04, gb300, latest-driver, latest-deps
    • Job: https://github.com/NVIDIA/cuvs/actions/runs/38160969192/job/114568250661 (stress PR [DO NOT MERGE] Stress test CI with combined flaky-test fixes #2793, head 23d182e3, attempt 4, a test-only re-run; the same build passed this job in attempts 1-3)
    • This is the unfiltered multi-partition path with a single partition, fp16, SINGLE_CTA, so it misses the 0.97 bar by 4 of 1000 neighbors. Workstream C's fix (fe569825, which derives filtering_rate in multi-partition search) is in this build, but it only affects filtered searches, so I don't think it's involved. I couldn't find any earlier hit in the 5-week CI history logs or anywhere else in the stress run: the test has passed in about 10 other GB300 shard-3 executions and about 290 executions in other configs.
20. bdice commented on Oct 11, 2026

    

    bdice

    on Oct 11, 2026

    ContributorAuthor

    More actions

    Workstream Q (CAGRA multi-partition fp16 recall): draft reproducer #2838.

    • Step 0: no other hit of any AnnCagraMultiPartitionTest* case in nightly test.yaml failures (2026-08-10 to 2026-10-10, 179 failed test jobs) or in the mined pr.yaml failed-then-rerun attempts. The only known hit is S's job.
    • In that same log, the sibling cases show where L2 sits: Search/0 (single-cta, L2) 0.966, Search/3 (multi-cta, L2) 0.976, while InnerProduct and Cosine are at 0.991 to 0.993. The test comment says that floors of about 0.974 were observed when 0.97 was chosen. So L2 runs close to the threshold.
    • The dataset seed is fixed (1234), and CAGRA search is deterministic for a given graph, so run-to-run variation has to come from the NN-descent graph build. For fp16, GenerateRoundingErrorFreeDataset gives only 8 distinct values per dimension (resolution 4 for dim=64), so the data is coarse and uniform.
    • [DO NOT MERGE] Reproduce flaky AnnCagraMultiPartitionTestF16_U32.Search/0 recall #2838 builds arm64 CUDA 13.3 and runs gb300 and a100 x2. Each job repeats Search/0-5 (num_partitions=1) 250 times, runs a dataset-seed sweep, and runs an f32 comparison. Every repetition logs the multi-partition recall, the recall of a regular single-index cagra::search on the same index, the graph hash, and per-query misses, which together give the recall distribution.
21. bdice commented on Oct 11, 2026

    

    bdice

    on Oct 11, 2026

    ContributorAuthor

    More actions

    Workstream Q (CAGRA multi-partition fp16 recall): I have the recall distribution. It's a tail event on a stable, unimodal distribution, not a bug. Run: 38177641942, with gb300 x2 and arm64 a100 x2. The jobs show as "failed" only because of the diagnostic script's exit code; no test case failed.

    F16 Search/0 (num_partitions=1, single-cta, L2, itopk=64), seed 1234:

    GPU	n	mean	sd	min	p1	p5
    GB300	1100	0.9786	0.0032	0.967	0.970	0.974
    A100	1100	0.9845	0.0028	0.972	0.977	0.980

    On GB300 the histogram is a single peak at about 0.979, with a left tail reaching 0.967 (the test fails at <= 0.966). The other L2 cases look the same: Search/3 (multi-cta, L2) has a GB300 mean of 0.981 and a min of 0.969. InnerProduct and Cosine sit at 0.993 to 0.996. f32 Search/0 on GB300 has a mean of 0.9886 and a min of 0.981.

    Hypotheses ruled out:

    • Multi-partition path (num_partitions=1) vs regular search: a regular single-index cagra::search on the same index gave exactly the same recall in 800/800 runs. The index differences are only the order of tied distances; the coarse fp16 data has many exact ties. The single-cta search is also deterministic: repeating it on the same index gave identical results 800/800 times.
    • Fix NN-Descent early stopping #2708 (NN-descent early stopping): not involved. For this shape, NN-descent never converges and always runs the full max_iterations=20, with about 115k updates still happening in the last iteration. The old criterion, the new criterion and termination_threshold=0 produce the same distribution (mean 0.9786 / 0.9787 / 0.9786 on GB300).
    • Variation source: every repetition builds a different graph (the NN-descent build is nondeterministic). The misses are spread over 16 to 25 queries per run, so this is not a disconnected region.

    Why it shows up on GB300: the "fixed" dataset is not the same on every GPU. raft::random launches its RNG kernels with 4 * multiProcessorCount blocks, so seed 1234 produces a different dataset on GB300 than on A100. The GB300 dataset happens to be harder (mean 0.9786 vs 0.9845). When I sweep the seed (600 runs per GPU), both GPUs average 0.984 with sd 0.004 and a min of 0.969.

    So the 0.97 threshold is about 3.5 sd below the GB300 mean. The test comment says 0.97 was picked from observed floors of about 0.974, which is too little headroom for a per-build sd of 0.003 to 0.004 plus a dataset that changes with the GPU's SM count. Next run: the same cases including FilteredSearch (which uses the same 0.97), the full multi-partition suite for all four dtypes, and itopk_size 64 vs 128. The goal is to fix the test by giving these cases enough search budget rather than lowering the bar.
22. bdice commented on Oct 11, 2026

    

    bdice

    on Oct 11, 2026

    ContributorAuthor

    More actions

    Workstream Q (CAGRA multi-partition fp16 recall): I have the root cause and a candidate fix, which is being verified.

    Root cause: the test is under-provisioned, not the library. In the core sweep the num_partitions=1 cases search one 10000-row index with itopk_size=64. For the coarse fp16, int8 and uint8 datasets, recall at that budget sits around 0.98, and because the NN-descent graph differs from build to build it has a tail that reaches the 0.97 threshold. With num_partitions=8, each 1250-row partition contributes its own 64 candidates and recall is about 1.0. The 0.97 floor was a workaround for this gap.

    Measured in run 38180387035: gb300 and arm64 a100, 2 jobs each, about 11k test repetitions per job. Each row is one case, with the min/mean over all repetitions:

    num_partitions=1 case (fp16)	itopk=64 min / mean	itopk=128 min / mean
    Search/0 L2 single-cta, seed 1234, GB300	0.968 / 0.9785	0.997 / 0.9991
    Search/0, 200 dataset seeds per GPU	0.971 / 0.984	0.994 / 0.999
    FilteredSearch/0 L2, seed 1234, GB300	0.970 / 0.9799	0.995 / 0.9985
    FilteredSearch/0, 200 dataset seeds per GPU	0.966 (fails) / 0.982	0.995 / 0.999
    int8 / uint8 Search/0, FilteredSearch/0	0.977 to 0.981 / ~0.985	0.995 to 0.998 / ~0.998

    The seed sweep also reproduced a real failure: FilteredSearch/0 scored 0.966 on GB300. With itopk=128, the whole multi-partition suite for all four dtypes passed, and every case's minimum is at or above 0.994.

    Fix (test-only): bdice:fix-cagra-mp-recall-itopk @ 587b72cc. It sets itopk_size=128 in the core multi-partition sweep and raises min_recall from 0.97 to the single-index 0.985 convention. The test then tolerates build-to-build variation and is stricter against real merge/decode regressions. The k-spanning, dimensionality and max_queries-chunk blocks are unchanged, since their minimums are 0.985 or higher at their current budgets. Verification is running in #2838: the fix plus every multi-partition case for all four dtypes x40, plus num_partitions=1 seed sweeps, on gb300, a100, h100 and rtxpro6000 (CUDA 13.3). I'll post FIX READY when it passes.
23. bdice commented on Oct 12, 2026

    

    bdice

    on Oct 12, 2026

    ContributorAuthor

    More actions

    FIX READY: Workstream Q (CAGRA multi-partition fp16 recall). Branch bdice:fix-cagra-mp-recall-itopk, head 587b72cc51f59d9144c8aadccd06818e064d839c, draft PR #2839. This is a test-only change: the core AnnCagraMultiPartitionTest sweep uses itopk_size=128 instead of 64, and min_recall goes from 0.97 to 0.985. With itopk=64, the num_partitions=1 cases with fp16/int8/uint8 data have recall around 0.98 and a tail below 0.97.

    Verified in run 38184518531 (#2838 @ eaa8e39a, on GB300, arm64 A100, H100 and RTX PRO 6000, CUDA 13.3). The run repeated every multi-partition case for all four dtypes 40 times, plus 150 repetitions of the num_partitions=1 cases on a different dataset each time. Results: 52,000 recall evaluations, 0 failures. The num_partitions=1 minimum was 0.993 against the new 0.985 floor; before the change the minimums were 0.966 to 0.968 against 0.97. The num_partitions=8 minimum was 0.999. The unchanged k-spanning, dim and chunk blocks had a minimum of 0.990.
24. bdice commented on Oct 12, 2026

    

    bdice

    on Oct 12, 2026

    ContributorAuthor

    More actions

    Workstream S (stress run) final summary (2026-10-12 01:50 UTC), PR #2793 (left as a draft)

    Totals: 25 passes of the union of the PR and nightly test matrices (121 test jobs per pass: 19 C++ configs × 4 shards, 17 conda Python, 16 wheel, 4 java, 4 lucene, 4 standalone C), for 3025 test-job executions. 3001 passed and 24 failed. 19 of those failures were the one deterministic test bug in the first narrowed #2804 head, fixed in fe6517c6. There were no infrastructure failures. The passes were 7 full runs and 18 test-only re-runs through the stress-rerun-anchor job.

    Final fix set (head 23d182e3, run 38160969192, 9 passes, 1087/1089 test jobs passed):

    • A 390f97b0
    • B 696e4612
    • C fe569825
    • D 87ca4398
    • E Make k-means fit deterministic #2801 7dcc920a and Make balanced k-means donor selection and IVF-PQ codebook grouping independent of scheduling #2804 fe6517c6
    • F 733caab2
    • G f7c813d3
    • H ccb70f13
    • J Re-enable CagraRandomizedIT with a recall-based check #2830 9af2f6ba
    • K c799bd98
    • L 5af73320 and 18855627 (cuTile re-enabled)
    • N ae0f9c2f
    • O 37f4477e
    • P 0a27e5fa

    I's fix (#2422) is on main. Q (587b72cc, posted 00:43) is not included, because a rebuild could not finish before the deadline.

    Per-flake results with the fix included (failed / executions):

    Flake	Result
    A spectral	0/475 CLUSTER_TEST jobs, 0/25 on arm64 12.2.2 A100. Before the fix, the A100 reproducer failed 6/20 full CLUSTER_TEST runs; ~17 CI hits in 5 weeks.
    B H100 hang	0/51 H100 conda Python jobs, plus 0/24 before the fix was included (it was rare per job)
    C / D lucene	0/100 lucene jobs each
    E kmeans	0/475 CLUSTER_TEST jobs, 0/75 on H100
    F AnnMG	0/456 NEIGHBORS_MG_TEST jobs
    G / K RaBitQ	0/456 NEIGHBORS_ANN_IVF_RABITQ_TEST jobs
    H Python IVF	0/825 Python jobs
    I silhouette	0/475 STATS_TEST jobs
    J java	0/100 java jobs
    O lucene randomized HNSW	0/76 lucene jobs with the fix; 1 hit before it
    P lucene BQ multi-layer	0/40 lucene jobs with the fix; 1 hit before it

    New flakes found by the stress run (all now have workstreams):

    1. O: lucene TestCuVSRandomizedHNSWVectorSearch.testVectorSearch (comment), fixed by Use a recall-based check in TestCuVSRandomizedHNSWVectorSearch #2824.
    2. P: lucene TestAcceleratedHNSWMultiLayerRoundTrip BQ entry-node self-match (comment), fixed by Query the entry node's own vector in TestAcceleratedHNSWMultiLayerRoundTrip #2831.
    3. Q: AnnCagraMultiPartitionTestF16_U32.Search/0 recall 0.966 < 0.97 on GB300 (comment). Fix Use itopk_size=128 in the core CAGRA multi-partition test sweep #2839 was not stress-tested.

    Remaining failures:

    • ScaNN [CI] SCANN Test flaky #2549 (already cataloged): 2/475 shard-3 jobs, both GB300 (2/25 there), both ScaNN/f32_i64.build/12 with max reconstruction error 1.4301 vs 1.425 (job, job). The catalog only listed rtxpro6000 hits. Fix reconstruction test for ScANN #2743 is the open fix.
    • Q, as above.

    Integration notes:

    • E (Make k-means fit deterministic #2801 / Make balanced k-means donor selection and IVF-PQ codebook grouping independent of scheduling #2804) and N touch the same test_float_int64_t.cu includes only in the older E head; the current heads merge cleanly.
    • L's first head failed cmake-lint, which was fixed in 5af73320.
    • The first narrowed Make balanced k-means donor selection and IVF-PQ codebook grouping independent of scheduling #2804 head had a test that segfaulted, which was fixed in fe6517c6.
    • No deterministic failure came from combining the fixes.
25. bdice commented on Oct 12, 2026

    

    bdice

    on Oct 12, 2026

    ContributorAuthor

    More actions

    Workstream R (C++ test 6-hour timeouts): already fixed on main by #2729 (plus #2746 as a backstop). No new PR.

    I scanned every conda-cpp-tests job in pr.yaml and nightly test.yaml runs since 2026-09-01, all attempts included, and found 89 jobs that ran ≥ 5.5 h. For each one I took the last Start N: line from the log.

    What was running when the job was killed	Jobs	Class
    NEIGHBORS_DYNAMIC_BATCHING_TEST (ctest #28, shard 4/4)	86 (76 PR jobs on 15 PRs + 10 in the 2026-10-06 nightly)	real hang, already fixed
    Artifact download of cuvs_conda_cpp_libcuvs_* (ctest never started)	2 (#2540, V100 and arm64 A100)	infrastructure
    Shard 3 V100, slow artifact download (5h20m), then the tests passed in 37 min	1 (#2739, success at 5.97 h)	infrastructure

    The dynamic-batching hang. Every GPU and CUDA combination was hit: V100, L4 (amd64 and arm64), A100 (arm64), H100 (including ubuntu26.04 / CUDA 13.3), and GH200. Every shard-4 job of the 2026-10-06 nightly was stuck in this test for about 5h40m. Sample jobs:

    • V100 Rocky: PR #2702
    • H100 12.9.2: PR #2702
    • L4 arm64 13.0.3: PR #2702
    • Nightly H100: job, which ran Start 28: NEIGHBORS_DYNAMIC_BATCHING_TEST at 06:58:14 and was cancelled at 12:38:37.

    Timeline on main:

    • 10-02, Add gtest peak memory reporting and use async memory resource in C++ tests #2722: async memory resource in the C++ tests. Two V100 hangs in this test that day (#2725, #2716).
    • 10-05 17:05Z, Build the upstream index once per dynamic batching test #2728: the upstream index is built once per test. From then on the hang was close to deterministic. The first hit was in a run created at 18:10Z.
    • 10-06 20:21Z, Fix dynamic batching hangs and CAGRA multi-kernel search on NaN queries #2729: "Fix dynamic batching hangs and CAGRA multi-kernel search on NaN queries". The bug chain: the batch query buffer was never initialized, so with the async memory resource it could hold NaN. A NaN query made the CAGRA MULTI_KERNEL random-pickup read an uninitialized seed index, and the read went out of bounds. The upstream search then threw inside the dispatching thread. That thread never published the dispatch or returned the IO buffer, so every other request waited forever. The test also never called future.get(). The last hung job's run was created at 18:53Z on 10-06, before Fix dynamic batching hangs and CAGRA multi-kernel search on NaN queries #2729 merged.
    • 10-07 04:03Z, Add 100 minute timeout to C++ ctest runs #2746: adds timeout ... 100m to the ctest invocation in ci/test_cpp.sh. It explains why none of these logs show timeout: sending signal: the wrapper did not exist yet. It was not a timeout that failed to fire.

    After #2729: no C++ test job has run ≥ 5.5 h (all runs through 2026-10-12 were scanned). The test passes in about 68 s (10-12 nightly, L4), and the stress run had 0 hits in 3,025 test jobs.

    Relation to other workstreams:

    • B (HMM pageable memory, H100 Python): not related. The hang hit every GPU type, including V100, L4 and A100, which do not use HMM the same way, and the stuck threads were in the dynamic-batching dispatcher.
    • F (Fix CAGRA dataset descriptor race across threads sharing a handle #2807, CAGRA descriptor race): a different mechanism, although it is also CAGRA. Here a NaN seed caused an out-of-bounds read, and the dispatcher lost its IO buffer on an exception.
    • The two download stalls on 10-05 17:30–00:57Z (error writing zip archive … connection timed out / reset by peer, then rapids-retry) are GitHub artifact-storage infrastructure problems. Add 100 minute timeout to C++ ctest runs #2746's timeout does not cover the download step. That is an optional follow-up for CI owners; I made no change.

    Per-job table and logs are kept locally (scratchpad/wsR/lastcase.tsv).
26. rapids-bot

    added 3 commits that reference this issue on Oct 13, 2026

    Fix bin2c lookup so conda builds enable cuTile (#2820)

    ...

    853f8d2

    Load cuvs-lucene GPU indexes on first search for readers opened for a…

    ...

    73a4d89

    Query the entry node's own vector in TestAcceleratedHNSWMultiLayerRou…

    ...

    23bb188
27. bdice commented on Oct 14, 2026

    

    bdice

    on Oct 14, 2026

    ContributorAuthor

    More actions

    Status update for 2026-10-14

    Merged on 10-13: #2820 (cuTile configure check), #2795 (Lucene, D) and #2831 (Lucene multi-layer BQ test, P).

    Flake A (spectral clustering, #2094) has two open fixes. #2841 (@aamijar) and #2794 both change the expanded-L2 epilogues in l2_exp.cuh, so only one should merge:

    • Fix L2 expanded distance clamp for cutlass and legacy fused backend #2841 removes the near-zero self-distance clamp, keeps only the clamp of negative values to zero, and zeroes the diagonal when x == y.
    • Make the L2 expanded self-distance clamp relative to the norms #2794 keeps the clamp but makes its tolerance relative to the norms.

    Either one fixes the Result/7 failures on arm64 A100 CUDA 12.2 and turns #2801 green.

    CI scan, 10-12 12:00Z to 10-14: no new flaky tests.

    • Flake A (spectral) failed on [ANN_BENCH] Wrapper for Faiss CPU Binary IVF #2757, Fix bin2c lookup so conda builds enable cuTile #2820 and libcuvs-examples rebuilds libcuvs #2848.
    • Flake B (H100 oldest-deps pytest timeout, then the cuvs-bench test_cli failure) hit Clamp negative expanded L2 before sqrt in sparse distances #2738 and [FEA] int8 and uint8 k-means fit instantiations #2758. Do not treat pageable host memory as device memory on HMM systems #2829 fixes it.
    • ScaNN [CI] SCANN Test flaky #2549 failed on libcuvs-examples rebuilds libcuvs #2848. Fix reconstruction test for ScANN #2743 fixes it.
    • Everything else was infrastructure (yum repo metadata, sccache, HttpError, devcontainer), PR-specific style issues, or deterministic failures.

    CI watcher: finished after 16 checks. The 15 open fix PRs are green except #2801, which is still failing only on flake A.
28. rapids-bot

    added a commit that references this issue on Oct 14, 2026

    Use a recall-based check in TestCuVSRandomizedHNSWVectorSearch (#2824)

    ...

    32776b9
29. bdice

    changed the title [-][CI] Flaky test sweep: coordination for root-causing current CI flakes[/-] [+][CI] Flaky test sweep (Sep–Oct 2026): root causes and fix PRs[/+] on Oct 14, 2026