CASSANDRA-21659: Make schema diffing independent of the number of tables in the cluster

[pmcfadin]

CASSANDRA-21659

Every schema change allocates in proportion to the whole schema rather than to what changed. This addresses the diffing component.

What

Keyspaces.diff and Tables.diff build created/dropped with filter(), which removes every non-matching entry from a copy one at a time. On a CREATE TABLE no keyspace is created or dropped, so Keyspaces.filter falls into withoutKsTablesViews and removes every table in the cluster individually — twice per diff. Tables.diff then calls TableMetadata.compare for every surviving table, and compare has no identity fast path.

Several diffs run per DDL: AlterSchemaStatement:194 and :200, AlterSchema:160, DistributedSchema:213, :260 and :325.

Change

• Collect created/dropped directly instead of filtering whole collections.
• Skip comparing entries carried over by reference. Untouched TableMetadata are reference-identical across a schema change (Tables.Builder.add stores the instance verbatim), and x.compare(x) is empty by construction — so identity is exact here, not an approximation.

No signature, serialized-format or API changes.

Result

Allocation for adding one table:

	400 tables	3200 tables	growth
before	292,552 B	3,774,864 B	12.9x
after	3,944 B	3,944 B	1.0x

Before is superlinear because the BTreeMap teardown is O(N log N). After is independent of schema size.

Scope — please read before benchmarking

This removes the allocation term, not the scan. Time stays O(N) per diff and bulk creation remains quadratic in wall-clock; at small N wall-clock will barely move. What it removes is allocation that scales with the size of the schema. Two further per-DDL allocation sites are being addressed separately.

Tests

New KeyspacesDiffScalingTest: 2 scaling assertions, 3 correctness guards. It asserts an allocation ratio between a 400-table and a 3200-table fixture rather than an absolute byte count, so it needs no re-tuning per JDK or machine. Allocation rather than elapsed time because it is counted exactly rather than sampled, is independent of GC timing and machine load, and is the quantity that produces the failure.

Verified red before the change and green after, and re-verified that the final test still fails with the fix stashed.

Regression: org.apache.cassandra.schema 27 suites / 130 tests, org.apache.cassandra.tcm 13 suites / 55 tests, 0 failures. ant checkstyle and ant checkstyle-test clean.

[pmcfadin]

JMH numbers, as requested. Benchmark added at test/microbench/org/apache/cassandra/test/microbench/SchemaChangeBench.java (committed on the CASSANDRA-21660 branch, since that is where all three operations can be measured together).

ant clean && ant build-test
ant microbench -Dbenchmark.name=SchemaChangeBench \
    -Djmh.args="-prof gc -f 1 -wi 3 -i 5 -r 1 -w 1"

Baseline is 8cfc3a67be, tip of cassandra-6.0 and the merge-base of this branch. Fixture is one keyspace of N tables, 3 columns each; Keyspaces.diff compares it against the same keyspace plus one table.

gc.alloc.rate.norm — B/op

operation	N	baseline	this patch
Keyspaces.diff	400	279,585	3,760
Keyspaces.diff	3200	3,697,824	3,763

Average time — ns/op

operation	N	baseline	this patch
Keyspaces.diff	400	125,025	6,489
Keyspaces.diff	3200	1,434,360	54,966

Allocation is flat: 1.00x across an 8x larger schema, where the baseline is 13.2x. Time improves 26x at 3200 tables, though as noted in the description the diff still scans both collections, so it remains O(N) — the allocation is what stops scaling, not the walk.

JMH agrees with the ThreadMXBean figures in the description to within 5% (279,585 vs 292,552 at 400; 3,697,824 vs 3,774,864 at 3200; fixed 3,760 vs 3,944). JMH reads consistently ~3% lower, which is expected: the unit test measures 30 invocations and never reaches C2, while JMH measures after seconds of warmup where escape analysis removes a few short-lived objects. The unit test is a CI regression guard rather than a benchmark — it asserts a ratio in ~3s — so the two serve different purposes and both are kept.

[pmcfadin]

Correction to an earlier claim in the description.

The description previously stated that the allocation "OOMs a node at ~3200 tables in one keyspace regardless of heap size", citing runs at 8, 12 and 24 GiB. That claim was wrong and I have removed it. The heap size in those runs was never actually varied.

ant's <junit maxmemory> attribute defaults to 1024m (build.xml:1406) and is passed to the runner at build.xml:1426. Ant emits it as -Xmx after the target's own jvmargs, so it overrode the -Xmx8G at build.xml:2055. Every in-JVM dtest in that investigation ran on a 1 GiB heap regardless of what was passed.

The arithmetic closes exactly: retained heap is ~295,796 B per table (measured), and 3,500 × 295,796 B = 0.96 GiB. The node ran out of a 1 GiB heap, which says nothing about behaviour at a realistic heap size.

What survives, and what does not:

• Unaffected: every allocation number in this PR and in the JMH comment above. Those come from ThreadMXBean and JMH -prof gc in unit/microbenchmark JVMs, not from the dtest harness. Keyspaces.diff at 3200 tables really does go from 3,697,824 B/op to 3,763 B/op, and is flat to 100,000 tables.
• Withdrawn: the specific claim that heap size does not help, and the "~3,200 table" figure as a capacity number.

With a correctly-sized heap, 10,000 tables completes: 28 minutes, ~2.75 GiB retained, on a gently rising curve with no cliff. So the failure this patch was motivated by is real but less dramatic than described — it is a scaling cost, not an unavoidable wall.

Apologies for the noise. The measurement error was mine; better to correct it here than have a reviewer find it.

[pmcfadin]

Superseded by a single combined patch for CASSANDRA-21664, which carries this commit unchanged as one of its first three commits. Closing in favour of that PR.