[JackieTien97]

Description

Very large time-partition requests can keep query analysis running while retaining reservations from the operator memory pool. The Thrift buffer's 1.5x growth overflowed int above roughly 1.33 GiB, causing an entire large array to be copied for every small field write. The uncompressed frame limit was checked only after serialization, at flush().

The change makes these failures bounded and diagnosable:

• Use overflow-safe buffer growth and write positions. Bound both the initial allocation and geometric growth by the applicable write-buffer limit, and reject oversized uncompressed frames before the next allocation/copy.
• Describe the size at early rejection as a lower bound ("at least N bytes"), and identify the DataNode RPC frame-size setting.
• Preserve the limit on the compressed wire frame, while validating compression/decompression buffer sizes before allocation and closing invalid transports.
• Check the query deadline before and during time-partition enumeration.
• Propagate invalid-frame errors from ConfigNode RPCs immediately instead of repeatedly serializing the same request and replacing the error with a connection failure. Transient transport failures still retry.
• Report partition memory shortages as time-partition analysis failures with requested/available bytes. Stop applying stale result-column statistics to subsequent planning failures; keep the existing memory-quota status code.
• Reject overflowing pending memory reservations atomically, preserving earlier reservations for cleanup.
• Record an INFO context event containing the query ID and SQL when a failure escapes, including before registration. Callers retain responsibility for severity classification and stack traces, so semantic errors and timeouts are not escalated to WARN and exception stacks are not duplicated. Successful queries do not emit this context event.

Reproduction and validation

On the base revision 013622816a, a real 1,500,000,000-byte buffer grew by only 8 bytes for each of three successive writes, copying the entire array each time. With the fix, it grows once and subsequent writes reuse the buffer. This large-array check runs separately with a bounded 5 GiB heap; ordinary unit tests do not require multi-gigabyte allocations.

Regression tests exercise the real SQL parser, Coordinator, Analyzer, and generated ConfigNode Thrift serializer with test schema/partition fetchers and in-memory transports. Repeated oversized requests and partition-memory failures fully restore earlier operator-pool reservations. Tests verify INFO context records include the SQL and query ID for memory shortages, semantic errors, timeouts, and frame-size failures without an attached throwable, and that normal queries do not produce failure logs. Incremental 8-byte writes prove a 1 MiB cap never allocates beyond 1 MiB; initial capacities above the configured cap are also covered. A simulated OutOfMemoryError test verifies that SQL context is recorded and the original error is rethrown unchanged while earlier reservations are released.

84 distinct tests passed in a clean reactor build on JDK 21 (Java 17 target), covering normal and templated analysis, query retry, memory accounting, diagnostics, ConfigNode retry behavior, compressed/uncompressed frames, and buffer reuse/shrinking. Checkstyle and Spotless passed. The coverage job now explicitly selects JDK 21, matching Sonar, because the runner default JDK caused google-java-format to fail before any coverage tests could run.

mvn -B clean test -pl iotdb-core/datanode -am \
  -Dtest=AutoResizingBufferTest,TElasticFramedTransportTest,RpcUtilsTest,UrlUtilsTest,TSStatusCodeTest,QueryTimePartitionTest,QueryPartitionMemoryTest,AnalyzeTest,QueryExecutionRetryTest,MPPQueryContextTest,LocalExecutionPlannerOperatorsMemoryTest,ConfigNodeClientTest,TemplatedAnalyzeRetryTest \
  -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false

The local reproductions use controlled inputs and in-memory transports; they do not replay the original production SQL or start a live cluster.

• Self-reviewed.
• Added regression tests for failure handling and normal-query compatibility.
• Added comments explaining the overflow, limits, diagnostics, and timeout behavior.

[JackieTien97]

Reviewed at 856d4a0d82 (base 013622816a). I built base, 33277a6cf5, and this head and ran each as a 1C1D cluster (Windows, JDK 21, dn_thrift_max_frame_size=1048576, 64 devices with one point each). The inline numbers come from these clusters and from direct transport probes against the same jars.

Verified on this head

• The oversized partition request (select s1 from root.db.** where time >= 0 and time < 3000000000000, ≈3.8 MB) now fails in 35 ms with Frame size (1048582) larger than protect max size (1048576)!. At 33277a6cf5 and on base it took 15.2 s (15 reconnect attempts) and reported "Fail to connect to any config node", even with a 3 s JDBC query timeout.
• Compressed transport, two cases. At 33277a6cf5 both escaped as IllegalArgumentException, and the flush case left the transport open. Both now raise TTransportException and close the transport:
  • a maxCompressedLength overflow in flush() (forced 1.9 GB frame);
  • a varint length ≥ 2^31 in readFrame().
• A near-unbounded range (time >= 0 and time <= 9223372036854775806) now fails with 1707 "Not enough memory while analyzing time partitions … requested this time 340.87 GB …, current free memory 772.99 MB".
• The growth fix and the deadline checks look correct. startTime is only set by Coordinator, so standalone contexts are unaffected.
• All 56 tests in the 7 touched test classes pass locally: ConfigNodeClientTest 2, MPPQueryContextTest 7, QueryPartitionMemoryTest 5, QueryTimePartitionTest 16, LocalExecutionPlannerOperatorsMemoryTest 12, AutoResizingBufferTest 9, TElasticFramedTransportTest 5.

Comments

1. [P2] Coordinator: the new [QueryFailed] WARN logs every analysis/planning failure with a stack trace. That includes user errors that ErrorHandlingUtils keeps at INFO, and every other failure ends up logged twice.
2. [P3] TElasticFramedTransport#write: buffer growth still overshoots the limit by 1.30–1.44×, and the error now reports the size at the crossing write instead of the request size.

Nit: sonar-codecov.yml. codecov has failed on the last five dev/1.3 pushes with the same spotless NoSuchMethodError (e.g. run 43390944744), so this JDK pin fixes the branch, not just this PR. It may deserve its own commit or PR so the fix survives if this PR is reworked.

[JackieTien97]

[P2] This logs every analysis/planning failure at WARN with a stack trace, including user errors that ErrorHandlingUtils deliberately keeps at INFO

ErrorHandlingUtils#onQueryException already logs these failures:

• user-caused ones (SQL_PARSE_ERROR, SEMANTIC_ERROR, NO_PERMISSION, QUERY_TIMEOUT, …) at INFO with no stack trace;
• everything else with the stack.

On the client path that line already includes the SQL: Query Statement: "<sql>". executeStatement failed because …. This catch runs before that classification. DataNode log on this head, client statements via JDBC:

statement	existing ErrorHandlingUtils line	new [QueryFailed] line
select count(s1), s1 from root.db.d0 → 701	INFO, no stack	WARN + SemanticException stack (23 frames)
oversized partition request → 305	WARN + stack	WARN + the same stack again
select s1 from root.db.d0 where time >= 0 and time <= 9223372036854775806 → 1707	WARN + stack	WARN + the same stack again

The consequences:

• A dashboard that keeps sending an invalid query now writes a WARN stack trace on every execution.
• So does a query that hits the new analysis-time deadline check (QueryTimeoutRuntimeException).
• On the client path, every other failure that throws out of execution() is logged twice.

If the goal is to tie the query ID to the SQL when a failure happens before registration, two options:

• Log this line without the throwable, since callers already log it.
• Skip the classes that ErrorHandlingUtils treats as user errors (at least SemanticException and QueryTimeoutRuntimeException), or log those at DEBUG.

assertFailureLogs in QueryPartitionMemoryTest would need to follow whichever option is chosen.

[JackieTien97]

[P3] The reported size is now just past the limit, not the request size

For the same 64-device query:

• base logged Frame size (3811712) larger than protect max size (1048576)!;
• this head logs Frame size (1048582) larger than protect max size (1048576)!, which is also what the client now sees in the 305 message.

The old number showed how far over the limit the request was, which is what an operator needs in order to size dn_thrift_max_frame_size. The full size isn't known at this point anymore. Consider wording the message as "at least N bytes" and naming the setting.

[JackieTien97]

[P3] Growth can still overshoot the limit before the rejection

writeBuffer.write grows the buffer by 1.5× without knowing getMaxWriteBufferSize(). The last growth step before the rejection can therefore allocate well past the limit. Measured on this head with 8-byte writes and a 1024-byte initial buffer:

limit	capacity at rejection
1 MiB	1,513,342 (1.44×)
64 MiB (DataNode default for heaps ≥ 4 GiB)	87,266,902 (1.30×)

Passing the cap down to the write buffer would keep the array at or below the limit, e.g. min(grow, maxSize) when growing.

[codecov]

Codecov Report

❌ Patch coverage is 97.46835% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 47.47%. Comparing base (89dbf8f) to head (827763a).
⚠️ Report is 73 commits behind head on dev/1.3.

Files with missing lines	Patch %	Lines
.../org/apache/iotdb/rpc/TElasticFramedTransport.java	92.85%	1 Missing ⚠️
...e/iotdb/db/queryengine/common/MPPQueryContext.java	93.75%	1 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##             dev/1.3   #19269      +/-   ##
=============================================
+ Coverage      45.56%   47.47%   +1.91%     
- Complexity       198      221      +23     
=============================================
  Files           3667     3678      +11     
  Lines         251340   252115     +775     
  Branches       31156    31359     +203     
=============================================
+ Hits          114529   119699    +5170     
+ Misses        136811   132416    -4395     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud