Repository navigation
test(workflow-operator): add test helper for FileScanUtilsSpec - #9217
aglinxinyuan wants to merge 1 commit into
Conversation
Automated Reviewer SuggestionsBased on the
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #9217 +/- ##
============================================
- Coverage 92.05% 92.04% -0.01%
+ Complexity 5041 5038 -3
============================================
Files 1286 1286
Lines 55911 55911
Branches 7137 7137
============================================
- Hits 51469 51465 -4
- Misses 2692 2693 +1
- Partials 1750 1753 +3
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 356 | 0.217 | 27,466/37,625/37,625 us | 🔴 +21.0% / 🔴 +168.1% |
| 🔴 | bs=100 sw=10 sl=64 | 783 | 0.478 | 126,294/152,086/152,086 us | 🔴 +10.0% / 🔴 +58.0% |
| ⚪ | bs=1000 sw=10 sl=64 | 950 | 0.58 | 1,047,213/1,145,714/1,145,714 us | ⚪ within ±5% / 🔴 +24.6% |
Baseline details
Latest main f673554 from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 356 tuples/sec | 420 tuples/sec | 905.73 tuples/sec | -15.2% | -60.7% |
| bs=10 sw=10 sl=64 | MB/s | 0.217 MB/s | 0.257 MB/s | 0.553 MB/s | -15.6% | -60.7% |
| bs=10 sw=10 sl=64 | p50 | 27,466 us | 22,691 us | 11,331 us | +21.0% | +142.4% |
| bs=10 sw=10 sl=64 | p95 | 37,625 us | 33,374 us | 14,032 us | +12.7% | +168.1% |
| bs=10 sw=10 sl=64 | p99 | 37,625 us | 33,374 us | 17,338 us | +12.7% | +117.0% |
| bs=100 sw=10 sl=64 | throughput | 783 tuples/sec | 853 tuples/sec | 1,205 tuples/sec | -8.2% | -35.0% |
| bs=100 sw=10 sl=64 | MB/s | 0.478 MB/s | 0.521 MB/s | 0.735 MB/s | -8.3% | -35.0% |
| bs=100 sw=10 sl=64 | p50 | 126,294 us | 114,832 us | 89,237 us | +10.0% | +41.5% |
| bs=100 sw=10 sl=64 | p95 | 152,086 us | 144,599 us | 96,280 us | +5.2% | +58.0% |
| bs=100 sw=10 sl=64 | p99 | 152,086 us | 144,599 us | 111,344 us | +5.2% | +36.6% |
| bs=1000 sw=10 sl=64 | throughput | 950 tuples/sec | 959 tuples/sec | 1,239 tuples/sec | -0.9% | -23.3% |
| bs=1000 sw=10 sl=64 | MB/s | 0.58 MB/s | 0.585 MB/s | 0.756 MB/s | -0.9% | -23.3% |
| bs=1000 sw=10 sl=64 | p50 | 1,047,213 us | 1,041,252 us | 877,826 us | +0.6% | +19.3% |
| bs=1000 sw=10 sl=64 | p95 | 1,145,714 us | 1,101,149 us | 919,412 us | +4.0% | +24.6% |
| bs=1000 sw=10 sl=64 | p99 | 1,145,714 us | 1,101,149 us | 963,000 us | +4.0% | +19.0% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,562.28,200,128000,356,0.217,27465.68,37624.83,37624.83
1,100,10,64,20,2555.31,2000,1280000,783,0.478,126293.52,152086.45,152086.45
2,1000,10,64,20,21048.93,20000,12800000,950,0.580,1047212.81,1145713.68,1145713.68|
Superseded by and combined into consolidated PR #9237. |
What changes were proposed in this PR?
In
common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/scan/file/FileScanUtilsSpec.scala:scan(...)test helper providing canonical default arguments (displayFileName = "d",attributeType = FileAttributeType.STRING,fileEncoding = FileDecodingMethod.UTF_8,extract = false,outputFileName = false,fileScanOffset = None,fileScanLimit = None) aroundFileScanUtils.createTuplesFromFile.createTuplesFromFiletest cases by adopting thescan(...)helper instead of repeating lengthy 8-argument parameter lists.memoryFailureMessage(...)test helper to encapsulate thesafeToByteArrayerror interception across 5 memory guard test cases.All 18 test cases, docstrings, assertions, and edge-case verifications are preserved untouched.
Any related issues, documentation, discussions?
Closes #9216
How was this PR tested?
Verified with sbt unit tests, operator metadata tests, and linters:
All 18 tests in
FileScanUtilsSpecpassed, all 229 operator metadata tests passed, and all scalafmt / scalafix checks passed.Was this PR authored or co-authored using generative AI tooling?
Generated-by: Antigravity (Gemini 2.5 Pro)