Repository navigation
Conversation
### What changes were proposed in this PR? Previously, a variety of things caused Texera images built from the same source to not have the same binary output from invocation to invocation. Having this kind of binary build reproducibility is a requirement when performing Apache releases not on developer-owned hardware, because the release has to be verified on developer owned hardware by rebuilding it and getting the exact same artifact. This change allows all release artifacts to be binary reproducible with effectively one piece of information, though in two different formats. These build arguments contain the date: SOURCE_DATE_EPOCH and PACKAGE_SNAPSHOT. The first sets a uniform timestamp wherever a timestamp is present, and the second sets the snapshot of the apt repositories used in image builds. One other piece of information is implicit, but important in practice. Binary reproducibility is inherently dependent on the versions of tools used to produce said binaries. Most of them are pinned within the container build files, but the container builder itself is another piece of necessary information (e.g. buildah vs. buildx) for reproducing a build from its source and time. ### Any related issues, documentation, discussions? Closes #8809. Discussion: #8807. ### How was this PR tested? `bin/README.md` documents what is pinned and how to rebuild an image; the vote email gives voters the command that rebuilds both tarballs. Meng and I went back and forth, building all of the release images on each of our machines (an ARM Mac, and a x86 Linux machine) and compared the hashes each of us got. We eventually got all of them to line up. ### Was this PR authored or co-authored using generative AI tooling? Yes, Claude Opus 5.5 --------- (backported from commit d79ac16) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
The cherry-pick conflicted and was committed with conflict markers. Resolve the conflicts on this branch, then mark this PR ready for review. Conflicting files:
|
github-actions
Bot
added
engine
dependencies
Automated Reviewer SuggestionsBased on the
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## release/v1.3 #9235 +/- ##
==================================================
- Coverage 94.11% 94.11% -0.01%
+ Complexity 4811 4810 -1
==================================================
Files 1194 1194
Lines 48648 48657 +9
Branches 5868 5868
==================================================
+ Hits 45784 45792 +8
Misses 1410 1410
- Partials 1454 1455 +1
☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
@parshimers Thanks for resolving this. One more thing: |
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 435 | 0.265 | 22,622/27,464/27,464 us | 🟢 -24.3% / 🔴 +91.9% |
| ⚪ | bs=100 sw=10 sl=64 | 966 | 0.59 | 102,630/119,214/119,214 us | ⚪ within ±5% / 🔴 +19.7% |
| 🔴 | bs=1000 sw=10 sl=64 | 1,131 | 0.691 | 879,122/1,025,886/1,025,886 us | 🔴 +9.2% / 🔴 +7.1% |
Baseline details
Latest main 8b661db from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 435 tuples/sec | 464 tuples/sec | 863.45 tuples/sec | -6.3% | -49.6% |
| bs=10 sw=10 sl=64 | MB/s | 0.265 MB/s | 0.283 MB/s | 0.527 MB/s | -6.4% | -49.7% |
| bs=10 sw=10 sl=64 | p50 | 22,622 us | 20,040 us | 11,790 us | +12.9% | +91.9% |
| bs=10 sw=10 sl=64 | p95 | 27,464 us | 36,303 us | 14,459 us | -24.3% | +89.9% |
| bs=10 sw=10 sl=64 | p99 | 27,464 us | 36,303 us | 17,630 us | -24.3% | +55.8% |
| bs=100 sw=10 sl=64 | throughput | 966 tuples/sec | 949 tuples/sec | 1,138 tuples/sec | +1.8% | -15.1% |
| bs=100 sw=10 sl=64 | MB/s | 0.59 MB/s | 0.579 MB/s | 0.694 MB/s | +1.9% | -15.0% |
| bs=100 sw=10 sl=64 | p50 | 102,630 us | 103,044 us | 92,850 us | -0.4% | +10.5% |
| bs=100 sw=10 sl=64 | p95 | 119,214 us | 124,759 us | 99,554 us | -4.4% | +19.7% |
| bs=100 sw=10 sl=64 | p99 | 119,214 us | 124,759 us | 114,049 us | -4.4% | +4.5% |
| bs=1000 sw=10 sl=64 | throughput | 1,131 tuples/sec | 1,133 tuples/sec | 1,167 tuples/sec | -0.2% | -3.1% |
| bs=1000 sw=10 sl=64 | MB/s | 0.691 MB/s | 0.691 MB/s | 0.712 MB/s | 0.0% | -3.0% |
| bs=1000 sw=10 sl=64 | p50 | 879,122 us | 878,742 us | 915,630 us | +0.0% | -4.0% |
| bs=1000 sw=10 sl=64 | p95 | 1,025,886 us | 939,443 us | 958,207 us | +9.2% | +7.1% |
| bs=1000 sw=10 sl=64 | p99 | 1,025,886 us | 939,443 us | 1,006,078 us | +9.2% | +2.0% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,460.11,200,128000,435,0.265,22621.71,27464.13,27464.13
1,100,10,64,20,2069.85,2000,1280000,966,0.590,102630.46,119214.17,119214.17
2,1000,10,64,20,17676.17,20000,12800000,1131,0.691,879121.97,1025885.57,1025885.57
|
Ah good catch. The auto merge isn't so good. Lemme fix that. |
mengw15
left a comment
mengw15
left a comment
There was a problem hiding this comment.
LGTM, thanks for addressing the comments
What changes were proposed in this PR?
Automated backport of #8812 to
release/v1.3.Source: d79ac16 · automation run
Any related issues, documentation, discussions?
Backport of #8812. Originally linked #8809.
How was this PR tested?
Release-branch CI runs on this branch once the conflicts are resolved and this PR is marked ready for review.
Was this PR authored or co-authored using generative AI tooling?
No.