feat: DH-23748: Multipass Release Matrix Workflow by stanbrub · Pull Request #438 · deephaven/benchmark

stanbrub · 2026-10-01T23:15:22Z

Reworked Release workflow to use matrix configs
Improved dealing with APT conflicts after auto-provision
Get version label from actual deephaven.version platform property
Changed to using two always-on servers instead of one
For release, use same iteration count for all benchmarks

Copilot

Copilot review overview

🟡 Changes recommended

The release profile remains temporary, repeated artifact names fail, mutable edge images are unpinned, and APT services are still interrupted.

Review effort: Balanced
Findings: 2 · 2

Open (4)

No-block stop can orphan active APT processes · New
Release matrix artifact names collide across rows · New
Release profile runs incomplete benchmark suite · New
Mutable edge image tag makes release non-reproducible · New

What changed in this PR

Introduces matrix-driven release benchmarks, dedicated benchmark hosts, improved provisioning, and engine-derived release labels.

Changes:

Adds release and existing-server matrix orchestration.
Separates nightly and matrix hosts.
Improves APT setup, result labeling, filtering, and log redaction.

File	Description
`docs/ForkSetup.md`	Documents separate host secrets.
`.github/workflows/remote-benchmarks.yml`	Adds host routing and controlled clearing.
`.github/workflows/release-remote-benchmarks.yml`	Converts releases to a matrix workflow.
`.github/workflows/nightly-remote-benchmarks.yml`	Enables pre-run clearing.
`.github/workflows/matrix-exist-remote-benchmarks.yml`	Integrates clearing and matrix run types.
`.github/workflows/compare-remote-benchmarks.yml`	Uses the matrix host.
`.github/workflows/adhoc-exist-remote-benchmarks.yml`	Moves clearing into the reusable workflow.
`.github/workflows/adhoc-auto-remote-benchmarks.yml`	Clears before both comparison runs.
`.github/scripts/setup-test-server-remote.sh`	Revises automatic-update handling.
`.github/scripts/setup-ssh-local.sh`	Detects unresolved host aliases.
`.github/scripts/run-ssh-local.sh`	Redacts host addresses from logs.
`.github/scripts/run-benchmarks-remote.sh`	Applies class filters with tag filters.
`.github/scripts/matrix_array.sh`	Gives releases uniform iterations.
`.github/scripts/fetch-results-local.sh`	Derives labels from engine versions.
`.github/scripts/build-matrix.sh`	Adds run types and matrix validation.
`.github/scripts/adhoc.sh`	Adds cloud-init data to provisioning.
`.github/resources/adhoc-server-cloud-init.yml`	Disables automatic APT services at boot.
`.github/matrix/release.sh`	Defines repeated release passes.
`.github/matrix/gc-train-heap-fail.sh`	Adds matrix run-type output.
`.github/matrix/gc-train-1000-100.sh`	Adds matrix run-type output.
`.github/matrix/example-regression-matrix.sh`	Adds matrix run-type output.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot

Copilot review overview

🔵 Needs a closer look

Release rows reuse an artifact name, and mutable edge images can change between sequential passes.

Review effort: Balanced
Findings: 2 · 1

Open (3)

Release matrix artifact names collide across rows
No-block stop can orphan active APT processes
Mutable edge image tag makes release non-reproducible

Resolved since last review (1)

Release profile runs incomplete benchmark suite

Copilot

Copilot review overview

🔵 Needs a closer look

Repeated release artifact names will fail after the first matrix leg, and failed dpkg repairs are currently ignored.

Review effort: Balanced
Findings: 1 · 1

Open (2)

Release matrix artifact names collide across rows
Mutable edge image tag makes release non-reproducible

Resolved since last review (1)

No-block stop can orphan active APT processes

Previously missed (1)

In code that hasn't changed since last review

Do not ignore dpkg repair failures

.github/scripts/setup-test-server-remote.sh:69

The repair command's failure is discarded even though the following package installs depend on a consistent dpkg state. If an interrupted transaction cannot be configured, continuing only defers the failure (or leaves the host partially configured); let errexit stop setup at the repair failure.

Copilot

Copilot review overview

🔵 Needs a closer look

The coupled remote-host lifecycle, cloud provisioning, and release publication changes require end-to-end workflow validation.

Review effort: Balanced
Findings: 1 · 1

Open (2)

Release matrix artifact names collide across rows
Mutable edge image tag makes release non-reproducible

Previously missed (1)

In code that hasn't changed since last review

Use canonical capitalization for GitHub

.github/scripts/run-ssh-local.sh:44

Use the product name’s canonical capitalization: “GitHub.”

Copilot

Copilot review overview

🟡 Changes recommended

Server cleanup depends on matrix scheduling order, which can cause stale state or mid-run clearing.

Review effort: Balanced
Findings: 2

Open (2)

Matrix order does not guarantee server initialization · New
Matrix scheduling can mix image revisions across release jobs · New

Resolved since last review (2)

Release matrix artifact names collide across rows
Mutable edge image tag makes release non-reproducible

      distribution: ${{ matrix.distribution }}
-      test_device_addr: "<default>"
+      test_device_addr: "<matrix>"
+      clear_server: ${{ strategy.job-index == 0 }}


+      scale_row_count: ${{ matrix.scale_row_count }}
+      distribution: ${{ matrix.distribution }}
+      test_device_addr: "<matrix>"
+      clear_server: ${{ strategy.job-index == 0 }}


Copilot

Copilot review overview

🔵 Needs a closer look

The release matrix still uses temporary values that publish only two benchmark classes across three passes.

Review effort: Balanced
Findings: 2

Open (2)

Matrix scheduling can mix image revisions across release jobs
Matrix order does not guarantee server initialization

stanbrub added 4 commits September 30, 2026 12:16

Updating workflows and server clear for two always-on servers

c1e170d

Reworked Release runs to use matrix

638cbe7

Reduce iterations/tests for testing

37ed711

Fix APT deadlock

8cde852

stanbrub self-assigned this Oct 1, 2026

Copilot AI balanced review requested due to automatic review settings October 1, 2026 23:15

Copilot started reviewing on behalf of stanbrub October 1, 2026 23:15 View session

Copilot AI reviewed Oct 1, 2026

View reviewed changes

Comment thread .github/scripts/setup-test-server-remote.sh Outdated

Comment thread .github/workflows/release-remote-benchmarks.yml

Comment thread .github/matrix/release.sh

Comment thread .github/workflows/release-remote-benchmarks.yml

Put release matrix config back to production values

cadae96

Copilot AI balanced review requested due to automatic review settings October 1, 2026 23:19

Copilot started reviewing on behalf of stanbrub October 1, 2026 23:19 View session

Copilot AI reviewed Oct 1, 2026

View reviewed changes

fixed another apt stop issue

a474200

Copilot AI balanced review requested due to automatic review settings October 1, 2026 23:30

Copilot started reviewing on behalf of stanbrub October 1, 2026 23:31 View session

Copilot AI reviewed Oct 1, 2026

View reviewed changes

Fixed GH upload artifact name to be unique to the run

84fa407

Copilot AI balanced review requested due to automatic review settings October 1, 2026 23:44

Copilot started reviewing on behalf of stanbrub October 1, 2026 23:45 View session

Copilot AI reviewed Oct 1, 2026

View reviewed changes

Pull any images only if absent

e8164b1

Copilot AI balanced review requested due to automatic review settings October 2, 2026 01:10

Copilot started reviewing on behalf of stanbrub October 2, 2026 01:10 View session

Test short matrix run for edge caching

5c0fd2f

Copilot AI reviewed Oct 2, 2026

View reviewed changes

Copilot AI balanced review requested due to automatic review settings October 2, 2026 01:15

Copilot started reviewing on behalf of stanbrub October 2, 2026 01:16 View session

Copilot AI reviewed Oct 2, 2026

View reviewed changes

Navigation Menu

feat: DH-23748: Multipass Release Matrix Workflow - #438

feat: DH-23748: Multipass Release Matrix Workflow#438
stanbrub wants to merge 9 commits into
deephaven:mainfrom
stanbrub:release-matrix-and-dash

stanbrub commented Oct 1, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

stanbrub commented Oct 1, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🔵 Needs a closer look

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants