Repository navigation
Fix release build: native ARM runner and proper failure detection - #140
Merged
Merged
Conversation
The arm64 release job emulated aarch64 via QEMU, which made the per-distro openssl/curl/zlib builds take hours and occasionally stall (timeout-minutes was 1440, so a stuck run held a runner for a day). - Run the arm64 matrix leg on ubuntu-24.04-arm (free for public repos) and drop the QEMU setup step - Fetch the ghr binary matching the runner architecture; it was pinned to linux_amd64, which only worked because the host was x86_64 - Lower timeout-minutes to 120 so a stalled build fails fast
build.sh started each distro container with a bare `docker compose up` and discarded both its exit code and the result of `wait`. Since `docker compose up` does not propagate a container failure without --exit-code-from, a distro whose build broke still left `make pkg` successful, and ghr published a release with the packages missing. v2.6.9 and v2.6.10 shipped with no deb/rpm at all, and v2.6.11 with amd64 only, while every job reported success. - Pass --exit-code-from so a failing container fails its compose run - Collect the exit status of each background job via wait and exit 1 when any service failed - Record image build failures too, and skip starting those services
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景
Release ワークフローの arm64 ジョブが終わらなくなっていた (32815416360 は build ステップで 1h50m 停滞)。調べる過程で、リリースにパッケージが公開されていないという別の問題も判明した。
原因は 2 つある。
1. arm64 を QEMU エミュレーションでビルドしていた
arm64 は
docker/setup-qemu-actionによる aarch64 エミュレーション上で、7 ディストリ分の openssl / curl / zlib をソースからフルビルドしていた。常に amd64 の 5 倍前後かかり、負荷次第で頭打ちになる。timeout-minutes: 1440のため、停滞すると丸一日 runner を占有する。flowchart LR subgraph before["変更前"] A1[ubuntu-latest<br/>x86_64] --> A2[QEMU<br/>aarch64 emulation] --> A3[7 distro × openssl/curl/zlib<br/>48分〜停滞] end subgraph after["変更後"] B1[ubuntu-24.04-arm<br/>native aarch64] --> B3[7 distro × openssl/curl/zlib] end2. build.sh がコンテナの失敗を握り潰していた
各ディストロのコンテナを素の
docker compose upで起動し、その終了コードもwaitの結果も捨てていた。docker compose upは--exit-code-fromを付けないとコンテナの失敗を終了コードに反映しないため、ビルドが全滅してもmake pkgは成功し、ghrが中身の無いbuilds/をそのまま公開して終わっていた。上表の v2.6.9 以降がこれにあたる。変更内容
.github/workflows/release.ymlubuntu-24.04-armのネイティブ ARM runner で実行する。matrix をinclude形式にしてruns-onをアーキごとに切り替え、QEMU セットアップステップを削除ghrのダウンロードをmatrix.arch追従に変更。linux_amd64固定だったものが、ネイティブ ARM runner では実行できないためtimeout-minutesを 1440 → 120 に短縮build.shdocker compose upに--exit-code-fromを渡し、コンテナの失敗を compose の終了コードに反映waitで回収し、1 つでも失敗したらexit 1docker compose build失敗も記録し、そのサービスは起動しない動作確認
build.shを軽量な compose ファイル (busybox) に差し替えて検証した。The following services failed: bCONCURRENT_LIMIT=5)release.ymlは YAML をパースしてジョブ定義を検証済み。ghrv0.16.2 にlinux_arm64アセットが存在することも確認済み。実際のリリースビルドはタグ push 時に走るため、所要時間とパッケージの公開はマージ後の次リリースで確認したい。