[CI Postmortem] AppSizeTest failures across multiple PRs (API diff not updated)

[rolfbjarne]

Summary

Xamarin.Tests.AppSizeTest tests are failing across 4 unrelated PRs (10 builds) with API diff mismatches. The tests detect added/removed APIs that haven't been accounted for in the known-failures baseline.

Failure Details

Tests: 26 AppSizeTest variants across all platforms (iOS, tvOS, macOS, Mac Catalyst)
Configurations: MonoVM, CoreCLR, NativeAOT, Interpreter, R2R, TrimmableStatic
Error pattern:

Multiple failures or warnings in test:
1) No added APIs (set the environment variable WRITE_KNOWN_FAILURES=1 and run the test again to update the expected set of APIs)
Expected: <empty>
But was: < "Microsoft.tvOS.dll:ObjCRuntime.NSObjectProxyAttribute", ...>

2) No removed APIs (set the environment variable WRITE_KNOWN_FAILURES=1...)

Key added APIs:

• ObjCRuntime.NSObjectProxyAttribute
• ObjCRuntime.ProtocolProxyAttribute
• ObjCRuntime.RegistrarHelper.LookupUnmanagedFunction
• ObjCRuntime.Runtime.ConstructINativeObject (new overload)

Key removed APIs:

• *._Xamarin_ConstructINativeObject
• *._Xamarin_ConstructNSObject
• ObjCRuntime.RegistrarHelper.LookupUnmanagedFunction (old overload)

Classification

• Type: Shared regression / baseline drift
• Impact: 4 PRs, 10 builds, 26 test variants
• Confidence: High — consistent across unrelated PRs

Affected PRs

#25077, #25079, #25105, #25197

Resolution

Run the tests with WRITE_KNOWN_FAILURES=1 to update the expected API baseline, or review if the API changes are intentional and update accordingly.

---

Filed automatically by CI postmortem analysis.

[rolfbjarne]

To Copilot: this particular test (AppSizeTest) is expected to fail in many different pull requests, because it's a very sensitive test (by design).

As such, this is not a CI issue that needs to be fixed.

[dalexsoto]

Recurrence update (2026-08-27)

Stable signature: Xamarin.Tests.AppSizeTest | TVOS tvos-arm64 | runtime-update baseline drift

Classification: deterministic expected-baseline drift; no retry is appropriate.

Newly tracked occurrences:

• PR #26454, head b956034e64eae11ea91afcec93b11996be9adbe0: run 15086535, attempt 1, dotnettests_tvos. Six runtime configurations exceeded size baselines and four preserved-API baselines changed. A later head updated the expected sizes and those baseline checks passed.
• PR #26356, head 070e656c01f70d18b2f0a3eac813c9ffbe3bc1e7: run 15114195, attempt 1, dotnettests_tvos on AcesShared 31. Four AppSizeTest cases failed: CoreCLR Interpreter grew from 8,241,763 to 8,463,881 bytes (+222,118), CoreCLR R2R grew from 11,352,311 to 11,575,430 bytes (+223,119), and the MonoVM/MonoVM Interpreter preserved-API sets changed. NUnit reported 901 total, 165 passed, 4 failed, 2 inconclusive, and 730 skipped; there were no crashes, timeouts, launch failures, or missing devices. Diagnostic artifacts: TestSummary-simulator_testsdotnettests_tvos-1, HtmlReport-simulator_testsdotnettests_tvos-1, and updated-expected-sizes-simulator_testsdotnettests_tvos-1.

Confirmed count: 2 occurrences across 2 distinct PRs. No retries were requested.

Issue-specific checklist: the issue body has no Additional info to collect section.

Diagnosis/workaround: this matches the by-design sensitivity described in the closure comment. Review and update the expected size and preserved-API baselines when the dependency changes are intentional.

Not reopening: the current evidence matches, rather than invalidates, the NOT_PLANNED closure reason.

[dalexsoto]

Additional platform recurrences (2026-08-27)

Three platform-specific Xamarin.Tests.AppSizeTest signatures are now confirmed across two monitored PRs:

• iOS / ios-arm64 — PR #26454 head b956034e..., run 15086535 and PR #26356 head 070e656c..., run 15114195. The current occurrence has four baseline failures: CoreCLR Interpreter +255,895 bytes, CoreCLR R2R +256,896 bytes, and preserved-API changes in both MonoVM configurations.
• Mac Catalyst / maccatalyst-arm64 — PR #26454, run 15086535 and PR #26356, run 15114195. The current occurrence has four baseline failures: CoreCLR Interpreter +198,073 bytes, CoreCLR R2R +200,745 bytes, and preserved-API changes in both MonoVM configurations.
• macOS / osx-arm64;osx-x64 — PR #26454, run 15086535 and PR #26356, run 15114195. The current occurrence has three file-list/size baseline failures, with CoreCLR configurations growing by approximately 4.85 MB.

Each stable signature has 2 confirmed occurrences across 2 distinct PRs. No retries were requested; these are deterministic expected-baseline changes. Diagnostic artifacts for run 15114195 include each platform's TestSummary, HtmlReport, and updated-expected-sizes artifacts.

Issue-specific checklist: the issue body has no Additional info to collect section.

The later #26454 heads updated the expected baselines and passed these checks. Review and update the expected size, file-list, and preserved-API baselines when the dependency changes are intentional.

Not reopening: these recurrences match, rather than invalidate, the NOT_PLANNED by-design-sensitivity closure reason.

[dalexsoto]

tvOS baseline fix validation (2026-08-27)

PR #26356 head 34a614dfc2cae18a1a8c8968621493ba6ed6083a updates the expected app-size files. In run 15115286, dotnettests_tvos:

• All six tvOS AppSizeTest configurations ran and passed: CoreCLR Interpreter, CoreCLR R2R, MonoVM, MonoVM Interpreter, NativeAOT, and NativeAOT TrimmableStatic.
• NUnit reported 901 total, 169 passed, 0 failed, 2 inconclusive, and 730 skipped.
• BuildNet10_0App(TVOS, "tvossimulator-arm64") also passed.

This validates the tvOS expected-baseline update. The overall simulator pipeline is still running, so validation for the other platform signatures remains pending.

Issue-specific checklist: the issue body still has no Additional info to collect section.

The issue remains closed; this successful expected-baseline update is consistent with the existing NOT_PLANNED closure reason.

[dalexsoto]

iOS baseline fix validation (2026-08-27)

PR #26356 head 10600548eb4614d4042b832a8dcc5388b9e612cf passed run 15115823, dotnettests_ios:

• All six iOS AppSizeTest configurations ran and passed: CoreCLR Interpreter, CoreCLR R2R, MonoVM, MonoVM Interpreter, NativeAOT, and NativeAOT TrimmableStatic.
• NUnit reported 901 total, 260 passed, 0 failed, 2 inconclusive, and 639 skipped.
• BuildNet10_0App(iOS, "ios-arm64") also passed, with no _mono_helper_box_nullable linker trace.

This validates the iOS expected-baseline update. Mac Catalyst validation remains active.

Issue-specific checklist: the issue body still has no Additional info to collect section.

The issue remains closed; this successful expected-baseline update is consistent with the existing NOT_PLANNED closure reason.

[dalexsoto]

Mac Catalyst and macOS baseline fix validation (2026-08-27)

PR #26356 head 10600548eb4614d4042b832a8dcc5388b9e612cf passed both remaining platform jobs in run 15115823:

• dotnettests_maccatalyst: all six Mac Catalyst AppSizeTest configurations passed. NUnit reported 901 total, 215 passed, 0 failed, 2 inconclusive, and 684 skipped.
• dotnettests_macos: all five applicable macOS AppSizeTest configurations passed. NUnit reported 901 total, 217 passed, 0 failed, 2 inconclusive, and 682 skipped.

Together with the previously reported iOS and tvOS results, this validates the expected-baseline updates on all affected platforms.

Issue-specific checklist: the issue body still has no Additional info to collect section.

The issue remains closed; these successful expected-baseline updates are consistent with the existing NOT_PLANNED closure reason.

[dalexsoto]

macOS expected-size recurrence (2026-09-02)

Stable signature: Xamarin.Tests.AppSizeTest | macOS osx-arm64;osx-x64 | expected-size baseline drift

New occurrence: PR #26513, head a354efde93df4b8e7ad9eb15a88ebd64916b2be1, run 15184608 attempt 1, T: dotnettests_macos on AcesShared 43.

Xamarin.Tests.AppSizeTest.CoreCLR_R2R(MacOSX,"osx-arm64;osx-x64",False) expected 331,609,114 bytes but produced 331,619,866 bytes: +10,752 bytes against a 10,240-byte tolerance. The updated-baseline artifact attributes 5,632 bytes to the arm64 Microsoft.macOS.dll and 5,120 bytes to the x64 Microsoft.macOS.dll. Diagnostic artifacts: TestSummary-simulator_testsdotnettests_macos-1, HtmlReport-simulator_testsdotnettests_macos-1, and updated-expected-sizes-simulator_testsdotnettests_macos-1.

Classification: deterministic expected-size baseline drift. This PR already changes MacOSX-CoreCLR-R2R-size.txt; the current merge build needs the newly generated expected value reviewed and applied. No retry was requested, and the Rolf-authored branch remains unchanged.

Confirmed macOS-signature count: 3 occurrences across 3 distinct PRs (#26454, #26356, #26513).

Issue-specific checklist: the issue body has no Additional info to collect section.

Not reopening: this recurrence matches the documented by-design sensitivity and does not invalidate the NOT_PLANNED closure reason.

[dalexsoto]

New macOS expected-size occurrence on PR #26313 @ 16a37c0640f986dfcf2b8c8d7a94a96c21cd738f:

• Build 15185589 attempt 1, T: dotnettests_macos ran on AcesShared 82.
• Xamarin.Tests.AppSizeTest.CoreCLR_R2R(MacOSX,"osx-arm64;osx-x64",False) expected 331,621,659 bytes and produced 331,634,202 bytes: +12,543 bytes against the 10,240-byte tolerance.
• The updated-baseline artifact attributes +7,168 bytes to arm64 Microsoft.macOS.dll, +6,144 bytes to x64 Microsoft.macOS.dll, -736 bytes to the app executable, and -33 bytes to Info.plist.
• Classification: deterministic expected-size baseline drift. The PR intentionally changes Export-attribute trimming and already updates related preserved-API and size baselines; the newly generated expected value requires source review.
• Build 15185589 remains active. No retry was requested, and the Rolf-authored branch remains unchanged.

The macOS-signature count is now 4 confirmed occurrences across 4 distinct PRs (#26454, #26356, #26513, #26313).

The issue body has no Additional info to collect checklist. Not reopening: this remains consistent with the documented by-design sensitivity and NOT_PLANNED closure reason.

[dalexsoto]

Fix outcome for PR #26513, replacement head dc916b616979c11042468b0a11894d28ed84ea17:

• The replacement commit updates MacOSX-CoreCLR-R2R-size.txt with the reviewed expected-size baseline.
• Simulator build 15194730 / T: dotnettests_macos completed successfully at 2026-09-03 06:00:19 UTC, and the overall GitHub simulator check completed successfully at 06:56:04 UTC.
• AppSizeTest.CoreCLR_R2R passed; the prior [CoreFoundation] Fix numerous issues with CFMessagePort. #26513 expected-size occurrence is resolved on this head.

The macOS-signature count remains 4 confirmed occurrences across 4 distinct PRs. The issue remains closed as NOT_PLANNED; this expected baseline update does not invalidate the by-design closure reason. The issue body has no Additional info to collect checklist.

[dalexsoto]

New .NET 11 dependency-driven macOS AppSize recurrence:

• PR [main] Update dependencies from dotnet/dotnet #26554, head 862c1b35c2bc5f9308ea8fa65fb02f907d205839, build 15198573 / T: dotnettests_macos, on AcesShared 50.
• The PR updates the .NET 11 RC dependency set from 11.0.100-rc.2.26452.102 to 11.0.100-rc.2.26452.117.
• Three expected-size cases crossed the 10,240-byte tolerance:
  • CoreCLR_Interpreter_TrimmableStatic: 220,774,802 → 220,764,474 bytes (-10,328).
  • CoreCLR_Interpreter: 220,774,802 → 220,764,474 bytes (-10,328).
  • CoreCLR_R2R: 331,782,521 → 331,768,634 bytes (-13,887).
• The generated updated-expected-sizes-simulator_testsdotnettests_macos-1 artifact contains the corresponding three expected-file diffs. The harness recorded no crash, timeout, launch failure, or missing device.

This matches the prior #26521 .NET 11 dependency/AppSize signature. The exact signature now has 24 confirmed test-case occurrences across 2 distinct PRs (#26521 and #26554), including two macOS job-level recurrences. Classification remains deterministic expected-baseline drift; no retry was requested, and the bot-authored branch was not modified.

The issue body has no Additional info to collect checklist. The issue remains closed as NOT_PLANNED; this by-design-sensitive recurrence does not invalidate its closure reason.

[dalexsoto]

One replacement-head macOS AppSize occurrence remains on PR #26554 @ a17b2e3290d91f5f760494df421527c800dfd61e:

• Build 15209106 attempt 1 / T: dotnettests_macos / Run tests completed at 2026-09-04T04:17:01Z.
• Xamarin.Tests.AppSizeTest.CoreCLR_R2R(MacOSX,"osx-arm64;osx-x64",False) expected 331,782,521 bytes and produced 331,768,866 bytes: -13,655 bytes, beyond the ±10,240-byte tolerance.
• Artifact: updated-expected-sizes-simulator_testsdotnettests_macos-1/MacOSX-CoreCLR-R2R-size.txt.diff.

The two CoreCLR_Interpreter cases that failed on the superseded head no longer cross the threshold; only CoreCLR_R2R remains. The canonical .NET 11 AppSize signature now contains 25 confirmed test-case occurrences across 2 distinct monitored PRs.

Classification remains deterministic expected-size baseline drift. No retry was submitted, and the bot-authored branch was not modified. This evidence does not invalidate the issue's NOT_PLANNED closure, so it remains closed.

Issue-specific checklist: none; the issue body has no Additional info to collect section.

[dalexsoto]

Fix validation for PR #26554 replacement head 7e8c4edb02eb98aa7ad72f1da2e6755a61af1517 ([tests] Update expected app size files):

• Simulator build 15210709 / T: dotnettests_macos completed successfully at 2026-09-04T06:38:07Z.
• Its Run tests task completed successfully with zero errors at 2026-09-04T06:37:14Z.
• Xamarin.Tests.AppSizeTest.CoreCLR_R2R(MacOSX,"osx-arm64;osx-x64",False) passed; the prior replacement-head delta of -13,655 bytes did not recur.

This validates the expected-size update for the recorded #26554 AppSize occurrence. The canonical count remains 25 confirmed test-case occurrences across 2 distinct monitored PRs.

The issue remains closed as NOT_PLANNED; this expected baseline update does not invalidate its closure reason. These were fresh automatic CI runs, not retries, and the bot-authored branch was not modified by the monitor.

Issue-specific checklist: none; the issue body has no Additional info to collect section.

[dalexsoto]

Two matched macOS R2R size-baseline occurrences; keeping this issue closed

Two independently inspected runs fail Xamarin.Tests.AppSizeTest.CoreCLR_R2R(MacOSX,"osx-arm64;osx-x64",False) with the same narrow four-assembly growth pattern:

• [main] Update dependencies from dotnet/xharness #26574 / 36d53003571d8cf3c57b5fb182de26320e57108e — run 15400505, attempt 1, task log 2890; test 2026-09-21 18:42:29–18:43:55 UTC.
• [main] Update dependencies from dotnet/macios #26669 / superseded head 84a0c36a97a761eea2654171cb17b0a0561c90c3 — run 15403840, attempt 1, task log 2613; test 22:01:30–22:02:49 UTC. This is not attributed to current head 9416a96ea970676e00cd4a1824d45ed51dcfd486, whose CI is separate.

Both use pipeline xamarin-macios-sim-pr-tests (21922), job T: dotnettests_macos, task Run tests, and stage/job/task attempt 1. This is the actual app-byte-size assertion, not an informational current-PR API diff.

PR	Expected bytes	Actual bytes	Difference
#26574	333,112,974	333,126,159	+13,185
#26669, old head	333,102,819	333,115,407	+12,588

The exact-head gate allows ±10,240 bytes; these exceed it by 2,945 and 2,348 bytes. Full per-file delta sums match the totals.

Correlation beyond the test name: in both runs, Microsoft.macOS.dll grows 4,096 bytes arm64 / 3,584 bytes x64, and System.Private.CoreLib.dll grows 3,072 / 2,560 bytes relative to each run's own baseline. The actual CoreLib binaries are byte-identical across the two produced app bundles for each architecture. Both use the same AppSizeTest source and SDK 11.0.100-rc.2.26465.108. Other file changes net −127 and −724 bytes. The binding assemblies are not claimed byte-identical, nor is a particular compiler optimization or shared PR source regression inferred.

Classification/counts: deterministic expected-size baseline mismatch. 2 confirmed test occurrences across 2 distinct monitored PRs for this exact four-component signature. These scoped counts do not replace older, different signatures or historical totals in this issue.

Decision: reuse this existing closed / NOT_PLANNED baseline record; do not reopen. The evidence is consistent with the documented by-design sensitivity, not proof of a new infrastructure defect. No retry, source edit, suppression, or expected-file update was performed. There is no verified retry/fix outcome for these exact records. Baseline review belongs to the branch owners; the older #26669 result does not establish failure or recovery on its replacement head.

Diagnostics/checklist: the refreshed issue body has no Additional info to collect section. For each occurrence, exact head/run/attempt/job/task/test, original NUnit assertion/output, exact-head baseline/gate, full size deltas and actual failed app-bundle binary comparisons were collected. The lower-level reason for the R2R layout differences remains unestablished; no such explanation is invented. Worker identifiers and IP addresses are withheld.

Evidence is in each linked run's HtmlReport-simulator_testsdotnettests_macos-1, updated-expected-sizes-simulator_testsdotnettests_macos-1, and failed-app-size-bundles-simulator_testsdotnettests_macos-1 artifacts, subject to retention.

Stable signature: deterministic-test|AppSizeTest.CoreCLR_R2R|macOS-universal|Microsoft.macOS-and-System.Private.CoreLib-size-growth|outside-10240