Test plan: Playwright integration across gh-aw, AWF networking, and runtimes

[pelikhan]

Goal

Define and implement a shared integration-test plan for Playwright in gh-aw workflows and AWF configuration. The suite should verify both browser integrations, both local-server placements, agent-driven browser use, pre-agent JavaScript E2E use, and every supported AWF runtime.

This follows up on #2406 and #2444.

Test dimensions

Playwright integration

• Playwright MCP: browser runs in a separate container.
• Playwright CLI: browser runs as a process in the primary agent environment.

Local web server placement

• Inside the agent: server binds in the agent runtime; CLI should use loopback, while the MCP container must use the supported cross-container address/path.
• Outside the agent: server is started by the workflow/runner before AWF; both integrations must use the supported host-access address/path.

Usage mode

• Agent-driven test: ask the agent to inspect the page through Playwright and report the expected value.
• Pre-agent JavaScript E2E test: run a normal Playwright JavaScript test before an agent starts. This represents the primary non-agent Playwright usage and verifies that AWF/gh-aw setup does not break standard E2E execution.

AWF runtime

Exercise every supported primary-agent runtime/configuration:

• Docker/runc (default)
• gVisor/runsc
• Docker sbx
• Cloud Hypervisor

If a runtime is unavailable on a runner, the test must report a clear, intentional skip with the missing prerequisite rather than silently reducing coverage. The test plan should document which runner class owns each runtime.

Canonical agent test fixture

1. Generate a cryptographically random, non-secret title value for the test run.
2. Start a minimal local HTTP server in the selected location (inside or outside the agent).
3. Serve a page whose initial HTML does not contain the title; JavaScript populates document.title after page load. This prevents curl or static HTML inspection from satisfying the test.
4. Ask the agent explicitly to use the configured Playwright integration to open the page and read its rendered title.
5. Assert that the agent reports the exact generated title.
6. Preserve enough diagnostics on failure to identify the runtime, browser integration, server placement, resolved URL, and browser/network error, without logging credentials or unrelated environment data.

The fixture should use a dynamically allocated port, wait for server readiness, avoid external network dependencies, and always clean up the server/browser processes or containers.

JavaScript E2E fixture

Use the same dynamic-title page and assert the rendered document.title from a standard Playwright test. Run this before agent startup so failures are attributable to the browser/runtime setup rather than model behavior.

Required matrix

Track all combinations explicitly:

Playwright	Server location	Agent-driven	Pre-agent JS E2E	Runtime coverage
MCP container	Inside agent	Yes	N/A	All supported runtimes
MCP container	Outside agent	Yes	N/A	All supported runtimes
CLI process	Inside agent	Yes	N/A	All supported runtimes
CLI process	Outside agent	Yes	N/A	All supported runtimes
Standard Playwright JS	Inside agent context	N/A	Yes	All applicable runtimes
Standard Playwright JS	Runner/outside agent	N/A	Yes	All applicable runtimes

For combinations that are structurally impossible or not meaningful, document the reason and the nearest equivalent coverage instead of omitting them.

gh-aw and AWF configuration coverage

• Add representative gh-aw workflow sources for MCP and CLI modes.
• Verify the compiled workflow passes the required AWF networking and host-access configuration.
• Cover direct AWF configuration where behavior can differ from compiler-generated configuration.
• Assert the expected URL/address selection for each server placement; do not rely on incidental Docker bridge addresses.
• Verify the Playwright MCP image/container and Playwright CLI/browser version are pinned reproducibly.
• Keep external domain access disabled for the fixture unless installation/bootstrap requires a separately tested allowlist.

Acceptance criteria

• The matrix is represented in test definitions with stable, descriptive case names.
• The canonical dynamic-title agent test passes for MCP and CLI with the server inside and outside the agent.
• A standard JavaScript Playwright E2E test runs before agent startup and verifies the same rendered-title behavior.
• Docker/runc, gVisor/runsc, Docker sbx, and Cloud Hypervisor are covered or have explicit prerequisite-based skips assigned to suitable runners.
• Both compiler-generated gh-aw configuration and direct AWF configuration are exercised where relevant.
• Failures identify the matrix coordinates and retain actionable browser/server/network diagnostics.
• Tests use random ports, readiness checks, bounded timeouts, and reliable cleanup.
• Documentation explains the supported address/URL for each Playwright and server-placement combination.
• The suite does not permit static-fetch tools to pass the canonical agent test.

[pelikhan]

@copilot test the top 3 potential use cases to avoid case explosion.