Skip to content

feat(action): add configurable model fallback - #134

Open
l30t wants to merge 2 commits into
devfrom
leo/fail-closed-model
Open

l30t wants to merge 2 commits into
devfrom
leo/fail-closed-model

Conversation

@l30t

@l30t l30t commented Aug 29, 2026 •

Copy link
Copy Markdown

Summary

Workflows that require their requested model can opt into model_fallback: fail. Today the Action may replace a blocked or invalid model with a concrete tier-alias stand-in or the organization's default model. Fail mode stops instead, while leaving model selection and the default fallback behavior unchanged.

  • Expose the input on the main, review, security, and base actions, and forward it through preparation and execution of both review passes.
  • Use one shared parser to validate the input before policy lookup or execution.
  • Reject known policy denials during preparation and stop recognized model failures after one execution, using the existing retry predicate and error classifiers.
  • Keep transient retries, usage-limit and turn-limit protections, and current model/reasoning defaults.

Scope and impact

This is independent Action hardening, not a prerequisite for CLI-2122 or Sonnet 4 retirement. The default remains organization-default; existing workflows do not opt into strict mode automatically.

No changes to security or validator model selection, reasoning settings, debug artifact names, CLI policy enforcement, or GitLab component inputs. In fail mode, an older CLI that rejects a tier alias fails instead of using the concrete-model compatibility fallback. Tier aliases still resolve in the CLI, and a failed policy lookup still defers enforcement to the CLI/server.

Testing

Validated on 33026ce:

  • bun test: 754 passed, 0 failed.
  • bun run typecheck: passed.
  • bun run format:check: passed.
  • actionlint .github/workflows/*.yml: passed.
  • Red-first regression run against unmodified dev: 11 failures covering strict policy rejection, eager configuration validation, action input wiring, invalid aliases, and policy errors reported through stderr, result events, and agent-loop events. These pass after implementation; coverage also checks invalid concrete models and unchanged arguments on transient retries.
  • Process-level tests use a fake CLI to exercise real subprocess and retry behavior. Live Factory CLI/GitHub Action end-to-end execution was not run.

@factory-droid

factory-droid Bot commented Aug 29, 2026 •

Copy link
Copy Markdown
Contributor

Droid finished @l30t's task —— View job


Security Review

Droid review summary

Solid wiring change overall, with tests covering the new input across action variants. The main gaps are around fail-closed ergonomics: fail mode still retries deterministic model failures, and invalid fallback configuration is only validated on the disallowed-model path.

Comment thread base-action/src/run-droid.ts Outdated
Comment thread src/utils/model-policy.ts Outdated
Keep model selection unchanged and reuse existing error classifiers and retry handling. Reject blocked or invalid models without substituting another model when model_policy_fallback is fail.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
@l30t
l30t force-pushed the leo/fail-closed-model branch from 0b559cd to c372298 Compare October 10, 2026 21:31
Generalize the input name across all actions, preparation and execution, tests, and documentation without changing fallback behavior.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
@l30t l30t changed the title feat(action): add fail-closed model policy mode feat(action): add configurable model fallback Oct 10, 2026
@l30t
l30t requested a review from factory-nizar October 10, 2026 21:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant