From 094dedac8f43bde52efbc5dc26ad8ba61eba1c7e Mon Sep 17 00:00:00 2001 From: Alex Steiner Date: Fri, 4 Sep 2026 20:06:08 -0700 Subject: [PATCH 1/2] fix(escalation): require fresh consistent failure evidence Signed-off-by: Alex Steiner --- benchmark/SWE_ATLAS_ESCALATION_REPORT.md | 81 ++++++ crates/libsy/src/algorithms/escalation.rs | 179 +++++++++++- .../libsy/src/algorithms/util/escalation.rs | 260 +++++++++++++++++- crates/libsy/src/algorithms/util/llm_judge.rs | 2 +- crates/libsy/src/prompts/escalation/prompt.md | 81 ++++-- .../libsy/src/prompts/escalation/schema.json | 11 +- crates/switchyard-server/tests/server.rs | 3 +- docs/reference/toml_schema.md | 2 +- .../escalation_router_routing.md | 20 +- 9 files changed, 575 insertions(+), 64 deletions(-) create mode 100644 benchmark/SWE_ATLAS_ESCALATION_REPORT.md diff --git a/benchmark/SWE_ATLAS_ESCALATION_REPORT.md b/benchmark/SWE_ATLAS_ESCALATION_REPORT.md new file mode 100644 index 000000000..f79b447ad --- /dev/null +++ b/benchmark/SWE_ATLAS_ESCALATION_REPORT.md @@ -0,0 +1,81 @@ +# SWE-Atlas escalation-router findings + +## Summary + +The escalation router could interpret duplicate command serialization in a NeMo Gym Terminus +trajectory as repeated failed work. A single assistant turn may contain the same command both as +raw JSON text and as a structured tool call. The judge prompt previously treated a command shown +two or more times as loop evidence without requiring those attempts to occur in distinct turns. + +This change makes escalation depend on fresh, consistent failure evidence across turns. It keeps +the existing one-way switch and latch: a session can move from the efficient model to the capable +model at most once and remains there afterward. + +## Changes + +- Normalize only the transcript sent to the escalation judge. When a raw Terminus command batch + has a one-to-one match with structured tool calls in the same assistant turn, the redundant raw + representation is omitted. The worker model still receives the original history. +- Require the judge to classify positive evidence as `repetition`, `false_progress`, `drift`, + `desperation`, or `capability_gap`, and report whether the evidence is new. +- Advance the confirmation streak only for fresh positive evidence in the same category. A + category change starts a new streak; a decline or stale evidence resets it. An unavailable judge + preserves the prior streak rather than converting an infrastructure failure into a routing + decision. +- Clarify in the judge prompt that duplicate serialization within one turn and an immediate, + adaptive change of terminal strategy are not repeated failed attempts. +- Preserve the existing maximum-one-switch and post-switch latch behavior. + +## Live SWE-Atlas observations + +The live checks used GLM 5.2 as the efficient model and Opus 4.8 as the capable model. The reward +column is the task's binary verifier result. "GLM only" means that the patched router did not +switch models during the trajectory. + +| Dataset | Task | Patched behavior | Reward | Observation | +| --- | --- | --- | ---: | --- | +| RF | `697e7458be1623d850a88838` | 28 GLM calls, then 36 Opus calls; one switch | 1 | Two fresh `repetition` findings confirmed the escalation path and latch. | +| RF | `69391d8d1ce51c407be1e531` | GLM only; 93 logged episodes | 0 | Findings changed from `repetition` to `false_progress`, so they did not form one confirmation streak. The earlier router switched without improving the reward. | +| RF | `694b4b99829f00e24fd11889` | GLM only; 34 logged episodes | 0 | No positive escalation finding. | +| TW | `6902ef3ab97fe23e2ad271f3` | GLM only; 58 logged episodes | 0 | One isolated `repetition` finding did not trigger a switch. The earlier router switched without improving the reward. | +| RF | `696719205599a51110d4b45f` | Three GLM-only runs: 72, 93, and 70 logged episodes | 0, 0, 0 | One isolated positive finding across the replicas; no latch. Direct GLM and direct Opus also scored 0. | +| RF | `696719205599a51110d4b455` | Three GLM-only runs: 51, 60, and 48 logged episodes | 0, 0, 0 | No transition in any replica. Direct GLM and direct Opus also scored 0. | + +A separate passing run of `697e7458be1623d850a88838` produced a positive finding followed by a +decline, stayed on GLM, and scored 1. This checks that non-consecutive findings do not accumulate. + +The positive switched task also scored 1 in a direct-GLM run. It therefore demonstrates correct +switching, confirmation, and latching, but does not establish a causal accuracy improvement. +Similarly, the unsuccessful controls show improved escalation precision and avoided Opus calls; +they do not show that the efficient model could solve those tasks. + +## Validation evidence + +The implementation was validated with Rust 1.96.1 on a Slurm compute node using: + +```text +cargo fmt --all --check +cargo clippy --workspace --all-targets --all-features -- -D warnings +cargo test --workspace --all-features +uv run ruff check . +uv run mypy switchyard +uv run maturin develop --uv +uv run pytest tests/ -v +``` + +The validation completed successfully: the Rust workspace passed, including 289 `libsy` tests; +ruff and mypy passed; the native wheel built and installed; and pytest reported 117 passed tests. +Focused regression coverage includes exact and multi-command transcript normalization, partial and +multiplicity mismatches, category changes, stale evidence, unavailable-judge behavior, and the +one-way latch. + +No live provider calls are made by the repository validation commands. The live SWE-Atlas checks +described above were separate, explicitly configured benchmark runs. + +## Conclusion + +The evidence supports a narrower conclusion than an aggregate benchmark improvement: the router +now distinguishes repeated cross-turn failure from within-turn serialization noise, avoids several +observed unnecessary escalations, and still performs a confirmed one-time escalation when fresh +same-category evidence persists. A larger paired run is required to estimate accuracy and cost +effects with confidence. diff --git a/crates/libsy/src/algorithms/escalation.rs b/crates/libsy/src/algorithms/escalation.rs index dfe2c7fe3..5b4f8fe35 100644 --- a/crates/libsy/src/algorithms/escalation.rs +++ b/crates/libsy/src/algorithms/escalation.rs @@ -13,7 +13,9 @@ use switchyard_protocol::{ use super::util::buffered_response::buffer_response; use super::util::classifier_contract::ClassifierContractConfig; use super::util::decisive; -use super::util::escalation::{self, EscalationJudge, EscalationJudgeConfig, EscalationPolicy}; +use super::util::escalation::{ + self, EscalationCategory, EscalationJudge, EscalationJudgeConfig, EscalationPolicy, +}; use super::util::llm_judge::JudgeClassifier; use crate::core::algorithm::Driver; use crate::core::classifier::{Classification, Classifier}; @@ -22,6 +24,8 @@ use crate::{LibsyError, Result}; /// Session-state key holding the consecutive-escalate streak. const STREAK_KEY: &str = "escalation_streak"; +/// Session-state key holding the category currently being confirmed. +const CATEGORY_KEY: &str = "escalation_category"; fn streak(state: &State) -> u32 { match state.extra.get(STREAK_KEY) { @@ -30,6 +34,27 @@ fn streak(state: &State) -> u32 { } } +fn category(state: &State) -> Option<&str> { + match state.extra.get(CATEGORY_KEY) { + Some(StateValue::String(category)) => Some(category), + _ => None, + } +} + +fn bounded_reason(reason: &str) -> String { + reason + .chars() + .map(|character| { + if character.is_control() { + ' ' + } else { + character + } + }) + .take(240) + .collect() +} + fn assistant_message(response: &AggLlmResponse) -> Message { Message { role: Role::Assistant, @@ -142,18 +167,46 @@ impl Classifier for EscalationClassifier { .messages .push(assistant_message(&efficient_response.agg)); - let (classification, _) = self.judge.score(state, &mut judge_request, driver).await?; + let verdict = self.judge.verdict(state, &judge_request, driver).await; let held = streak(state); - let best = classification.argmax(false)?; - let (escalate, pending) = match &best { - Some(score) if score.target == capable => (true, held + 1), - Some(_) => (false, 0), - None => (false, held), + let held_category = category(state).map(str::to_string); + let (escalate, pending, pending_category) = match verdict.as_ref() { + Some(verdict) => { + let category = verdict.category.label(); + tracing::info!( + escalate = verdict.escalate, + category, + new_evidence = verdict.new_evidence, + reason = %bounded_reason(&verdict.reason), + "escalation judge verdict" + ); + if verdict.escalate + && verdict.new_evidence + && verdict.category != EscalationCategory::None + { + let next = if held_category.as_deref() == Some(category) { + held + 1 + } else { + 1 + }; + (true, next, Some(category.to_string())) + } else { + (false, 0, None) + } + } + None => (false, held, held_category), }; state .extra .insert(STREAK_KEY.to_string(), StateValue::Count(pending)); + if let Some(category) = pending_category { + state + .extra + .insert(CATEGORY_KEY.to_string(), StateValue::String(category)); + } else { + state.extra.remove(CATEGORY_KEY); + } if escalate && pending >= self.confirmations { // Streak confirmed: drop the efficient response, caller will serve capable. @@ -277,13 +330,13 @@ mod tests { } } - /// Builds a router with escalation enabled (`confirmations=1` latches immediately). - fn escalation_router() -> Result> { + /// Builds a router with escalation enabled. + fn escalation_router_with_confirmations(confirmations: u32) -> Result> { Ok(Arc::new(LlmTaskClassifier::new( LlmClassifierConfig::Escalation { contract: ClassifierContractConfig::default(), config: EscalationJudgeConfig { - confirmations: 1, + confirmations, ..EscalationJudgeConfig::default() }, max_output_tokens: DEFAULT_JUDGE_MAX_OUTPUT_TOKENS, @@ -291,9 +344,16 @@ mod tests { )?)) } + /// Builds a router that latches on its first supported escalation verdict. + fn escalation_router() -> Result> { + escalation_router_with_confirmations(1) + } + #[tokio::test] async fn serves_efficient_when_judge_declines() -> Result<()> { - let judge = Queue::new([r#"{"escalate":false,"reason":"progressing"}"#]); + let judge = Queue::new([ + r#"{"escalate":false,"category":"none","new_evidence":false,"reason":"progressing"}"#, + ]); let model = Queue::new(["efficient answer"]); let (selected_model, response) = test_drive_with_models( @@ -329,7 +389,9 @@ mod tests { }) }); recorded.lock().extend(prompt); - std::future::ready(Ok(reply(r#"{"escalate":false,"reason":"progressing"}"#))) + std::future::ready(Ok(reply( + r#"{"escalate":false,"category":"none","new_evidence":false,"reason":"progressing"}"#, + ))) } else { std::future::ready(Ok(reply("efficient answer"))) } @@ -351,7 +413,9 @@ mod tests { #[tokio::test] async fn upgrades_to_capable_when_judge_escalates() -> Result<()> { - let judge = Queue::new([r#"{"escalate":true,"reason":"stuck in a loop"}"#]); + let judge = Queue::new([ + r#"{"escalate":true,"category":"repetition","new_evidence":true,"reason":"stuck in a loop"}"#, + ]); let model = Queue::new(["efficient draft", "capable answer"]); let (selected_model, response) = test_drive_with_models( @@ -370,9 +434,96 @@ mod tests { Ok(()) } + #[tokio::test] + async fn confirmation_streak_requires_the_same_category() -> Result<()> { + let judge = Queue::new([ + r#"{"escalate":true,"category":"repetition","new_evidence":true,"reason":"repeated command"}"#, + r#"{"escalate":true,"category":"drift","new_evidence":true,"reason":"off task"}"#, + r#"{"escalate":true,"category":"drift","new_evidence":true,"reason":"still off task"}"#, + ]); + let model = Queue::new(["efficient t1", "efficient t2", "efficient t3", "capable t3"]); + let router = escalation_router_with_confirmations(2)?; + let request = classify_session_request(); + + let (first, _) = test_drive( + router.clone(), + request.clone(), + queued(Arc::clone(&model), Arc::clone(&judge)), + ) + .await?; + let (second, _) = test_drive( + router.clone(), + request.clone(), + queued(Arc::clone(&model), Arc::clone(&judge)), + ) + .await?; + let (third, _) = test_drive(router, request, queued(model, judge)).await?; + + assert_eq!(first, "efficient"); + assert_eq!(second, "efficient"); + assert_eq!(third, "capable"); + Ok(()) + } + + #[tokio::test] + async fn verdict_without_new_evidence_resets_the_streak() -> Result<()> { + let judge = Queue::new([ + r#"{"escalate":true,"category":"repetition","new_evidence":true,"reason":"repeated command"}"#, + r#"{"escalate":true,"category":"repetition","new_evidence":false,"reason":"only old evidence remains"}"#, + r#"{"escalate":true,"category":"repetition","new_evidence":true,"reason":"new repeated command"}"#, + ]); + let model = Queue::new(["efficient t1", "efficient t2", "efficient t3"]); + let router = escalation_router_with_confirmations(2)?; + let request = classify_session_request(); + + for _ in 0..3 { + let (selected, _) = test_drive( + router.clone(), + request.clone(), + queued(Arc::clone(&model), Arc::clone(&judge)), + ) + .await?; + assert_eq!(selected, "efficient"); + } + Ok(()) + } + + #[tokio::test] + async fn unavailable_judge_preserves_the_category_streak() -> Result<()> { + let judge = Queue::new([ + r#"{"escalate":true,"category":"repetition","new_evidence":true,"reason":"repeated command"}"#, + "not json", + r#"{"escalate":true,"category":"repetition","new_evidence":true,"reason":"another repeated command"}"#, + ]); + let model = Queue::new(["efficient t1", "efficient t2", "efficient t3", "capable t3"]); + let router = escalation_router_with_confirmations(2)?; + let request = classify_session_request(); + + let (first, _) = test_drive( + router.clone(), + request.clone(), + queued(Arc::clone(&model), Arc::clone(&judge)), + ) + .await?; + let (second, _) = test_drive( + router.clone(), + request.clone(), + queued(Arc::clone(&model), Arc::clone(&judge)), + ) + .await?; + let (third, _) = test_drive(router, request, queued(model, judge)).await?; + + assert_eq!(first, "efficient"); + assert_eq!(second, "efficient"); + assert_eq!(third, "capable"); + Ok(()) + } + #[tokio::test] async fn stays_capable_after_latch() -> Result<()> { - let judge = Queue::new([r#"{"escalate":true,"reason":"stuck"}"#]); + let judge = Queue::new([ + r#"{"escalate":true,"category":"repetition","new_evidence":true,"reason":"stuck"}"#, + ]); let model = Queue::new(["efficient draft", "capable t1", "capable t2"]); let router = escalation_router()?; let request = classify_session_request(); diff --git a/crates/libsy/src/algorithms/util/escalation.rs b/crates/libsy/src/algorithms/util/escalation.rs index 7f5e3ae20..161c3c2f7 100644 --- a/crates/libsy/src/algorithms/util/escalation.rs +++ b/crates/libsy/src/algorithms/util/escalation.rs @@ -52,8 +52,9 @@ const MAX_REQUEST_CHARS: usize = 18_000; #[derive(Clone, Debug, Deserialize)] #[serde(default, deny_unknown_fields)] pub struct EscalationJudgeConfig { - /// Consecutive escalate verdicts required before a turn moves to the capable tier, which - /// is also the turn that latches the session. Any decline clears the streak. + /// Consecutive fresh-evidence verdicts in the same category required before a turn moves to + /// the capable tier, which is also the turn that latches the session. Any decline or stale + /// evidence clears the streak. /// `1` escalates on the first verdict; the router's main cost dial. /// `2` or higher needs a session id, since the streak is retained per session. pub confirmations: u32, @@ -93,15 +94,39 @@ impl Default for EscalationJudgeConfig { } } -/// The judge's verdict. The schema also requires a `reason`, which makes the judge state its -/// case and measurably sharpens the verdict. Routing reads only the boolean; the reason is -/// kept solely so an operator can see why the judge held or escalated when the -/// `switchyard_libsy::algorithms::util::escalation` target is enabled at `debug`. +/// Bounded trouble pattern used to correlate escalation confirmations across turns. +#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq)] +#[serde(rename_all = "snake_case")] +pub(crate) enum EscalationCategory { + None, + Repetition, + FalseProgress, + Drift, + Desperation, + CapabilityGap, +} + +impl EscalationCategory { + /// Stable state and telemetry label. + pub(crate) const fn label(self) -> &'static str { + match self { + Self::None => "none", + Self::Repetition => "repetition", + Self::FalseProgress => "false_progress", + Self::Drift => "drift", + Self::Desperation => "desperation", + Self::CapabilityGap => "capability_gap", + } + } +} + +/// The judge's typed verdict, including the evidence needed to confirm a stable pattern. #[derive(Deserialize)] pub(crate) struct EscalationVerdict { - escalate: bool, - #[serde(default)] - reason: String, + pub(crate) escalate: bool, + pub(crate) category: EscalationCategory, + pub(crate) new_evidence: bool, + pub(crate) reason: String, } /// Builds the condensed trajectory presented to the escalation judge. @@ -223,21 +248,89 @@ pub(crate) fn conversation_turn(request: &Request) -> usize { /// relies on. fn message_text(message: &Message) -> String { let mut parts = Vec::new(); - collect_text(&message.content, &mut parts); + let terminus_commands = if message.role == Role::Assistant { + message + .content + .iter() + .filter_map(|block| match block { + ContentBlock::ToolCall(call) if call.name == "bash_command" => call + .arguments + .get("keystrokes") + .and_then(|value| value.as_str()), + _ => None, + }) + .collect::>() + } else { + Vec::new() + }; + collect_text(&message.content, &mut parts, &terminus_commands); parts.join(" ") } +/// Removes a Terminus command batch when a structured bash call carries the same action. +/// +/// The model-facing request remains untouched. Only the judge's plain-text view is normalized, +/// so one action cannot look like two attempts while the agent still sees its native history. +fn without_duplicated_terminus_commands(text: &str, tool_commands: &[&str]) -> String { + for (start, character) in text.char_indices() { + if character != '{' { + continue; + } + + let mut values = + serde_json::Deserializer::from_str(&text[start..]).into_iter::(); + let Some(Ok(mut value)) = values.next() else { + continue; + }; + let end = start + values.byte_offset(); + let Some(commands) = value + .get("commands") + .and_then(|commands| commands.as_array()) + else { + continue; + }; + let Some(command_batch) = commands + .iter() + .map(|command| command.get("keystrokes").and_then(|value| value.as_str())) + .collect::>>() + else { + continue; + }; + let mut unmatched_tool_commands = tool_commands.to_vec(); + let fully_encoded = command_batch.iter().all(|command| { + let Some(index) = unmatched_tool_commands + .iter() + .position(|candidate| candidate == command) + else { + return false; + }; + unmatched_tool_commands.swap_remove(index); + true + }); + if command_batch.is_empty() || !fully_encoded { + continue; + } + + value["commands"] = serde_json::Value::Array(Vec::new()); + let Ok(normalized) = serde_json::to_string(&value) else { + continue; + }; + return format!("{}{}{}", &text[..start], normalized, &text[end..]); + } + text.to_string() +} + /// Appends the judge-relevant text of each block, descending into tool results. -fn collect_text(content: &[ContentBlock], parts: &mut Vec) { +fn collect_text(content: &[ContentBlock], parts: &mut Vec, tool_commands: &[&str]) { for block in content { match block { ContentBlock::Text { text } | ContentBlock::Refusal { text } => { - parts.push(text.clone()); + parts.push(without_duplicated_terminus_commands(text, tool_commands)); } ContentBlock::ToolCall(call) => { parts.push(format!("tool_call {}({})", call.name, call.arguments)); } - ContentBlock::ToolResult(result) => collect_text(&result.content, parts), + ContentBlock::ToolResult(result) => collect_text(&result.content, parts, &[]), _ => {} } } @@ -541,6 +634,147 @@ mod tests { assert_eq!(message_text(&result), "no such file"); } + #[test] + fn message_text_deduplicates_terminus_commands_for_the_judge() { + let first_command = "grep -n bug app.py\n"; + let second_command = "sed -n '1,80p' app.py\n"; + let message = Message { + role: Role::Assistant, + content: vec![ + ContentBlock::Text { + text: format!( + "Before\n```json\n{}\n```\nAfter", + json!({ + "analysis": "inspect the reported file", + "commands": [ + {"keystrokes": first_command, "duration": 0.1}, + {"keystrokes": second_command, "duration": 0.1}, + ], + }) + ), + }, + ContentBlock::ToolCall(ToolCall { + id: "call-1".to_string(), + name: "bash_command".to_string(), + arguments: json!({"keystrokes": first_command, "duration": 0.1}), + }), + ContentBlock::ToolCall(ToolCall { + id: "call-2".to_string(), + name: "bash_command".to_string(), + arguments: json!({"keystrokes": second_command, "duration": 0.1}), + }), + ], + }; + + let text = message_text(&message); + + assert!(text.contains("inspect the reported file"), "{text}"); + assert!(text.contains("Before"), "{text}"); + assert!(text.contains("After"), "{text}"); + assert!(text.contains(r#""commands":[]"#), "{text}"); + assert_eq!(text.matches("grep -n bug app.py").count(), 1, "{text}"); + assert_eq!(text.matches("sed -n '1,80p' app.py").count(), 1, "{text}"); + assert_eq!(text.matches("tool_call bash_command(").count(), 2, "{text}"); + } + + #[test] + fn message_text_keeps_terminus_commands_without_matching_tool_call() { + let command = "grep -n bug app.py\n"; + let text = json!({ + "analysis": "inspect the reported file", + "commands": [{"keystrokes": command}], + }) + .to_string(); + let without_tool_call = Message { + role: Role::Assistant, + content: vec![ContentBlock::Text { text: text.clone() }], + }; + let mismatched_tool_call = Message { + role: Role::Assistant, + content: vec![ + ContentBlock::Text { text }, + ContentBlock::ToolCall(ToolCall { + id: "call-1".to_string(), + name: "bash_command".to_string(), + arguments: json!({"keystrokes": "sed -n '1,20p' app.py\n", "duration": 0.1}), + }), + ], + }; + + assert_eq!( + message_text(&without_tool_call) + .matches("grep -n bug app.py") + .count(), + 1 + ); + assert_eq!( + message_text(&mismatched_tool_call) + .matches("grep -n bug app.py") + .count(), + 1 + ); + } + + #[test] + fn message_text_keeps_a_partially_encoded_terminus_batch() { + let first_command = "grep -n bug app.py\n"; + let second_command = "sed -n '1,80p' app.py\n"; + let message = Message { + role: Role::Assistant, + content: vec![ + ContentBlock::Text { + text: json!({ + "commands": [ + {"keystrokes": first_command}, + {"keystrokes": second_command}, + ], + }) + .to_string(), + }, + ContentBlock::ToolCall(ToolCall { + id: "call-1".to_string(), + name: "bash_command".to_string(), + arguments: json!({"keystrokes": first_command}), + }), + ], + }; + + let text = message_text(&message); + + assert_eq!(text.matches("grep -n bug app.py").count(), 2, "{text}"); + assert_eq!(text.matches("sed -n '1,80p' app.py").count(), 1, "{text}"); + assert!(!text.contains(r#""commands":[]"#), "{text}"); + } + + #[test] + fn message_text_keeps_duplicate_commands_without_one_tool_call_each() { + let command = "grep -n bug app.py\n"; + let message = Message { + role: Role::Assistant, + content: vec![ + ContentBlock::Text { + text: json!({ + "commands": [ + {"keystrokes": command}, + {"keystrokes": command}, + ], + }) + .to_string(), + }, + ContentBlock::ToolCall(ToolCall { + id: "call-1".to_string(), + name: "bash_command".to_string(), + arguments: json!({"keystrokes": command}), + }), + ], + }; + + let text = message_text(&message); + + assert_eq!(text.matches("grep -n bug app.py").count(), 3, "{text}"); + assert!(!text.contains(r#""commands":[]"#), "{text}"); + } + #[test] fn truncate_middle_keeps_head_and_tail() { let text = "a".repeat(40) + &"z".repeat(40); diff --git a/crates/libsy/src/algorithms/util/llm_judge.rs b/crates/libsy/src/algorithms/util/llm_judge.rs index 4798178e9..cb8277dae 100644 --- a/crates/libsy/src/algorithms/util/llm_judge.rs +++ b/crates/libsy/src/algorithms/util/llm_judge.rs @@ -261,7 +261,7 @@ where /// mid-stream, or unparseable reply — is logged and folded into `None` for the policy's /// fallback branch. A closed driver stream is folded too; the algorithm's next driver /// call surfaces it, so nothing is masked. - async fn verdict( + pub(crate) async fn verdict( &self, state: &mut State, request: &Request, diff --git a/crates/libsy/src/prompts/escalation/prompt.md b/crates/libsy/src/prompts/escalation/prompt.md index 6d57369bb..d29a4017f 100644 --- a/crates/libsy/src/prompts/escalation/prompt.md +++ b/crates/libsy/src/prompts/escalation/prompt.md @@ -9,11 +9,20 @@ messages and tool results). Judge the *trajectory* — is the agent making real progress toward the stated task — not the difficulty of the task itself. Return exactly one JSON object: -{"escalate": boolean, "reason": "one short sentence naming the pattern"} +{"escalate": boolean, "category": "none|repetition|false_progress|drift|desperation|capability_gap", "new_evidence": boolean, "reason": "one short sentence naming the pattern"} + +Use `category: "none"` whenever `escalate` is false. `new_evidence` +is true only when the NEWEST assistant turn or its resulting tool output +adds evidence for the named pattern. Older turns may establish context, +but do not repeat an escalation vote solely because old trouble remains +visible in the rolling transcript. If the newest turn recovered, adapted, +or made progress, return `escalate: false`, `category: "none"`, and +`new_evidence: false`. Escalation is one-way for the rest of the task and expensive. Escalate only on a clear PATTERN of trouble, never on a single failed command. -When the evidence is thin or ambiguous, return {"escalate": false}. +When the evidence is thin or ambiguous, decline with `category: "none"` +and `new_evidence: false`. The bar is not "is there friction" — agentic coding is full of friction the efficient tier works through on its own. The bar is "is this run @@ -58,8 +67,10 @@ Hold weak — no model can fix these, so escalation is pure waste: # Trouble patterns — escalate when you see these Repetition and loops (the most common way agent runs die): -- The same command or edit failing 2+ times with materially the same - error, especially with unrelated changes in between. +- The same command or edit failing across 2+ DISTINCT assistant turns + with materially the same error, especially with unrelated changes in + between. Count executed attempts across turns, not repeated renderings + inside one message. - Near-identical tool calls repeated, or the same files re-read, without new information gained — including longer cycles (A -> B -> C -> A). - Fighting the environment: repeatedly invoking a missing executable, @@ -105,6 +116,15 @@ Desperation: # Expected friction — do NOT escalate on these Agentic coding is full of failures that are part of healthy work: +- A command appearing both in the assistant's JSON/text and as one or + more structured tool-call blocks in that SAME turn. Agent harnesses + commonly serialize one intended action more than once; this is one + attempt, not a loop. Repetition is evidence only when separate turns + show separate executions with materially the same failed result. +- Terminal-input serialization trouble (for example tabs triggering + completion or a heredoc being mangled) when the next turn changes the + write mechanism, quoting, or transport. That is adaptation, not + repeated failure. - A test written to fail first (TDD) or a bug being reproduced on purpose. - A compile, lint, or test error fixed or meaningfully acted on in the @@ -130,49 +150,56 @@ Agentic coding is full of failures that are part of healthy work: - A long-running command (build, install, test suite) that simply has not finished, or the agent waiting on information it asked for. -The distinguishing question: is each failure producing new information -that changes the next action? Failing forward is fine; failing in place -is trouble. Also weigh the session's own recovery record: if this same -session already shows friction the agent subsequently cleared (a failure -followed by a verified fix or passing check), lean toward holding — a -session that has recovered before will usually recover again. +The distinguishing question: across DISTINCT assistant turns, is each +failure producing new information that changes the next action? Failing +forward is fine; failing in place is trouble. Never infer a multi-turn +pattern from duplicated representations inside one turn. Also weigh the +session's own recovery record: if this same session already shows +friction the agent subsequently cleared (a failure followed by a verified +fix or passing check), lean toward holding — a session that has recovered +before will usually recover again. # Worked examples (none drawn from any benchmark task set) * Turn 3; the agent ran the test suite, 4 tests fail, and it is now - reading the first failing test. -> {"escalate": false} — reproducing - failures is the job. + reading the first failing test. -> {"escalate": false, "category": + "none", "new_evidence": false, "reason": "working through the first + reproduced failure"} * The agent has run `pytest tests/test_api.py` 4 times with the same ImportError, editing an unrelated config file between attempts. -> - {"escalate": true, "reason": "same ImportError 4 times while editing - unrelated files"} + {"escalate": true, "category": "repetition", "new_evidence": true, + "reason": "same ImportError 4 times while editing unrelated files"} * `conda` is not installed; the agent has tried `conda install` five ways instead of using the `pip` that earlier output showed present. - -> {"escalate": true, "reason": "fighting missing executable instead - of adapting"} + -> {"escalate": true, "category": "repetition", "new_evidence": + true, "reason": "fighting missing executable instead of adapting"} * Task: "make the provided integration tests pass." Recent turns: renaming variables and reformatting docstrings; tests not run in 8 - turns. -> {"escalate": true, "reason": "drifted to cosmetic edits, - verification abandoned"} + turns. -> {"escalate": true, "category": "drift", "new_evidence": + true, "reason": "drifted to cosmetic edits, verification abandoned"} * The agent says "All tests pass, task complete" but the last visible test output shows "2 failed, 11 passed". -> {"escalate": true, - "reason": "claims success contradicted by latest test output"} + "category": "false_progress", "new_evidence": true, "reason": + "claims success contradicted by latest test output"} * The agent wrote a reproduction script that exits 0 without invoking the code path the issue describes, concluded "bug not reproducible", - and is wrapping up. -> {"escalate": true, "reason": "reproduction - never exercised the reported code path"} + and is wrapping up. -> {"escalate": true, "category": + "false_progress", "new_evidence": true, "reason": "reproduction never + exercised the reported code path"} * Two turns of edits, one failed build, then a fixed build and a - passing test. -> {"escalate": false} + passing test. -> {"escalate": false, "category": "none", + "new_evidence": false, "reason": "latest build recovered and passed"} * `npm install` has been running for one turn with no output yet. -> - {"escalate": false} — slow command, not a stall. + {"escalate": false, "category": "none", "new_evidence": false, + "reason": "command is still running"} * Four different serialization libraries failed to import; the agent is now writing the converter with a fifth approach it has not tried - before. -> {"escalate": false} — sequential alternatives are - adaptation, even when none has succeeded yet. + before. -> {"escalate": false, "category": "none", "new_evidence": + false, "reason": "latest turn changed approach"} * Task: tune a slow batch pipeline. The agent is investigating why the message broker fails to start, since the pipeline cannot be - measured without it. -> {"escalate": false} — unblocking - verification serves the task. + measured without it. -> {"escalate": false, "category": "none", + "new_evidence": false, "reason": "working to unblock verification"} Do not emit markdown, commentary, or chain-of-thought — only the JSON object. diff --git a/crates/libsy/src/prompts/escalation/schema.json b/crates/libsy/src/prompts/escalation/schema.json index 977c41882..e98b3c5fd 100644 --- a/crates/libsy/src/prompts/escalation/schema.json +++ b/crates/libsy/src/prompts/escalation/schema.json @@ -10,12 +10,21 @@ "type": "boolean", "description": "True when the run is likely doomed without escalation to the strong tier." }, + "category": { + "type": "string", + "enum": ["none", "repetition", "false_progress", "drift", "desperation", "capability_gap"], + "description": "The single trouble pattern supporting escalation, or none when escalation is false." + }, + "new_evidence": { + "type": "boolean", + "description": "True only when the newest assistant turn or its resulting tool output adds evidence for the named trouble pattern." + }, "reason": { "type": "string", "description": "One short sentence naming the trouble pattern, or stating why the run is progressing." } }, - "required": ["escalate", "reason"], + "required": ["escalate", "category", "new_evidence", "reason"], "additionalProperties": false } } diff --git a/crates/switchyard-server/tests/server.rs b/crates/switchyard-server/tests/server.rs index 29bb561e7..e051ef310 100644 --- a/crates/switchyard-server/tests/server.rs +++ b/crates/switchyard-server/tests/server.rs @@ -409,7 +409,8 @@ async fn upstream_chat( .pointer("/response_format/json_schema/schema/properties/escalate") .is_some() { - r#"{"escalate":false,"reason":"making progress"}"#.to_string() + r#"{"escalate":false,"category":"none","new_evidence":false,"reason":"making progress"}"# + .to_string() } else if model == "model/classifier" && requests_schema_invalid_verdict { r#"{"crux":"bounded task","primary_rule":"SUP-1","capability_boundary":"supported","p_solve":0.1,"unexpected":true}"#.to_string() } else if model == "model/classifier" { diff --git a/docs/reference/toml_schema.md b/docs/reference/toml_schema.md index 4be77ed86..4516e8b0c 100644 --- a/docs/reference/toml_schema.md +++ b/docs/reference/toml_schema.md @@ -240,7 +240,7 @@ Escalation mode serves the weak target first and judges the completed turn. See | `strong_target` | Yes | — | Target used after the session latches. | | `weak_target` | Yes | — | Target served before the latch. | | `prompt` | No | packaged prompt | Replaces the trajectory-judge prompt. | -| `escalation.confirmations` | No | `2` | Consecutive escalate verdicts required to latch. Above `1` needs a session ID. | +| `escalation.confirmations` | No | `2` | Consecutive fresh-evidence verdicts for the same failure category required to latch. Above `1` needs a session ID. | | `escalation.recent_turn_window` | No | `28` | Trailing messages shown to the judge. | | `escalation.window_message_chars` | No | `500` | Per-message cap inside that window. | diff --git a/docs/routing_algorithms/escalation_router_routing.md b/docs/routing_algorithms/escalation_router_routing.md index 3fecddab6..608b179ba 100644 --- a/docs/routing_algorithms/escalation_router_routing.md +++ b/docs/routing_algorithms/escalation_router_routing.md @@ -50,7 +50,9 @@ The route-level `prompt` key replaces the packaged trajectory-judge prompt. It uses the escalation verdict schema rather than the capability verdict schema. Switchyard supplies that schema according to the route's `response_format_type`: through the structured-output request in the default `json_schema` mode, or in -the prompt in `json_object` mode. +the prompt in `json_object` mode. The verdict includes an `escalate` decision, a +bounded failure `category`, whether the newest turn adds `new_evidence`, and a +short `reason`. ## How the decision works @@ -60,8 +62,10 @@ For each turn on an unlatched session, Switchyard: 2. Appends that reply to the transcript and asks the judge to rule on the completed turn. The judge therefore rates work the weak model actually did, not a prediction about work it might do. -3. Increments a consecutive-escalate streak on an escalate verdict, and resets it - to zero on a decline. +3. Increments a confirmation streak only when an escalate verdict cites fresh + evidence for the same failure category as the preceding vote. A different + category starts a new streak at one; a decline or stale evidence resets it to + zero. 4. Serves the buffered weak reply when the streak has not yet reached `confirmations` — so a judged turn that does not escalate costs one weak call plus one judge call, and no strong call. @@ -106,7 +110,7 @@ configuration, so a bare `escalation = {}` is a valid, tuned route: | Key | Default | Meaning | |---|---|---| -| `confirmations` | `2` | Consecutive escalate verdicts required before the session latches to strong. Must be at least `1`. | +| `confirmations` | `2` | Consecutive fresh-evidence verdicts for the same failure category required before the session latches to strong. Must be at least `1`. | | `recent_turn_window` | `28` | Trailing messages shown to the judge on top of the anchors. Must be at least `1`. | | `window_message_chars` | `500` | Per-message truncation cap inside that trailing window. Must be at least `50`. | @@ -116,8 +120,8 @@ retained per session — without one, every turn starts from zero and the route never latches. Clients supply it with `x-switchyard-session-id`. Anchor and transcript caps remain fixed. Set the route-level -`max_output_tokens` key to change the judge's reply budget. Any decline still -resets the streak to zero. +`max_output_tokens` key to change the judge's reply budget. Any decline or +verdict without new evidence resets the streak to zero. ## Run the route @@ -162,6 +166,10 @@ per-session routing stats under the judge's model id, tagged with the `classifier` tier — so per-session token accounting includes judge overhead alongside the tiers the session was served by. +The server log records each parsed escalation verdict's category, +`new_evidence` flag, and bounded reason so false-positive or missed escalation +decisions can be diagnosed without retaining unbounded judge output. + ## When not to use escalation routing - **One-shot requests.** No trajectory to judge. Use From c6f2dfc0469bd273c6b58ec6336ee26712add8cc Mon Sep 17 00:00:00 2001 From: Alex Steiner Date: Sat, 5 Sep 2026 09:17:52 -0700 Subject: [PATCH 2/2] docs(benchmark): report paired SWE-Atlas results Signed-off-by: Alex Steiner --- benchmark/SWE_ATLAS_ESCALATION_REPORT.md | 150 +++++++++++++++++++++-- 1 file changed, 143 insertions(+), 7 deletions(-) diff --git a/benchmark/SWE_ATLAS_ESCALATION_REPORT.md b/benchmark/SWE_ATLAS_ESCALATION_REPORT.md index f79b447ad..d45e38730 100644 --- a/benchmark/SWE_ATLAS_ESCALATION_REPORT.md +++ b/benchmark/SWE_ATLAS_ESCALATION_REPORT.md @@ -11,6 +11,13 @@ This change makes escalation depend on fresh, consistent failure evidence across the existing one-way switch and latch: a session can move from the efficient model to the capable model at most once and remains there afterward. +On a deterministic 40-task paired sample (20 SWE-Atlas RF and 20 SWE-Atlas TW), the patched router +matched the two single-model arms at 15/40 correct and improved over the original router's 10/40. +It switched 5 times instead of 9. Under the synthetic token prices defined below, it cost +$0.889/task versus $1.171/task for the original router. GLM-only also scored 15/40 and remained the +least expensive arm at $0.501/task, so the result supports the patch over the original router but +does not show that routing beats the best fixed model on this sample. + ## Changes - Normalize only the transcript sent to the escalation judge. When a raw Terminus command batch @@ -26,7 +33,7 @@ model at most once and remains there afterward. adaptive change of terminal strategy are not repeated failed attempts. - Preserve the existing maximum-one-switch and post-switch latch behavior. -## Live SWE-Atlas observations +## Focused SWE-Atlas reproducer observations The live checks used GLM 5.2 as the efficient model and Opus 4.8 as the capable model. The reward column is the task's binary verifier result. "GLM only" means that the patched router did not @@ -49,6 +56,131 @@ switching, confirmation, and latching, but does not establish a causal accuracy Similarly, the unsuccessful controls show improved escalation precision and avoided Opus calls; they do not show that the efficient model could solve those tasks. +## Paired benchmark + +### Design + +The paired benchmark used the following fixed setup: + +- 20 tasks from `scale-ai/swe-atlas-rf@1` and 20 from `scale-ai/swe-atlas-tw@1`. + Tasks were selected deterministically by ranking the hash of + `20260904::`; they were not selected by outcome. +- Four arms per task: direct GLM, direct Opus, the original escalation router, and the patched + escalation router. This produced 160 task-arm runs. +- GLM was `nvidia/zai-org/glm-5.2`; Opus was + `aws/anthropic/bedrock-claude-opus-4-8` with medium effort. GLM also served as the escalation + judge. The two escalation arms used two confirmations, maximum one switch, and a post-switch + latch. +- The original router was built from upstream commit + `9a743e89223a0d5b14011f1226d5b068f730a3b8`; the patched router was built from + `507376b1d851b825378cf147d09a8b5594773298`. +- There was no agent-turn limit. Worker requests allowed up to 128,000 output tokens and a + 900-second model-server timeout. Five initial runs reached Harbor's separate 3,600-second agent + timeout. Only those five task-arm runs were repeated with a 3x agent-timeout multiplier; all five + retries completed and replaced the invalid attempts. The final effective matrix is 160/160 valid. +- Each table entry is one trial per task and arm. Binary verifier reward is reported as correct. + Because model sampling is stochastic, differences between unswitched GLM router runs and direct + GLM runs are not necessarily routing effects. + +### Accuracy and routing + +| Dataset | Arm | Correct | Accuracy | Switches | Correct after switch | +| --- | --- | ---: | ---: | ---: | ---: | +| RF (20) | Direct GLM | 7 | 0.350 | -- | -- | +| RF (20) | Always Opus | 9 | 0.450 | -- | -- | +| RF (20) | Original escalation | 6 | 0.300 | 7 | 2 | +| RF (20) | **Patched escalation** | **9** | **0.450** | **4** | **2** | +| TW (20) | Direct GLM | 8 | 0.400 | -- | -- | +| TW (20) | Always Opus | 6 | 0.300 | -- | -- | +| TW (20) | Original escalation | 4 | 0.200 | 2 | 0 | +| TW (20) | **Patched escalation** | **6** | **0.300** | **1** | **0** | +| Combined (40) | Direct GLM | 15 | 0.375 | -- | -- | +| Combined (40) | Always Opus | 15 | 0.375 | -- | -- | +| Combined (40) | Original escalation | 10 | 0.250 | 9 | 2 | +| Combined (40) | **Patched escalation** | **15** | **0.375** | **5** | **2** | + +Against the original router on the same 40 tasks, the patched router won 7 outcomes, lost 2, and +tied 31. Against direct GLM it won 5 and lost 5; against direct Opus it won 6 and lost 6. All 14 +switched trajectories across the two router arms had exactly one GLM-to-Opus transition and no +hand-back, confirming the switch and latch invariants in live execution. + +The patch traded earlier sensitivity for precision. Original-router switches occurred after a +median of 23 GLM calls; patched-router switches occurred after a median of 51. One patched RF retry +did not switch until 141 GLM calls, then made 57 latched Opus calls and still scored 0. Direct Opus +and the original router also scored 0 on that task. This avoided a premature switch but exposed a +cost problem: judging every turn can be expensive when the destination model is unlikely to help. + +### Synthetic cost calculation + +The inference service did not provide billable cost. The values below are synthetic estimates for +comparing routing behavior, not NVIDIA prices or charges. The assumed rates are: + +| Model | Uncached input / 1M tokens | Cached input / 1M tokens | Output / 1M tokens | +| --- | ---: | ---: | ---: | +| GLM 5.2 | $0.50 | $0.05 | $2.00 | +| Opus 4.8 | $5.00 | $0.50 | $25.00 | + +For each model in each trajectory: + +```text +uncached_input_tokens = max(prompt_tokens - cached_tokens, 0) + +model_cost = ( + uncached_input_tokens * uncached_input_rate + + cached_tokens * cached_input_rate + + output_tokens * output_rate +) / 1,000,000 + +trajectory_cost = sum(worker_model_costs) + sum(escalation_judge_costs) +``` + +Cache-creation tokens are part of the non-cached prompt-token remainder and therefore receive the +uncached-input rate. The calculation includes GLM judge calls for the two router arms. It excludes +the Harbor verifier, cluster resources, and other infrastructure because those calls are outside +Switchyard's per-trajectory statistics. + +| Dataset | Arm | Synthetic cost | Cost/task | Cost/correct | +| --- | --- | ---: | ---: | ---: | +| RF (20) | Direct GLM | $12.953 | $0.648 | $1.850 | +| RF (20) | Always Opus | $32.196 | $1.610 | $3.577 | +| RF (20) | Original escalation | $35.169 | $1.758 | $5.862 | +| RF (20) | **Patched escalation** | **$27.200** | **$1.360** | **$3.022** | +| TW (20) | Direct GLM | $7.093 | $0.355 | $0.887 | +| TW (20) | Always Opus | $14.829 | $0.741 | $2.471 | +| TW (20) | Original escalation | $11.657 | $0.583 | $2.914 | +| TW (20) | **Patched escalation** | **$8.356** | **$0.418** | **$1.393** | +| Combined (40) | Direct GLM | $20.045 | $0.501 | $1.336 | +| Combined (40) | Always Opus | $47.025 | $1.176 | $3.135 | +| Combined (40) | Original escalation | $46.826 | $1.171 | $4.683 | +| Combined (40) | **Patched escalation** | **$35.556** | **$0.889** | **$2.370** | + +The patched router was 24.1% cheaper than the original router and produced five additional correct +outcomes. It was 24.4% cheaper than always-Opus at the same aggregate accuracy. It was 77.4% more +expensive than direct GLM at the same aggregate accuracy. Of the patched router's $35.556 synthetic +total, $3.698 (10.4%) came from judge calls; this motivates reducing judge cadence after repeated +negative verdicts. + +### TW model-order reversal + +TW demonstrates a limitation that this patch does not solve: the escalation route assumes its +configured Opus target has positive expected gain over GLM. On this sample, direct GLM scored 8/20 +while direct Opus scored 6/20, so that assumption is not valid at the workload level. + +The patched router switched on only one TW task. Direct GLM, direct Opus, and both router arms all +scored 0 on that task, so the switch added cost without changing its outcome. The patched router's +two-correct deficit versus direct GLM was not caused by switching: it lost three independently +sampled GLM-only outcomes and gained one different GLM-only outcome. The original router provides +a clearer harmful-switch example on task `6902ef3ab97fe23e2ad2727e`: direct GLM scored 1, direct Opus +scored 0, and the original router switched and scored 0. The patch avoided that switch, although its +independent GLM-only run also scored 0. + +The next routing change should therefore be separate from the trajectory-evidence fix: gate +escalation on a workload- or task-conditioned estimate of expected accuracy gain and cost. When +recent paired evidence says the nominal strong model is worse, the route should disable or reverse +that transition rather than asking only whether the current trajectory looks stuck. This requires +held-out calibration or online exploration; the router cannot infer relative model quality from one +model's failing trajectory alone. + ## Validation evidence The implementation was validated with Rust 1.96.1 on a Slurm compute node using: @@ -64,7 +196,8 @@ uv run pytest tests/ -v ``` The validation completed successfully: the Rust workspace passed, including 289 `libsy` tests; -ruff and mypy passed; the native wheel built and installed; and pytest reported 117 passed tests. +ruff and mypy passed; the native wheel built and installed; and pytest reported 115 passed, +2 deselected, and 2 subtests passed. Focused regression coverage includes exact and multi-command transcript normalization, partial and multiplicity mismatches, category changes, stale evidence, unavailable-judge behavior, and the one-way latch. @@ -74,8 +207,11 @@ described above were separate, explicitly configured benchmark runs. ## Conclusion -The evidence supports a narrower conclusion than an aggregate benchmark improvement: the router -now distinguishes repeated cross-turn failure from within-turn serialization noise, avoids several -observed unnecessary escalations, and still performs a confirmed one-time escalation when fresh -same-category evidence persists. A larger paired run is required to estimate accuracy and cost -effects with confidence. +The patch fixes the concrete duplicate-serialization false positive and materially improves the +tested router: it matches the single-model aggregate accuracy, gains 5 correct outcomes over the +original router, cuts switches from 9 to 5, and reduces synthetic cost by 24.1%. It does not make +the router the best cost/accuracy arm; direct GLM matches its aggregate accuracy at substantially +lower cost. The remaining work is not to loosen the new evidence rule globally. It is to add +model-order calibration for workloads such as TW and reduce judge overhead or stop judging when +escalation has low expected value. Replicated trials on held-out tasks are needed before treating +the point estimates as stable production gains.