feat: work in the background, and ask before taking the mouse - #547
Merged
Merged
Conversation
…icon settings The shared surface the background-operation work builds on: background_mode (on by default), mouse_control (ask or always) with its prompt-dismissed flag, and dock_icon_visible for the menu-bar-only mode. Claude-Session: https://claude.ai/code/session_01XsDCBs2QS5BsYaZ5o1Tdo9
The contract the Rust side emits and the UI answers: an input-control request, the state while Juno drives the cursor, the offer to stop asking, and the reopen-attempt notice for menu-bar mode. Claude-Session: https://claude.ai/code/session_01XsDCBs2QS5BsYaZ5o1Tdo9
The agent now reaches other apps through accessibility actions and events posted straight to the target process, so a person can keep typing while Juno works. The physical cursor is the last resort, and it is never taken silently: the tool layer asks first and says so while it has the mouse. What changed: - Deleted ensure_main_window_focus in commands/mouse.rs and its four callers. Juno focusing its own window before injecting a click is what made the agent steal focus; right_click and mouse_move never did it and worked fine. - Wired the two Phase 4 pieces that had no callers. activate_without_raise now runs before keystrokes are posted to a background process, which is the difference between clicks working in the background and the agent working in the background. get_ax_observer_check_remote now registers a remote observation per target process so Chromium and Electron do not suspend their accessibility tree while their windows are occluded. - Added no-warp variants for every remaining mutating action: triple and middle click, drag, mouse down and up, scroll, key, hold key, release key and type. Each takes allow_physical; with it false they return Ok(None) rather than quietly dropping to the HID tap. In background mode a mouse_move updates Juno's virtual cursor only and leaves the real pointer where the user put it. - click_auto sent every browser straight to cursor-warping simulation. Browsers now go through the process-targeted path with the existing Chromium activation primer. - Reading an accessibility tree no longer activates the app being read, and the app-wide Desktop is built background-friendly: menu-bar apps visible, target activation off. Same for the headless, CLI, element and accessibility-tool engines. - focus(element) sets AXFocusedUIElement without AXRaise in background mode, and reports a miss instead of falling back to a coordinate click. - open_application no longer shells out to `open -a` to bring an already-running app forward. - All of it is gated on the background_mode setting, which stays the single source of truth; the input layer keeps a cache that is dropped whenever agent settings change. The consent contract, for the UI to build on: InputTier plus InputOutcome say which machinery carried an action. When only the physical cursor will do, "always" proceeds and emits input-control-state, "ask" emits input-control-request and waits for respond_to_input_control, and a refusal comes back to the agent as a recoverable error that names another route. After a granted takeover, input-control-offer fires once per run. Claude-Session: https://claude.ai/code/session_01XsDCBs2QS5BsYaZ5o1Tdo9
`activate_without_raise` redirects the WindowServer's input focus to the target process, and nothing put it back. So after the agent typed into a backgrounded Chrome, the user's next keystrokes landed in Chrome: no window raised, no cursor moved, the characters just went to the wrong app. That is worse than the behaviour this feature replaced, because it is silent. Input focus is now held by an RAII guard, BackgroundKeyboardFocus, for exactly as long as the keystrokes are being posted. It captures the frontmost app before redirecting, and on drop reads the frontmost app again: unchanged means the user never moved, so focus goes back where it came from; changed means they switched apps mid-flight, so focus follows them to the app they are looking at now. A redirect that failed restores nothing, and a target that was already frontmost costs nothing in either direction. Drop rather than an explicit call, so an error or a panic unwind cannot strand the keyboard. The guard is bound at all five keyboard sites: press, hold, release, the element-level Cmd+V fallback and the process-targeted paste, which also covers type_text_no_warp through paste_text. The restore reads as redundant until you know SLPS focus redirection is not symmetric with ordinary activation, so the guard says so in a comment. Claude-Session: https://claude.ai/code/session_01XsDCBs2QS5BsYaZ5o1Tdo9
The restore rule was 'give focus to whoever is frontmost now', which is right when the user switches apps mid-run. But if the SLPS redirect is what made the target frontmost, that rule returns focus to the agent's target, which is the one outcome the guard exists to prevent. The decision now takes the target pid and can never return it, so it is correct whether or not the redirect perturbs frontmostApplication. The old invariant test looped over three frontmost values and omitted the target, so it could not catch this.
Juno can act on other apps without taking the pointer. This is the half of that the person sees. Settings (Advanced > Background): "Work in the background", a two-option "Mouse control" picker (Ask each time / Always allow), and "Show in Dock" whose description carries the whole way back out of menu-bar-only mode. All three load together, share one loading and one error state, revert on a failed write, and are findable through settings search. Consent: input-control-request renders an inline prompt under the conversation in both the main window and the bar's pane, built from the existing Confirmation primitives. It names what Juno wants and the app it wants it in, offers Allow once / Always allow / Not now, never takes focus, and reopens a dismissed bar pane so the question is on screen. Driving indicator: while input-control-state is active the floating bar goes full width with a system-blue dot and "Juno is using the mouse in <app>", and clears the moment the pointer comes back. Stop asking: input-control-offer shows one calm offer per session, and only while mouse control is still "ask" and the offer was not dismissed. Menu-bar mode: get_dock_icon_visible / set_dock_icon_visible persist through SettingsManager and apply macOS ActivationPolicy at runtime and at startup. A reopen with no Dock icon counts attempts in a rolling minute, brings the main window up, emits app-reopen-attempt and shows a menu-bar hint; three in a minute escalates to "Show Juno in the Dock again" and "Quit Juno". Tests: 47 new vitest cases (consent decisions, offer gating, reopen escalation, settings rows, bar driving state) plus 4 Rust unit tests for the attempt-window counting.
…-mode * origin/feat/bg-ui: feat(background): consent, driving indicator and menu-bar recovery UI # Conflicts: # src-tauri/src/lib.rs
…n screen A run started by the scheduler or the cloud has no conversation open, so the in-app consent prompt has nowhere to appear and the agent would wait out the whole timeout asking nobody. The request now also raises a system notification naming the app and what Juno wants to do.
| useEventListener(EVENTS.INPUT_CONTROL_OFFER, () => { | ||
| if (offeredThisSession.current) return; | ||
| void (async () => { | ||
| let mode: MouseControlMode = "ask"; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Juno acts on other apps without taking your cursor or your active window, so you can keep typing while it works. When a step genuinely cannot be done quietly, it asks first.
Why this was mostly wiring
The focus-free input stack was already written and merged: SkyLight
SLEventPostToPidfalling back toCGEventPostToPid, plus accessibility actions that work on background windows. It was connected to four actions (left, right and double click, and AX typing) and nothing else.activate_without_raise, the call that routes keystrokes to a background app, and the AX-observer call that stops Chromium and Electron suspending their accessibility tree when a window goes behind, both sat in the tree with zero callers.The ten blockers, fixed
ensure_main_window_focusactivated Juno itself immediately before injecting a click, on four of the click commands. Deleted, with its callers.right_clickandmouse_movenever called it, which was the proof it was unnecessary.activate_without_raisenow runs on the keyboard paths,get_ax_observer_check_remotekeeps Chromium and Electron accessibility trees awake while backgrounded.click_autohard-coded every browser to cursor-warping simulation. Browsers now try the element path first and reuse the existing Chromium activation primer.focus(element)no longer raises a window or falls back to a coordinate click in background mode.open_applicationno longer shellsopen -apurely to focus an app that is already running.background_mode, default on. With it off, behaviour is exactly what it is today.accessibility,process_targeted,physical_cursor), andOk(None)means "this needs your mouse".mouse_movein background mode drives only the virtual cursor and never the real pointer.Asking for the mouse
When a step needs the physical cursor and mouse control is "ask", Juno stops and asks in the conversation: what it wants to do, in which app, and that your cursor gets bigger while it works. Allow once, Always allow, Not now. Denial and the 60 second timeout both return a recoverable error naming another route, so the agent can carry on rather than stall. A run with no conversation on screen, from the scheduler or the cloud, also raises a system notification, or it would be asking nobody.
While Juno has the mouse: the system cursor enlarges (as it already did), and the bar shows a blue status dot, a hairline, and "Juno is using the mouse in Safari". An RAII guard raises and clears that state, so a panic cannot strand the indicator on.
After the first granted takeover, one offer, once per session: "Let Juno use the mouse without asking?" Yes, No, Don't ask again.
Settings, under Advanced
Getting back a hidden app
Turning off the Dock icon toasts where Juno went. If someone later opens Juno from Applications while it is running, macOS sends a reopen: the first one or two say "Juno is already running, it lives in the menu bar", and three within a minute, which is what being stuck looks like, offers the two things that actually help: put the Dock icon back, or quit.
Fixed in review, worth knowing
Verified
cargo fmt,cargo clippy -D warningson both crates, 519 Rust tests, 238 frontend tests,bunx tsc --noEmit. New unit tests cover tier selection, the consent state machine (an unknown decision string is a refusal, never consent), background-mode gating, the focus-restore rules, and the reopen escalation window.Needs a hands-on pass on hardware
Three things cannot be proven without a running app, and all three degrade to today's behaviour rather than breaking: that SkyLight really routes keystrokes into a backgrounded Chrome or VS Code, that the private AX-observer registration really stops Chromium suspending its tree, and that a process-targeted scroll lands in the right view. The specific check: type into a backgrounded Chrome while an editor is focused, then confirm your next keystroke still lands in the editor.