Skip to content

feat: work in the background, and ask before taking the mouse - #547

Merged
lacymorrow merged 9 commits into
mainfrom
feat/background-mode
Sep 15, 2026
Merged

lacymorrow merged 9 commits into
mainfrom
feat/background-mode

Conversation

@lacymorrow

Copy link
Copy Markdown
Owner

Juno acts on other apps without taking your cursor or your active window, so you can keep typing while it works. When a step genuinely cannot be done quietly, it asks first.

Why this was mostly wiring

The focus-free input stack was already written and merged: SkyLight SLEventPostToPid falling back to CGEventPostToPid, plus accessibility actions that work on background windows. It was connected to four actions (left, right and double click, and AX typing) and nothing else. activate_without_raise, the call that routes keystrokes to a background app, and the AX-observer call that stops Chromium and Electron suspending their accessibility tree when a window goes behind, both sat in the tree with zero callers.

The ten blockers, fixed

  1. ensure_main_window_focus activated Juno itself immediately before injecting a click, on four of the click commands. Deleted, with its callers. right_click and mouse_move never called it, which was the proof it was unnecessary.
  2. Phase 4 wired: activate_without_raise now runs on the keyboard paths, get_ax_observer_check_remote keeps Chromium and Electron accessibility trees awake while backgrounded.
  3. Background variants for everything that was left: middle click, triple click, drag, mouse down and up, key, hold key, release key, type, and scroll. Scroll used to warp the real cursor to the target first; it now posts to the process with the location set on the event.
  4. click_auto hard-coded every browser to cursor-warping simulation. Browsers now try the element path first and reuse the existing Chromium activation primer.
  5. Reading a window's accessibility tree no longer activates the app it belongs to. The app-wide desktop engine now sees background and menu-bar apps and never activates.
  6. focus(element) no longer raises a window or falls back to a coordinate click in background mode.
  7. open_application no longer shells open -a purely to focus an app that is already running.
  8. All of it gates on background_mode, default on. With it off, behaviour is exactly what it is today.
  9. Falling back to the physical cursor is now an explicit outcome, not a silent slide. Every action returns its tier (accessibility, process_targeted, physical_cursor), and Ok(None) means "this needs your mouse".
  10. mouse_move in background mode drives only the virtual cursor and never the real pointer.

Asking for the mouse

When a step needs the physical cursor and mouse control is "ask", Juno stops and asks in the conversation: what it wants to do, in which app, and that your cursor gets bigger while it works. Allow once, Always allow, Not now. Denial and the 60 second timeout both return a recoverable error naming another route, so the agent can carry on rather than stall. A run with no conversation on screen, from the scheduler or the cloud, also raises a system notification, or it would be asking nobody.

While Juno has the mouse: the system cursor enlarges (as it already did), and the bar shows a blue status dot, a hairline, and "Juno is using the mouse in Safari". An RAII guard raises and clears that state, so a panic cannot strand the indicator on.

After the first granted takeover, one offer, once per session: "Let Juno use the mouse without asking?" Yes, No, Don't ask again.

Settings, under Advanced

  • Work in the background, default on.
  • Mouse control: Ask each time, or Always allow.
  • Show in Dock, default on. Off makes Juno a menu-bar app that never becomes frontmost.

Getting back a hidden app

Turning off the Dock icon toasts where Juno went. If someone later opens Juno from Applications while it is running, macOS sends a reopen: the first one or two say "Juno is already running, it lives in the menu bar", and three within a minute, which is what being stuck looks like, offers the two things that actually help: put the Dock icon back, or quit.

Fixed in review, worth knowing

  • The mechanism that lets Juno type into a background app redirects keyboard focus to that app, and nothing handed it back, so your next keystrokes would have landed in the app Juno typed into, silently. A guard now restores focus on drop, to whatever app you are actually in.
  • That restore then had a subtler bug: "give focus to whoever is frontmost now" hands focus to Juno's target if the redirect is what made it frontmost. The decision now takes the target and can never return it, so it is correct either way.

Verified

cargo fmt, cargo clippy -D warnings on both crates, 519 Rust tests, 238 frontend tests, bunx tsc --noEmit. New unit tests cover tier selection, the consent state machine (an unknown decision string is a refusal, never consent), background-mode gating, the focus-restore rules, and the reopen escalation window.

Needs a hands-on pass on hardware

Three things cannot be proven without a running app, and all three degrade to today's behaviour rather than breaking: that SkyLight really routes keystrokes into a backgrounded Chrome or VS Code, that the private AX-observer registration really stops Chromium suspending its tree, and that a process-targeted scroll lands in the right view. The specific check: type into a backgrounded Chrome while an editor is focused, then confirm your next keystroke still lands in the editor.

…icon settings

The shared surface the background-operation work builds on: background_mode
(on by default), mouse_control (ask or always) with its prompt-dismissed
flag, and dock_icon_visible for the menu-bar-only mode.

Claude-Session: https://claude.ai/code/session_01XsDCBs2QS5BsYaZ5o1Tdo9
The contract the Rust side emits and the UI answers: an input-control
request, the state while Juno drives the cursor, the offer to stop
asking, and the reopen-attempt notice for menu-bar mode.

Claude-Session: https://claude.ai/code/session_01XsDCBs2QS5BsYaZ5o1Tdo9
The agent now reaches other apps through accessibility actions and events
posted straight to the target process, so a person can keep typing while
Juno works. The physical cursor is the last resort, and it is never taken
silently: the tool layer asks first and says so while it has the mouse.

What changed:

- Deleted ensure_main_window_focus in commands/mouse.rs and its four
  callers. Juno focusing its own window before injecting a click is what
  made the agent steal focus; right_click and mouse_move never did it and
  worked fine.
- Wired the two Phase 4 pieces that had no callers. activate_without_raise
  now runs before keystrokes are posted to a background process, which is
  the difference between clicks working in the background and the agent
  working in the background. get_ax_observer_check_remote now registers a
  remote observation per target process so Chromium and Electron do not
  suspend their accessibility tree while their windows are occluded.
- Added no-warp variants for every remaining mutating action: triple and
  middle click, drag, mouse down and up, scroll, key, hold key, release
  key and type. Each takes allow_physical; with it false they return
  Ok(None) rather than quietly dropping to the HID tap. In background mode
  a mouse_move updates Juno's virtual cursor only and leaves the real
  pointer where the user put it.
- click_auto sent every browser straight to cursor-warping simulation.
  Browsers now go through the process-targeted path with the existing
  Chromium activation primer.
- Reading an accessibility tree no longer activates the app being read,
  and the app-wide Desktop is built background-friendly: menu-bar apps
  visible, target activation off. Same for the headless, CLI, element and
  accessibility-tool engines.
- focus(element) sets AXFocusedUIElement without AXRaise in background
  mode, and reports a miss instead of falling back to a coordinate click.
- open_application no longer shells out to `open -a` to bring an
  already-running app forward.
- All of it is gated on the background_mode setting, which stays the
  single source of truth; the input layer keeps a cache that is dropped
  whenever agent settings change.

The consent contract, for the UI to build on: InputTier plus InputOutcome
say which machinery carried an action. When only the physical cursor will
do, "always" proceeds and emits input-control-state, "ask" emits
input-control-request and waits for respond_to_input_control, and a
refusal comes back to the agent as a recoverable error that names another
route. After a granted takeover, input-control-offer fires once per run.

Claude-Session: https://claude.ai/code/session_01XsDCBs2QS5BsYaZ5o1Tdo9
`activate_without_raise` redirects the WindowServer's input focus to the
target process, and nothing put it back. So after the agent typed into a
backgrounded Chrome, the user's next keystrokes landed in Chrome: no
window raised, no cursor moved, the characters just went to the wrong
app. That is worse than the behaviour this feature replaced, because it
is silent.

Input focus is now held by an RAII guard, BackgroundKeyboardFocus, for
exactly as long as the keystrokes are being posted. It captures the
frontmost app before redirecting, and on drop reads the frontmost app
again: unchanged means the user never moved, so focus goes back where it
came from; changed means they switched apps mid-flight, so focus follows
them to the app they are looking at now. A redirect that failed restores
nothing, and a target that was already frontmost costs nothing in either
direction. Drop rather than an explicit call, so an error or a panic
unwind cannot strand the keyboard.

The guard is bound at all five keyboard sites: press, hold, release, the
element-level Cmd+V fallback and the process-targeted paste, which also
covers type_text_no_warp through paste_text.

The restore reads as redundant until you know SLPS focus redirection is
not symmetric with ordinary activation, so the guard says so in a comment.

Claude-Session: https://claude.ai/code/session_01XsDCBs2QS5BsYaZ5o1Tdo9
The restore rule was 'give focus to whoever is frontmost now', which is
right when the user switches apps mid-run. But if the SLPS redirect is
what made the target frontmost, that rule returns focus to the agent's
target, which is the one outcome the guard exists to prevent. The
decision now takes the target pid and can never return it, so it is
correct whether or not the redirect perturbs frontmostApplication.

The old invariant test looped over three frontmost values and omitted
the target, so it could not catch this.
Juno can act on other apps without taking the pointer. This is the half
of that the person sees.

Settings (Advanced > Background): "Work in the background", a two-option
"Mouse control" picker (Ask each time / Always allow), and "Show in Dock"
whose description carries the whole way back out of menu-bar-only mode.
All three load together, share one loading and one error state, revert on
a failed write, and are findable through settings search.

Consent: input-control-request renders an inline prompt under the
conversation in both the main window and the bar's pane, built from the
existing Confirmation primitives. It names what Juno wants and the app it
wants it in, offers Allow once / Always allow / Not now, never takes
focus, and reopens a dismissed bar pane so the question is on screen.

Driving indicator: while input-control-state is active the floating bar
goes full width with a system-blue dot and "Juno is using the mouse in
<app>", and clears the moment the pointer comes back.

Stop asking: input-control-offer shows one calm offer per session, and
only while mouse control is still "ask" and the offer was not dismissed.

Menu-bar mode: get_dock_icon_visible / set_dock_icon_visible persist
through SettingsManager and apply macOS ActivationPolicy at runtime and
at startup. A reopen with no Dock icon counts attempts in a rolling
minute, brings the main window up, emits app-reopen-attempt and shows a
menu-bar hint; three in a minute escalates to "Show Juno in the Dock
again" and "Quit Juno".

Tests: 47 new vitest cases (consent decisions, offer gating, reopen
escalation, settings rows, bar driving state) plus 4 Rust unit tests for
the attempt-window counting.
…-mode

* origin/feat/bg-ui:
  feat(background): consent, driving indicator and menu-bar recovery UI

# Conflicts:
#	src-tauri/src/lib.rs
…n screen

A run started by the scheduler or the cloud has no conversation open, so
the in-app consent prompt has nowhere to appear and the agent would wait
out the whole timeout asking nobody. The request now also raises a system
notification naming the app and what Juno wants to do.
useEventListener(EVENTS.INPUT_CONTROL_OFFER, () => {
if (offeredThisSession.current) return;
void (async () => {
let mode: MouseControlMode = "ask";
@lacymorrow
lacymorrow merged commit 9a26ee8 into main Sep 15, 2026
7 checks passed
@lacymorrow
lacymorrow deleted the feat/background-mode branch September 15, 2026 04:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants