VoiceOS

A voice-driven computer-use agent for Windows 11. Hold a hotkey, say what you want — VoiceOS transcribes your speech locally, looks at your screen through a vision model, and drives the mouse and keyboard to do it. One universal loop that works in any application: no per-app integrations, no browser extensions, no accessibility trees.

Hold hotkey → speak → release
   │
   ├─ whisper.cpp (local) ──→ transcript, shown live in a glass overlay
   │
   └─ agent loop:  screenshot → vision LLM → one action → repeat
                   (click / type / keys / scroll / drag … until done)

Highlights

• Push-to-talk with a global hotkey (default Ctrl+Space); speech never leaves your machine — transcription is local whisper.cpp.
• Sees the screen like you do. Screenshots go to a grounding-capable vision model (default qwen/qwen2.5-vl-72b-instruct via OpenRouter); it answers with exactly one JSON action per step.
• Universal control: clicks, double/right/middle clicks, drags, scrolling, typing, and key combos, synthesized at native resolution with DPI-safe coordinate mapping.
• Glass overlay HUD: your words and the agent's steps/results float in a minimal Apple-style glassmorphism card, always on top, out of the way.
• Safety gates (PRD §9): deleting files, sending email, terminal commands, and installs always require an explicit Allow in the UI. Emergency stop by hotkey (default Ctrl+Shift+Escape) or just saying "stop".
• Workflows: record your mouse/keyboard once, replay any time with "run morning setup" — deterministic, no model in the loop.
• History of every command with per-step detail, stored in SQLite.

Getting started (Windows 11)

See docs/SETUP-WINDOWS.md for the full guide. Short version:

# 1. Prereqs: Rust (stable), Node 20+, and the MSVC build tools + CMake
# 2. Whisper model (~150 MB)
powershell -ExecutionPolicy Bypass -File scripts/download-models.ps1

# 3. Build & run
cd apps/desktop
npm install
npm run tauri dev      # or: npm run tauri build  → NSIS installer

Then open Settings in the app, paste your OpenRouter API key, hold Ctrl+Space, and say "open notepad and type hello".

Repository layout

Crate / dir	What it is
crates/voiceos-core	Platform-agnostic heart: action model, agent loop, coordinate scaling, safety gates, workflow replay. Fully unit-tested.
crates/voiceos-vlm	OpenRouter vision client: prompting, strict JSON action parsing, reprompt-once policy.
crates/voiceos-speech	cpal capture, audio preprocessing, whisper.cpp transcription.
crates/voiceos-storage	SQLite: settings, history, workflows.
crates/voiceos-platform	Real desktop adapters: xcap screenshots, enigo input, rdev workflow recorder.
crates/voiceos-mock	Simulated screen/input/model for tests.
apps/desktop	Tauri 2 app: pipeline wiring, hotkeys, overlay + main window (React/TS).

More detail in docs/ARCHITECTURE.md.

Development

cargo test --workspace          # core logic, VLM client, storage, speech DSP
cargo clippy --workspace
cd apps/desktop && npm run typecheck

The whole agent loop runs against mocks on any OS; only the thin voiceos-platform adapters and the Tauri shell need Windows to do anything real. CI builds the Windows NSIS installer on every push.

Status

MVP per the VoiceOS PRD: Windows 11, cloud vision model (local VLM planned), push-to-talk (wake word planned), ~8 agent steps per task by default.