Inspiration
Blind and visually impaired users face high barrier on modern PCs. Screen readers (NVDA, JAWS) require memorizing hundreds of keyboard shortcuts. Cloud voice assistants (Siri, Alexa, Copilot) fail core accessibility:
- Leak private screen/microphone data to external servers.
- Require active internet and paid API tokens.
- Guess interfaces via fragile visual screenshots rather than reading native OS controls.
- "Fire and forget": announce action succeeded without checking reality.
- Bloated resource footprint demanding ≥ 16 GB RAM and dedicated GPU.
Mission: build RELAY. Fast, 100% offline, voice-first operating system for Windows. Built for blind users, powerful for any hands-free operator. Budget hardware target: standard dual-core laptop with 4 GB RAM.
## What it does
RELAY transforms Windows 10 & 11 into voice-controlled computer:
• Hands-Free Desktop Control: Open apps, switch windows, manipulate files, dictate text into any field, execute multi-step workflows ("open Notepad, type shopping list, save to Documents").
• Native Screen Perception: Connects directly to Microsoft UI Automation (UIAutomationCore). Reads controls, text ranges, and focus changes. Falls back to on-demand RapidOCR for images/canvases.
• Closed-Loop Verification: Actions observed, never assumed. Verifier re-queries OS post-execution. Classifies outcome into set S = {VERIFIED, UNCERTAIN, FAILED}. Uncertain mutations never repeated blind.
• Deterministic 6-Tier Safety:
Tier ∈ {SAFE, REVERSIBLE, CAUTION, CONFIRM, ELEVATED, BLOCKED}
High-stakes operations (delete file, send email, checkout) mandate explicit spoken verification phrases ("confirm delete"). Casual "yes" strictly rejected.
• Offline Knowledge Engine: Instant factual Q&A and extractive web summarization via local Wikipedia engine without cloud LLMs.
• Screen Reader Coexistence: Detects NVDA/JAWS/Narrator; suppresses redundant focus announcements. Auto-routes audio to headphones; pauses speech immediately on disconnect.
## How we built it
[Microphone In]
│
▼
[webrtcvad (VAD)] ──> [Whisper INT8 (STT)]
│
▼
[WinEvents / UIA] ──> [Deterministic Grammar] ──> [Planner & Runner]
│
▼
[Piper Neural TTS] <── [Verifier (Observe OS)] <── [Safety Gate (6 Tiers)]
│
▼
[Executor (SendInput / UIA)]
• Speech Pipeline:
• Local STT: Quantized INT8 Whisper (faster-whisper / whisper.cpp) running CPU-only AVX2 vector math.
• Local TTS: Piper neural synthesis with ONNX runtime, falling back to Windows SAPI.
• VAD & Gating: Lightweight frame classifier (webrtcvad) with pre-roll buffer Δt = 300 ms to prevent clipping wake word.
• Perception & Accessibility Engine:
• Native COM interface to IUIAutomation via comtypes and ctypes.
• Hooked WinEvents (EVENT_SYSTEM_FOREGROUND, UIA_AutomationPropertyChangedEventId) for event-driven push rather than polling loops.
• Screen OCR fallback: rapidocr-onnxruntime + mss for non-accessible legacy canvases.
• Reasoning & Planning:
• Sub-millisecond deterministic regex/slot grammar (grammar.py) bypassing LLM inference for OS commands (t_{parse} < 1 ms).
• Compound intent splitter (compound.py) sequencing 10 + step autonomous plans.
• Core Memory & Persistence:
• 5-layer state store: Task Context, Journal, Preferences, Notes, and SQLite with FTS5 search.
• Frontend & IPC:
• Optional web visualizer / terminal: zero-dependency HTML5, CSS3, Vanilla JS.
• Authenticated loopback IPC: Python stdlib ThreadingHTTPServer + Server-Sent Events (SSE) streaming state changes with constant-time token comparison.
## Challenges we ran into
### 1. CPU Oversubscription & Spin-Locks on Dual-Core Chips
• Problem: Default ONNX runtime and OpenMP threads used active spin-waiting. On 2-core budget CPU emulation, idle CPU spiked to ∼ 36% and TTS stuttered with Real-Time Factor RTF = 1.61.
• Fix: Sized threadpools to available physical cores (≤ 4), disabled busy-wait spins, and enforced environment constraints:
KMP\_BLOCKTIME = 0, OMP\_WAIT\_POLICY = PASSIVE
Dropped idle CPU to 2.5% and improved TTS RTF to 0.23 (4 × faster than real-time playback).
### 2. Audio Latency Budget on Low-End Laptops
Total user wait time defined by:
t = t + t + t + t + t
wait VAD ASR parse exec TTS
• On Celeron-class CPU (2 cores @ 25% limit): unoptimized baseline was t_{wait} ≈ 10.7 s.
• Through INT8 weight quantization, aggressive silence hangover tuning (Δt_{hang} = 1.1 s), and streaming first-sentence TTS synthesis, reduced latency to:
t = 0.6 s + 2.11 s + 0.001 s + 0.02 s + 0.57 s = 3.3 s
wait
On unconstrained hardware, t_{wait} = 1.3 s.
### 3. Acoustic Feedback & Half-Duplex Loop
• Problem: Laptop speakers playing TTS output triggered microphone, causing assistant to transcribe and obey its own voice.
• Fix: Implemented half-duplex mic suppression in loop.py: microphone stream discarded during TTS playback plus tail buffer τ = 450 ms. Keyboard interrupt (Alt+Space) provides hardware barge-in.
### 4. Cross-Process UIA COM Deadlocks
• Problem: Synchronous UIA calls to unresponsive applications hung entire execution thread indefinitely.
• Fix: Isolated perception into self-terminating worker threads (worker.py) wrapped in bounded timeouts (Δt_{timeout} = 3.0 s), automatically dropping hung threads without leaking OS handles.
### 5. False-Positive Verifications
• Problem: Naive screen checks reported "success" if arbitrary background pixel or clock updated.
• Fix: Bound verification to deterministic assertions: specific window HWND registration, process ID matching via psutil, control value patterns, or target disk file existence:
P(False Positive) → 0
## Accomplishments that we're proud of
• 100% Offline with Zero Cloud Reliance: Runs speech-to-text, text-to-speech, intent parsing, perception, and verification locally without transmitting single byte over internet.
• Rigorous Memory Footprint: Fits within 4 GB RAM budget. Verified under Windows Job Object with hard 1 GB memory limit:
M = 35 MB, M = 402 MB ≤ 1024 MB
idle peak
• UIA Batch Query Optimization: Replaced repetitive COM lookups with single IUIAutomationCacheRequest, converting IPC overhead:
O(N) → O(1) COM round trips
• Rock-Solid Safety Engine: 6 risk tiers prevent accidental deletion or message sending. Casual affirmative words ("yeah", "yep") fail-safe. • High Test Coverage & Rigorous Gates: • 383 passing tests in test suite. • 8/8 offline acceptance checks verified in acceptance.py. • 12/12 live desktop diagnostics passed.
## What we learned
- Accessibility Requires Determinism: Probabilistic LLM agents hallucinate UI actions. Blind users need 100% predictable state machines where critical operations have zero ambiguity.
- Input Sent ≠ Action Completed: An OS assistant cannot claim success upon firing keycode. Closed-loop verification is fundamental invariant for trust.
- Out-of-Process UIA Beats DLL Injection: Injecting DLLs into target apps (NVDA style) risks antivirus triggers and security blocks. High-performance out-of-process UIA with caching matches speed without risking system instability.
- Hardware Diversity Dictates Architecture: Optimizing for high-end dev workstations leads to brittle software. Enforcing hardware emulation caps (Job Objects) early exposes concurrency bottlenecks that pure unit tests miss.
## What's next for Relay
• Multilingual Edge Speech: Integrate quantized on-device multilingual models (Whisper small / Sherpa-ONNX) supporting Hindi, Telugu, Tamil, and Spanish. • Hardware Braille Display Support: Expose Braille display driver interfaces (BRLTTY) synchronized with speech cursor. • Cross-Platform Engine (Linux AT-SPI2 & macOS Accessibility API): Port core verifier and intent layers to Linux Wayland and macOS while preserving deterministic safety pipeline. • Signed Distribution: Secure Extended Validation (EV) code-signing certificate for standalone Windows installer executable (relay.exe).
Log in or sign up for Devpost to join the conversation.