Inspiration
Look at the current AI landscape: we are drowning in thousands of chat wrappers, note-takers, and coding copilots, yet we still lack a true Device Assistant. If you ask almost any modern AI app to actually operate your phone or computer—navigate a multi-step UI, manage system settings, automate a cross-app workflow, or bridge tasks between your Android phone and your desktop—it hits a wall. Existing native assistants are still stuck setting timers and reading weather forecasts, while modern LLM apps live trapped inside isolated chatboxes with zero agency over the operating system.
I built Device Assistant (PC-Assistant) to close that gap. Even as an MVP, it moves beyond passive Q&A to actually perceive, reason about, and operate your devices safely. Rather than giving you instructions on how to do something on your device, it does it with you and for you.
What it does
Device Assistant is a cross-platform voice, vision, and action AI agent designed for both Android and Desktop, unifying real-time screen perception, low-latency speech, and guarded OS control:
- True Device Operation & Automation: Executes real system actions, UI interactions, and device workflows across desktop and mobile environments rather than stopping at text generation.
- NVIDIA Nemotron-3 Smart Model Routing: Dynamically evaluates task complexity—from lightweight conversational queries to deep multi-step GUI reasoning—and routes requests to the optimal model to balance ultra-low latency with high-accuracy execution.
- Full-Duplex Voice & Live Screen Perception: Pairs continuous, interruptible voice interaction with real-time visual screen context so the assistant sees what you see and responds naturally without push-to-talk friction.
- Autonomous Sub-Agent Orchestration: Breaks complex user goals down into specialized sub-tasks, delegating work across isolated sub-agents and bridging commands between Android and PC.
- Privacy-Shielded Execution: Includes a sensitive-application safety guard and encrypted local memory so the agent automatically pauses screen capture or restricts autonomous execution when banking, password managers, or sensitive windows are detected.
How we built it
I architected Device Assistant as a modular, cross-platform system focused on low-latency inference, safety, and extensibility:
- Inference & Smart Routing Layer: Powered by NVIDIA Nemotron-3 and cloud inference endpoints on Nebius, I implemented a utility-aware routing layer. Given a user command with complexity score $C(x)$, latency constraint $L_{\max}$, and token cost weight $\lambda$, the router selects the optimal model $m^* \in \mathcal{M}$ by maximizing expected utility: $$U(m \mid x) = \alpha \cdot \text{Accuracy}(m, C(x)) - \beta \cdot \text{Latency}(m) - \lambda \cdot \text{Cost}(m)$$
- Cross-Platform Client & OS Bridge: Built to run across Desktop and Android (with split ARM APK builds and automated CI/CD via GitHub Actions), hooking into native accessibility and system automation APIs to inspect UI trees, capture screen states, and dispatch input events.
- Sub-Agent Runtime & Memory: Designed an orchestrator that spawns scoped sub-agents for specialized system tasks while maintaining encrypted local context so user preferences persist privately on-device.
- Safety & Privacy Guardrails: Engineered an active window-inspection filter before any vision frame or UI tree leaves the device, preventing accidental exposure of credentials or private financial screens.
Challenges we ran into
- Balancing Agency with Safety: Giving an AI agent permission to click, type, and run system commands is inherently risky. Designing the sensitive-app shield and permission boundaries so the assistant remains genuinely useful without becoming a security liability required careful sandboxing.
- Latency in Perception-Action Loops: Screen capture + vision analysis + reasoning + OS action execution can easily compound into multi-second delays. Implementing smart model routing with NVIDIA Nemotron-3 and streaming full-duplex audio was critical to keeping interactions snappy.
- Cross-Platform OS Fragmentation: Bridging desktop OS APIs and Android accessibility/system controls under a unified agent architecture—and configuring clean, architecture-specific mobile builds in CI—took significant iteration.
Accomplishments that we're proud of
- Building a functional MVP that outperforms existing consumer assistants when it comes to real device control and multi-step execution.
- Seamlessly integrating NVIDIA Nemotron-3 smart routing to keep simple commands instantaneous while reserving heavy reasoning for complex device automation.
- Shipping both Desktop support and optimized Android ARM builds with a built-in privacy shield that respects user security out of the box.
What we learned
- Agency requires hierarchy: A single monolithic prompt cannot reliably converse with a user, monitor screen changes, and execute OS commands simultaneously. Splitting responsibilities between a fast conversational front-end, a smart router, and specialized execution sub-agents dramatically improved reliability.
- Guardrails must be deterministic: You cannot rely solely on an LLM's system prompt to "ignore passwords on screen." Deterministic OS-level window filtering before inference is essential for user trust.
What's next for Device Assistant
- Deeper Cross-Device Hand-off: Expanding seamless peer-to-peer task delegation so you can start a workflow from your Android phone on the go and have your PC finish the heavy lifting at home.
- Hybrid Local + Cloud Execution: Combining small on-device edge models for instant UI element grounding with Nebius/NVIDIA cloud models for complex multi-app orchestration.
- Plugin & Skill Ecosystem: Allowing developers to define custom, sandboxed device skills and automation macros with zero-trust execution policies.
Log in or sign up for Devpost to join the conversation.