Front-end
Back-end

Inspiration

Navigating a computer using standard keyboards and mice often feels slow and high-friction. I wanted to build an intelligent, hands-free desktop environment where natural voice commands and touchless hand gestures execute system tasks instantly without touching hardware, all while keeping user data completely private and secure.

What it does

PARC & Milo is an advanced, local-first autonomous desktop control and AI assistant engine. It runs 100% locally with zero internet dependency, guaranteeing absolute data security with no risk of user data theft or tracking. It combines real-time webcam gesture tracking with continuous voice commands to handle daily computer tasks and system interactions with ultra-fast speed. Additionally, when internet is connected, PARC & Milo enables remote desktop commanding directly from your mobile phone.

How we built it

Intelligence & Local Voice Pipeline: Written in Python (milo.py and jarvis.py) to process contextual logic, parse speech input, and execute OS actions completely offline.
Computer Vision Engine: Powered by MediaPipe hand landmark tracking (hand_landmarker.task) and OpenCV for low-latency, touchless spatial gesture tracking via standard webcam.
Mobile Remote Bridge: Configured with network protocol handling to accept remote commands from a mobile phone when connected to an active network.
Target Environment: Designed and optimized natively for Linux for fast, kernel-level execution.

Challenges we ran into

Offline Latency & Performance: Optimizing real-time computer vision and voice loops locally so they run fast without relying on cloud APIs or heavy external servers.
Absolute Privacy Architecture: Structuring the system so video streams, audio data, and system commands stay strictly on the local machine with zero data leakage.
Mobile-to-Desktop Synchronization: Building a low-latency bridge to trigger desktop actions remotely from a mobile device when internet connectivity is enabled.

Accomplishments that we're proud of

Building a secure, fast, and advanced desktop agent that requires zero cloud connectivity to operate.
Achieving full touchless spatial control using a standard webcam—no expensive external sensors needed.
Delivering dual-mode capability: 100% private local execution offline, plus mobile remote command capabilities when connected.

What we learned

Multi-threading local computer vision workloads with speech recognition loops without consuming excessive system resources.
Implementing low-latency socket handling for seamless mobile-to-desktop interaction.
Designing privacy-first architecture for local autonomous agents.

What's next for PARC & MILO

Integrating fully offline local LLMs (via Ollama) for zero-latency, private reasoning without cloud dependencies.
Developing a dedicated mobile app interface for seamless remote control over internet connections.
Expanding spatial gesture commands for multi-window management (swiping workspaces, pinching to resize).