Servent-AI

A 100% local, privacy-first Windows OS agent controlled using hand gestures (MediaPipe) and voice commands (Whisper) that plans and verifies task execution via local Gemma 4 and Moondream.

Comment

Inspiration

Computers are the gatekeepers to modern work, education, and financial independence. However, for people with motor or physical disabilities, traditional keyboards and mice can be massive barriers. We wanted to build an agent that makes computers accessible to everyone.

Servent-AI is designed to empower individuals with disabilities, allowing them to control their laptops entirely hands-free using natural voice commands and simple head/hand gestures. By enabling them to navigate, write, and even code on their computers, Servent-AI opens doors to digital careers, helping them earn a living, build projects, and contribute to society.

What it does

Servent-AI is a local, privacy-first computer control agent. It combines:

  1. Multimodal Inputs: Continuously transcribes voice commands using a local Whisper STT model and tracks hand movements/gestures using MediaPipe.
  2. Local Reasoning (Aria Brain): A local Gemma 4 E4B model acts as the planner, converting raw user commands into a structured sequence of JSON steps.
  3. Self-Correcting Execution (VISTA): Executes mouse clicks, keyboard navigation, and typing using PyAutoGUI, while a local Moondream vision model acts as the eyes to verify if each step succeeded (e.g. checking if a page loaded).

How we built it

  • Backend: Built in Python using asyncio, websockets, and PyAutoGUI.
  • Brain Engine: Configured to interface with LM Studio running Google's Gemma 4 E4B (quantized to Q4_K_M) for offline, privacy-safe reasoning.
  • Visual Verification: Integrated Moondream via Ollama to run screen checks locally on our device's integrated GPU.
  • Frontend: A sleek web dashboard served locally that displays real-time hand coordinates, system state, and active commands.

Challenges we ran into

We needed to make the agent fast enough to feel responsive while running entirely on consumer hardware (Core Ultra 5 with 16GB RAM). We optimized the local execution loop by:

  • Offloading visual checks to the small 1.6B Moondream model.
  • Quantizing Gemma 4 to fit comfortably within system RAM alongside MediaPipe and Whisper.
  • Correcting model outputs to make sure local reasoning bypasses unnecessary thinking tokens for faster planning latency.

Accomplishments that we're proud of

We successfully built a 100% local, offline loop that can open a browser, go to Gemini, wait for the page to load, write a prompt, copy the response, open WhatsApp, and send it to a contact—all from a single spoken voice command.

What we learned

Building Servent-AI taught us that you don't need expensive cloud APIs to build highly capable agentic workflows. Optimization of smaller, local open-weight models is the future of accessible, privacy-respecting computing.

Built With

Share this project:

Updates