Nexus AI Core

A privacy-first, autonomous desktop companion that blends cloud and local LLMs to safely automate your PC via voice without compromising your data.

Comment

Inspiration

What it does

How we built it

Challenges we ran into

Accomplishments that we're proud of## Inspiration

Traditional AI assistants come with a massive catch: to automate your workflow, you have to hand over your keys, your filesystem, and your private data to cloud servers. We wanted a powerful digital companion that could interact with our local machine, handle operating system automation, and launch applications seamlessly without ever compromising sensitive personal data.

We were inspired to build Nexus AI Core—a production-grade, highly autonomous desktop assistant that bridges lightning-fast cloud intelligence with secure, completely private offline local execution, all wrapped inside a beautiful, immersive sci-fi interface.

What it does

Nexus AI Core acts as an intelligent operating system companion. It operates via an advanced Modular Agentic Architecture that dynamically routes tasks based on privacy and complexity:

  • Local PC Automation (100% Offline): Uses a local Ollama instance running llama3.1 to execute host machine tasks like launching apps natively (e.g., WhatsApp via Microsoft Store protocols), searching the web, and running files completely offline.
  • Native Media Auto-Play: Integrated with pywhatkit, it captures your voice commands, finds the top video on YouTube, and spins it up instantly in full screen.
  • Hybrid Brain Router: Dynamically splits tasks between Groq llama-3.3-70b-versatile (sub-second lightning chat), Google Gemini (deep coding/reasoning), and Ollama (private workflow tasks). It natively supports and handles multi-lingual Hindi/Hinglish action routing perfectly.
  • Deep Semantic Memory: Combines Redis for instantaneous context management with a local ChromaDB vector database so it learns your habits and remembers past sessions.
  • 3D Holographic Interface: A responsive glassmorphic HUD designed with React, Vite, and Three.js featuring real-time telemetry and scanning animations.

How we built it

We built Nexus using a high-performance asynchronous FastAPI backend to coordinate the AI engines and local speech-to-text / text-to-speech pipelines via native Windows SAPI5 (ensuring 0ms network latency on voice).

The frontend relies on React, Vite, and Three.js to break out of the boring chat-box paradigm and create an interactive 3D HUD. Memory is persistently managed locally through ChromaDB and cached via Redis.

Challenges we faced

One of our biggest challenges was eliminating the latency usually associated with speech recognition and tool execution. Routing every prompt to the cloud felt sluggish, while routing everything locally dragged down machine resources.

We solved this by developing our custom Agentic Routing Engine. By analyzing the user's intent upfront, the system instantly hands off light chat requests to Groq, deep reasoning to Gemini, and highly private system-level scripts to a lightweight local Ollama model.

Accomplishments that we're proud of

  • Achieving a sub-800ms chat response latency by leveraging Groq pipelines.
  • Building a fully functional local tool registry with safety safeguards (like an absolute 60-second command abort window).
  • Creating a visually striking 3D interface that runs beautifully inside a web view without weighing down the system.

What we learned

We learned the intricacies of multi-model orchestration and semantic memory pairing. Managing a unified context window across different providers (Ollama, Groq, and Gemini) taught us how critical standardized JSON intent framing is for modular AI networks.

What's next for Nexus AI Core

Next on our roadmap is expanding into smart home IoT orchestration, supporting multi-agent swarms for complex local software engineering tasks, and packaging the frontend into a native desktop wrapper using Tauri or Electron.

What we learned

We learned the intricacies of multi-model orchestration and semantic memory pairing. Managing a unified context window across different providers (Ollama, Groq, and Gemini) taught us how critical standardized JSON intent framing is for modular AI networks

What's next for Nexus AI Core

Next on our roadmap is expanding into smart home IoT orchestration, supporting multi-agent swarms for complex local software engineering tasks, and packaging the frontend into a native desktop wrapper using Tauri or Electron

Built With

Share this project:

Updates