ORBIT: Personal AI Workspace for Understanding Systems

A personal AI workspace that turns any system or GitHub repo into an explorable 3D architecture, explains each part on demand, and remembers what you learn. Powered by NVIDIA Nemotron.

Comment

Inspiration

Learning how real software systems work is harder than it should be. Docs are walls of text, architecture diagrams are static, and every AI chat forgets what you were studying yesterday. We wanted a personal AI workspace where you can see how a system fits together, ask about any part, and keep what you learn in a memory you control.

What it does

ORBIT is a personal AI workspace for understanding systems.

  • Visual Learning Studio: search a system (for example a ChatGPT-style platform or an e-commerce backend) and explore it as an interactive 3D architecture. Every part explains what it does, its APIs, data model, failure modes, scaling and trade-offs, and a guided tour follows a request through the system.
  • Hand-gesture control (optional): point, pinch, swipe and open-palm gestures run in the browser with MediaPipe. Camera frames never leave the device.
  • Explain on demand: selecting a part never calls the model. Only pressing Send does, so the user controls every token spent. Answers arrive as structured sections, not raw text.
  • Personal memory: save what you learn, then search, edit or delete it. Memories belong to the signed-in user.
  • Tasks and daily plans with approval: proposed tasks and schedules are saved only after the user approves.
  • Reusable skills that can be switched on or off, and a debug panel showing the provider, model ID and latency for each turn.
  • [Include only if working] Repo to system design: paste a public GitHub repository and get an architecture scene grounded in the repo's real source files.

How we built it

  • Frontend: React, TypeScript, Vite, Tailwind, TanStack Query, Three.js with React Three Fiber, MediaPipe Tasks Vision.
  • Backend: FastAPI, Pydantic, SQLAlchemy, Alembic, JWT auth with per-user data scoping, rate limiting and an audit log.
  • Models: NVIDIA Nemotron through NVIDIA's OpenAI-compatible hosted API, routed by task (details below).
  • Safety: secret redaction, strict input validation, approval gates for write actions, and clear labels on model-generated designs.

How we use NVIDIA Nemotron

  • Nemotron 3.5 Super (120B-A12B) handles heavy generation such as designing 3D architecture scenes.
  • Nemotron 3.5 Lightning (30B-A3B) handles fast, frequent calls such as short replies and classification, which keeps responses quick and rate limits manageable.
  • [Confirm this is deployed] Nemotron 3.5 Embed (1B) powers semantic memory search.
  • We use the model's reasoning toggle (enable_thinking) off for JSON-producing calls and low-budget for explanations, and validate every structured output with Pydantic.

A note on Nebius

ORBIT was designed for Nebius Token Factory through an OpenAI-compatible provider layer, so Token Factory is a one-line configuration change. However, Nebius account verification requires a payment card that was not available to us from our country, so we could not create an account. The submitted demo therefore runs on NVIDIA's hosted Nemotron API. We would welcome a card-free path for participants in our region.

Challenges we ran into

  • Models echoing the JSON schema. Early answers displayed schema fragments. We switched to example-based prompts, added a normalizer and validation with a repair retry, and render structured answers instead of raw text.
  • Token control. Selecting a part originally triggered an explanation automatically. We made the chat strictly manual so no request happens without a deliberate Send.
  • Hallucinated architecture. A repo analysis once invented a CDN that did not exist. We isolated the analysis inputs and added grounding checks so every component must cite real files.
  • Schema drift and deployment. Database migrations, hosting on free tiers (cold starts, ephemeral disks) and production API wiring all needed real fixes.

What we learned

Structured model output needs validation, not trust. Cost control is a design feature. A good explanation is grounded in real data, and labelling what is model-generated builds trust.

What's next

Private repository support, drill-down into sub-systems, more curated scenes, and enabling Nebius Token Factory once access is available.

Built With

Share this project:

Updates

Submission history