Atlas

recursive model training
housing model
camera vision
model explosion

Inspiration

AI assistants live in someone else's cloud and only know what you type into them. Your life is spread across devices and apps: a camera at the door, a calendar, Canvas, group chats, a browser, a watch. We wanted one personal AI computer that lives in your home, connects to all of them, remembers what happens across them, helps with your actual week, and keeps that memory on the box while constantly improving.

What it does

Atlas is one everyday tool: a personal AI computer (an NVIDIA Jetson) that connects to your devices and apps, remembers what happens across them, and helps with your day.

Plugs into anything. Every device or app is a connector: reads sync into Atlas's memory, and anything that writes to the outside world waits for your tap. Today: a camera, the web (search and Reddit through a real browser), Canvas, your Outlook calendar, GroupMe, the campus site, your watch's health data, and voice on your phone. Smart lights, a gate or a car plug in the same way.
Remembers. Everything it hears about becomes events in an append-only log: its memory. The camera is one source: detection, tracking and zones run on the Jetson, and faces are blurred before anything is stored.
Answers in plain English, with evidence. "What happened at our table overnight?" "Where did I leave my keys?" "What's due this week?" Every answer cites the episode, keyframe, message or link it came from. If Atlas doesn't know, it says so.
Alerts and acts, with consent. A package left at the door buzzes your phone. Anything with an outside effect (a message, a calendar invite, a data export) is an approval card on your phone first.
Plans across all of it. Before an exam, Atlas writes a short plan from the course site, Reddit, your group chat and your free slots, and offers study blocks as calendar invites to approve.
Runs your day. Each morning it plans from your calendar, sleep and habits, re-plans when the day slips, and reviews it at night. Day Run lets you play tomorrow on a campus map before you live it. Voice in and out on the phone.
Friends' Atlases plan together. Atlases in a Circle agree on a time through Neon Postgres; each box only answers free, busy or a counter-offer, so nobody's schedule leaves their box.
Waste Watch. Atlas counts single-use items at the table, sees which bin they end up in, and reports the measured diversion rate before and after nudges.
Learns new tasks. Teach a task on the phone, approve the export of blurred keyframes, and a Qwen3.8-27B teacher on an AMD Strix Halo labels them while a LoRA fine-tune of the Jetson's small Qwen runs on a second one.
Pull the plug. Remembering, alerting and answering keep working with no internet; your memory never leaves the box.

How we built it

Jetson (all runtime inference): a Python 3.11 FastAPI modular monolith; an in-process event bus with a durable JSONL log as the source of truth (state rebuilds by replay); SQLite; ultralytics + TensorRT for detection and tracking; llama.cpp (CUDA) serving Qwen3.5-4B Q4_K_M with its vision projector; OpenCV YuNet for face blur; Whisper and Piper for voice. One Runtime interface hides every model.
Agent: common questions are pre-fetched in code and answered from one short prompt, which suits a small quantized model; open questions use a tool loop capped at 3 calls. Rules are YAML or plain English turned into validated JSON and evaluated on event time, so alerts work offline.
Connectors: each source is an MCP server; reads sync on a timer, writes become approval cards. The browser connector uses patchright in a visible, persistent profile.
Learning lab (AMD): two AMD Strix Halo pods (Radeon 8060S, gfx1151, 64 GB shared memory) on AMD's AUP Learning Cloud. The teacher is vLLM built from source for gfx1151 with our patch, serving Qwen3.8-27B; the trainer is PyTorch + PEFT/TRL LoRA. amd-smi and tegrastats give watts, so every answer and training run has a joule count.
Circle: Neon Postgres for shared state (agent messages, RSVPs, leaderboards), with a local outbox so the box works offline.
Phone: a web app served by the Jetson, a WebSocket event stream, and self-hosted ntfy for alerts and approval buttons.
Team process: four people, each with a coding agent, coordinating through Notion task locks, Q&A pages and an overnight status board.

Challenges we ran into

Making a 4B model reliable. Multi-step tool calling is shaky on a small quantized model, so the most common questions skip the tool loop. We wrote 48 ways a judge might ask our 8 demo questions: 35/42 at first, 48/48 after the fixes, answers in 0.6s.
Losing our GPUs mid-hackathon. The hub with our two Radeon AI PRO R9700s went down on Saturday, so we moved the lab to two Strix Halo pods and rebuilt vLLM for a new chip.
Fine-tuning on AMD. A 27B model is ~54 GB in bf16, too big to train on one device, which is why we went teacher-student.
Bot checks. Headless browsers get CAPTCHAs; a visible, persistent profile gets through for the sites a person would visit anyway.
Replay safety. Rebuilding state from the log must never resend a message or repeat an approved action, so handlers opt in to replay explicitly.

Accomplishments that we're proud of

Everything at runtime happens on the Jetson: one box for memory, answers and actions across every connected device; the GPUs are only where Atlas learns.
Every answer cites its evidence, and every outside action needs a tap.
From the box's own log on Saturday: ~10,700 frames seen, 9,738 objects and people observed, 19 episodes remembered, 55 cited answers, 725 calendar/Canvas/chat items read.
YOLO11n TensorRT FP16 at 41 FPS; a 30-minute soak at 15 fps with 0 errors; Qwen3.5-4B at 13–14 tok/s on the Orin Nano.

What we learned

Pre-fetching beats tool loops for small models.
An append-only event log makes a system safe to change while it runs.
Consent is a product feature: approval cards made the agent feel trustworthy, not risky.

What's next for Atlas

More connectors: smart lights, a garage gate, the car, so Atlas can see more of your life and (with a tap) act on it.
More learned skills (food, drink and recyclable recognition).
Replays of real footage: frame interpolation on AMD turns approved, blurred keyframes into a short video.
More campus connectors and place packs, so Day Run works on any campus.