Inspiration
VR glasses like Quest 3 save everything you see. Current systems store 11 frames of your bottle as 11 separate memories. When you search, even unrelated floral backgrounds match at 0.54 similarity. Gap between real and fake is only 0.19.
We asked: What if VR had human-like memory? One object = one centroid, not 11 frames.
What it does
RecallOS is one-centroid memory for VR glasses.
- Enroll:
enroll_video.py --video bottlevideo.mp4 --label pink_bottle-> Keeps 4 frames with real bottle, skips 7 floral crops, creates ONE 384-dim centroid - Recall:
recall.py --image pink_chair.jpeg-> 0.40 similarity (same bottle) - Reject:
recall.py --image unrelated.jpg-> 0.11 similarity (unrelated rejected) - Forget:
enroll_video.py --forget pink_bottle-> GDPR compliant delete
Gap improved: 0.19 -> 0.37. That's anti-overfit working.
How we built it
Stack: YOLOv8n (bottle detection, conf <0.01 = skip) + DINOv2-S ONNX 84MB (384-dim embeddings) + FAISS hot+cold + centroid variance check
Pipeline:
- Video at 30.8 fps, step 15 = 11 frames (2 fps sample)
- YOLO checks each frame:
YOLO bottle crop conf=0.58 -> [159, 42, 293, 423] - If no bottle: fallback to center 35% crop
No bottle -- center 35% crop - DINOv2 embeds each valid crop
- Compute centroid, check variance 0.0030 - if tight, store 1 vector not 11
ONNX ready for Quest 3 passthrough. 84MB model runs on-device.
Challenges we faced
Challenges we faced
1. The floral background overfitting - our biggest nightmare:
Our first model enrolled all 11 frames from bottlevideo.mp4. 7 of them had no bottle, just floral chair background. DINOv2 memorized the flowers. Result: unrelated.jpg with similar flowers matched at 0.54, almost as high as real bottle at 0.73. Gap only 0.19 - system useless.
We spent 6 hours debugging. Fix was brutal: Add YOLOv8n bottle detection with conf <0.01 = skip, plus center 35% crop fallback. Now: Kept 4 / skipped 7 (no bottle: 7) and unrelated dropped to 0.11. Gap 0.37. That YOLO gate saved the product.
2. Centroid vs multi-vector tradeoff:
Storing 11 vectors gives high recall but kills on-device memory and overfits. Storing 1 centroid risks variance collapse. We added variance check 0.0030 as health metric. If variance >0.01, we know we're averaging different objects. Tuned to keep 4 good frames, not 11 noisy ones.
3. ONNX size vs Quest 3 constraints: DINOv2-B is 340MB - won't fit Quest 3. DINOv2-S is 84MB but lower dim (384 vs 768). We had to prove 384-dim still separates bottle vs unrelated. Live test: 0.40 vs 0.11 proves it works. Threshold 0.35 is the sweet spot we found after 20+ test images.
4. Frame sampling rate dilemma: At 30 fps, 10 sec video = 300 frames. Too many. We sample at 2 fps (step 15) = 11 frames. But if bottle moves fast, you miss it. Had to balance between coverage and storage. 2 fps + YOLO filtering ended up optimal.
Accomplishments that we're proud of
Technical wins:
- Anti-overfit solved: Unrelated match dropped from 0.54 -> 0.11, gap improved 0.19 -> 0.37
- One centroid storage: 11 frames -> 1 vector (91% memory saving)
- Variance check 0.0030: Proves clean object clustering, not background memorization
- ONNX 84MB: Fits on Quest 3, no cloud needed
- Full CRUD: Enroll, Recall, Forget working in Mac terminal, pushed live commit ec856c8 to GitHub
Product win: Built what Meta VR needs - human-like memory where one object = one memory. Tested on real bottle video at 30.8 fps, 11 frames sampled at 2 fps, YOLO filtered correctly.
Submission ready: GitHub pushed, 30sec screen recording with capturing live proof, no mockups.
What we learned
- One centroid beats 11 frames for VR memory: less storage, better discrimination
- YOLO + DINOv2 is ideal for on-device: YOLO filters, DINOv2 understands
- Variance 0.0030 is key health metric - low variance = clean object memory
- Threshold 0.35 optimal for 0.40 vs 0.11 separation
What's next for RecallOS
- Quest 3 passthrough integration
- Multi-label memory graph
- Voice query: "Where is my pink bottle?"
