Zhangluo (小張羅): Multi-Modal Household & Terroir AI

🎬 Watch the 1m48s 1080p English-Narrated Demo Video: https://youtu.be/1wwW4HmMAUA

• 0:00 - 0:28 ⚡ The 30-Second Aha Moment: 12-Second VideoSweepRecorder (750ms pixel-diff >= 0.035) scans a home pantry/cellar and intercepts a duplicate purchase before checkout (Household Ledger 起居簿).
• 0:29 - 0:58 🧭 Anti-Paid-Ad Terroir & Qloo Taste Graph: Cross-grounding 2,843 counties (PostGIS + pgvector) with Qloo Taste AI (>= 10 yr heritage filter) and Tavily live merchant verification (Solar Terroir 风物志).
• 0:59 - 1:25 🧠 Auditable Sovereign Memory (OpenViking): Force-directed Taste Constellation (useForceLayout) with explicit consent_id inbox (Sovereign Memory 自留地).
• 1:26 - 1:48 💳 Fiduciary Agentic Checkout: comparison_digest cryptographic price/spec audit + PayPal Sandbox Orders v2 settlement with automatic inventory write-back.

What We Built: Beyond Chatbots — A Multi-Modal Personal & Household OS

Zhangluo (小張羅) solves the hardest problems in Personal & Household AI through seven production-grade engineering modules:

1. 12-Second Camera Video Sweep Inventory (VideoSweepRecorder + Vision Extraction):
   • Instead of typing item lists by hand, users pan their phone camera across a closet or storage room for 12 seconds.
   • The client samples frames every 750ms, computes pixel-difference scores (differenceScore >= 0.035) to extract non-redundant keyframes, and batch-extracts household items and hierarchical space_path locations (Guest Closet ➔ Top Shelf ➔ Vacuum Bag #2).
2. 4-Pillar Household Ledger (起居簿: Granary 粮仓 · Inventory 家当 · Guest Redlines 待客 · Gift Reciprocity 礼单 · Maintenance 维保):
   • Tracks pantry best-before windows, prevents duplicate purchases, remembers guest dietary restrictions (e.g., gout low-purine rules), and tracks seasonal gift reciprocity.
3. 5 Native iOS Swift Plugins (deer-flow/mobile/ios):
   • System-level ShareExtension (zhangluo://share-intake), VoiceDictationPlugin (SFSpeechRecognizer), ReminderCalendarPlugin (iOS 17+ EventKit with regex PII masking), ApnsNotificationPlugin, and HttpStreamPlugin.
4. Nebius Token Factory (nvidia/Llama-3.1-Nemotron-70B-Instruct-HF) + Tavily Search Verifier:
   • Runs bounded semantic DAG reasoning on Nebius Token Factory while verifying National GI standards (GB/T 20560), merchant longevity ($\ge 10\text{ years}$), and product specs in real time via Tavily Search API.

Structured Feedback on Nebius Token Factory & NVIDIA Open-Source Models

1. Onboarding & Integration:
   • Nebius Token Factory's OpenAI-compatible endpoint (https://api.studio.nebius.ai/v1/chat/completions) allowed us to wire nvidia/Llama-3.1-Nemotron-70B-Instruct-HF into Zhangluo's bounded semantic DAG and 4-pillar Household Ledger orchestrator in under 10 minutes.
2. Performance on Multi-Slot Household & Spatial Reasoning:
   • Llama-3.1-Nemotron-70B-Instruct-HF excelled at simultaneously reasoning over 4 household constraints (Granary expiring bamboo shoots + Guest gout low-purine redline + Study Tatami duplicate storage box interception + Pending return gift for Old Chen) while strictly respecting OpenViking consent_id boundaries.
3. Suggestions for Nebius & NVIDIA:
   • Providing multi-frame vision-language batch endpoints alongside Nemotron text inference on Nebius Token Factory would make our 12-second VideoSweepRecorder keyframe batch recognition even faster.

🧠 Model Routing Architecture: Which Model Does What (Nebius × NVIDIA + Tavily + Qloo)

Following the principle that one focused loop done well beats a bloated stack, every model and tool in Zhangluo serves a single, deterministic role inside the fiduciary household loop:

👥 Real User Validation & Accessibility Design

• Target Audience & User Interviews: Tested with 18 multi-generational households, 3-property families, and home hosts who manage elder dietary redlines (gout/low-purine, diabetes), seasonal gift reciprocity (人情礼单), and recurring appliance maintenance (维保).
• What Changed After User Testing: Early testers rejected generic recipe/shopping chatbots because they recommended ingredients already sitting in the freezer and ignored elder health restrictions. That directly drove our "Inventory-First Duplicate Interception" and "Hard Dietary Redline Gate" in the Household Ledger (起居簿).
• Accessibility & Elder-Friendly UX:
  • Voice-First & Zero-Typing Capture: Native streaming voice dictation + 12-second continuous camera sweep (VideoSweepRecorder) allows older family members to log pantry items without typing small text.
  • Family Join-Code (接头令): Multi-residence family sync with role-scoped visibility so adult children can remotely verify parents' pantry expiration alerts and appliance warranties.

⚠️ Documented Limitations, Failure Modes & Engineering Trade-offs

We explicitly document current failure boundaries and how the system degrades safely (Fail-Closed):

1. 12-Second Video Sweep (VideoSweepRecorder) Occlusion & Low-Light Boundary:
   • Failure Case: When scanning densely stacked deep cabinets or reflective glass jars under < 80 lux indoor lighting, pixel-difference keyframe extraction (Delta >= 0.035) can miss occluded rear labels or misread expiration dates.
   • Mitigation: Any item with OCR/vision confidence < 0.82 is flagged as NEEDS_HUMAN_CONFIRM in the OpenViking pending inbox rather than silently written to the Household Ledger.
2. Tavily Live Price & Stock Drift on Flash-Sale SKUs:
   • Failure Case: Fresh seasonal terroir items (e.g., pre-rain West Lake Longjing tea or Yangcheng Lake hairy crabs) can sell out between the Tavily search snapshot and PayPal checkout execution.
   • Mitigation: Our comparison_digest enforces a strict 15-minute TTL; if stale at checkout, the agent blocks auto-capture and re-verifies stock before charging.
3. Cold-Start Sparsity in Qloo Taste Graph for Hyper-Local County Specialties:
   • Failure Case: Niche county-level GI ingredients (among China's 2,843 counties) occasionally lack direct entity IDs in Qloo's global graph.
   • Mitigation: We fall back to semantic flavor-profile bridging (pgvector 768-dim terroir embeddings mapped to Qloo culinary taste clusters) and explicitly label the confidence tier in the UI.

🛠️ What Was Built & Updated During the Hackathon Submission Period

• Built the standalone, zero-proprietary-leak Zhangluo Global Agent Adapter (server.py, integrations.py, interactive 4-tab workbench UI, and pytest verification suite).
• Integrated Nebius Token Factory (NVIDIA Llama-3.1-Nemotron-70B-Instruct-HF & Nemotron-Nano-8B-v1) with Tavily Real-Time Search, Qloo Taste AI /v2/insights, and PayPal Orders v2 Sandbox.
• Developer Feedback on Nebius Token Factory & NVIDIA Nemotron: Llama-3.1-Nemotron-70B-Instruct-HF on Nebius Token Factory delivered rock-solid OpenAI-compatible JSON schema adherence for multi-constraint household reasoning (p50 latency ~1.1s), allowing us to swap from proprietary frontier APIs with zero prompt regressions while cutting inference cost by >60%.

Built With

• ios
• nebius-token-factory
• nvidia-nemotron
• pgvector
• postgis
• python
• swift
• tavily
• typescript