Nexus

architecture
Semantic Graph
Nexus Logo

Inspiration

If you forget a toothbrush on a vacation, it’s annoying. If you pack the wrong gear for a disaster relief mission, it’s dangerous.

Right now, logistics coordinators for high-stakes deployments—whether it’s search-and-rescue or remote medical aid—are still relying on spreadsheets and mental math. They have to mentally juggle weight limits, material science (e.g., knowing that "cotton kills" in hypothermia scenarios), and cross-domain utility (a mylar blanket is both shelter and medical gear).

We asked a simple question: Why can we search the entire internet in milliseconds, but we still have to rummage through bins to find our own stuff? We built Nexus to digitize the physical world and treat packing like the mathematical optimization problem it actually is.

What it does

Nexus is a search engine and logistics officer for your physical gear.

It "Sees" Physics: You don't type in data. You just snap a photo. Nexus analyzes the image to understand not just what an item is, but what it does. It extracting factors such as thermal ratings, materials, waterproofing, and medical utility automatically.

It Understands Context: You don't search for "flashlights." You search for "lighting for a power outage in heavy rain." Nexus understands the semantic difference between a keychain light and a tactical floodlight.

It Reasons for Safety: A cross-domain synthesis layer actively rejects dangerous item selections (e.g., packing cotton clothing for cold-weather survival) and explains why, flagging critical gaps like "No water purification detected."

It Solves the Knapsack Problem: You give it a mission ("72-hour cold-climate medical response") and constraints ("Bag with 15kg Weight Limit"). Nexus uses a constraint solver to mathematically prove the optimal loadout, ensuring you have enough medical gear without blowing your weight limit on heavy batteries.

How we built it

The "Eyes" (Vision & Extraction): We pipe camera feeds from Flutter directly to GPT-5. We prompt-engineered it to act like a materials scientist, extracting structured JSON metadata about an item's capabilities and failure modes via strict schema validation.
The "Brain" (Multimodal Embeddings): Standard text embeddings weren't enough. We used Voyage AI’s voyage-multimodal-3.5 model to generate 1024-dimensional vectors from the interleaved image plus serialized context text. This eliminates the modality gap, ensuring the vector representation includes both visual textures and explicitly extracted metadata.
The "Memory" (Vector DB): We utilized Supabase with the pgvector extension and HNSW indexing. It handles both relational metadata and lightning-fast cosine similarity search in a single instance.
The "Math" (Optimization): We use the Google OR-Tools CP-SAT solver to run a bounded knapsack optimization. It balances diversity (e.g., minimum medical items, tag requirements) against weight limits across multi-container bin packing scenarios.

Challenges we ran into

Dangerous Recommendations: Naive vector search returned items that were semantically similar but contextually dangerous (e.g., cotton for cold weather). We fixed this by explicitly extracting unsuitable_contexts and failure_modes during vision analysis, embedding them as first-class features.
A user might ask for "3 medical items" when they only own 2. A standard solver just crashes and says "Infeasible." We had to write a wrapper that progressively relaxes constraints (e.g., dropping the item count requirement) so the app can say, "I couldn't find 3 items, but here are the 2 you have," instead of just throwing an error.
Weight Estimation without Scales: Most users don't know the exact weight of their gear. We built an AI weight estimation system that maps vision-inferred categories to gram estimates, allowing the knapsack solver to function without precise data.

Accomplishments

It actually works in 5 seconds: We managed to get the full pipeline—snap photo → GPT-5 Vision extraction → Multimodal Embedding → Database Write. Watching a raw image turn into a queryable, data-rich inventory item feels like magic every time.
Cross-Domain Reasoning: We didn't hardcode rules. The system learned that a sleeping bag is useful for medical shock treatment purely through semantic similarity.
Optimal Packing: We aren't just guessing. When Nexus tells you what to pack, it's mathematically the best possible combination of items for your specific weight limit.
Visualizing the "Latent Space": We didn't just store vectors; we visualized them. We built a 3D force-directed graph (using a WebView bridge to React Three Fiber) that lets users physically "fly" through their inventory. You can visually see how "Batteries" cluster near "Flashlights" but far from "Socks," turning abstract math into a tangible map.

What we learned

Optimization > Search: Semantic search is cool, but it's easy. The real value is in the constraint solver. People don't just want to find their gear; they want mathematical proof of what they should take.

Vision Models are Materials Scientists: We were surprised by how much physical intuition GPT-5 Vision has. It correctly identified that a specific fabric looked like ripstop nylon without us telling it.

What's next for Nexus

Team Mode & Collaborative Inventories: Shared loadouts for search-and-rescue teams or expedition groups with role-based access (e.g., "I have the tent, you bring the stove").

Hardware Integration: Connecting to Bluetooth luggage scales to track weight in real-time as you drop items into the box.

Historical Analytics: Letting users rate their loadout post-mission ("The heavy sleeping bag was overkill") so the model learns preferences and improves future packing recommendations.