Oteka!

Inspiration

Oteka started from watching friends and family bounce between calorie apps, meal planners, and grocery lists, yet still feel burned out and confused about what to eat. We wanted a single system that could understand real plates of food, respect complex metabolic constraints, and quietly handle the boring logistics in the background. The project was also inspired by recent advances in multimodal models and on-device vision, and a desire to see if we could fuse those into something that feels like a coach, not a spreadsheet.

What it does

Oteka lets you log a meal by simply placing your hand next to the plate, snapping a photo, and letting the system estimate volume, mass, and calories from the image. It tracks pantry items with “entropy-based” decay scores so the app can nudge you to use ingredients before they expire, without nagging notifications. A metabolic advisor layer uses structured rules and LLM reasoning to adapt suggestions to your metabolic state and constraints (for example, Randle cycle, ketosis, or specific medical conditions). Together, this makes Oteka a camera-first food logger, pantry brain, and metabolic co-pilot in one.

AND IT OPTIMIZES FOR BODYBUILDING (Metabolic effects, etc. )

Your reviews for the food (asks you how you felt) integrated into the algorithm.

It also scaffolds ingredients of everything to be more accurate!

How we built it

We built Oteka as a Next.js 15 app with a Capacitor wrapper to reach mobile while keeping a single codebase for web and PWA. Supabase provides PostgreSQL with JSONB, pgvector, RLS, and cron jobs for stateful pieces like pantry entropy, metabolic rules, and workflow logs. The “agentic” workflows are orchestrated through a Vercel AI SDK style layer that chains vision, physics, reasoning, and planning steps.

On the vision side, the default path streams images to a server workflow that uses Gemini 3.0 Flash Vision for segmentation, labeling, and reading nutrition text directly from images. When the connection is weak or users enable data saver, we fall back to an on-device stack using Meta SAM 3 Tiny via onnxruntime-web with MediaPipe depth estimates, running on WebGPU where available. A physics core then converts depth maps and the calibrated hand width (( \text{handwidth_mm} )) into approximate volumes in ( \text{cm}^3 ), which we map to mass using density coefficients from our foods table.

For planning, we implemented a Rust port of a 16-metric NSGA-II engine, compiled to WASM and run in a dedicated Web Worker to keep the UI responsive. This engine balances constraints like macros, pantry inventory, metabolic rules, travel context, and user preferences, with a self-healing loop that relaxes constraints when no solution exists. Around that, we wired user flows like onboarding (hand/credit-card calibration), agentic logging, passive pantry cleanup, and travel menu parsing.

Challenges we ran into

One major challenge was making volumetric estimates reliable in messy, real-world lighting and angles. We had to introduce an AR hand silhouette overlay, confidence scores over depth variance, and a “tilt 45°” second-shot flow to keep errors bounded without turning the experience into a photoshoot. Balancing server-first intelligence with offline resilience was another tricky tradeoff; we had to aggressively gate WebGPU usage, cap NSGA-II iterations, and implement an offline store-and-forward queue with conflict resolution to avoid battery drain and inconsistent state.

Designing the metabolic reasoning layer was also difficult. We needed a schema that could encode mechanisms like “avoid certain macro combinations above thresholds” and still be audit-able, so we ended up with JSONB rules validated with Zod and templated prompts into a reasoning backend. Finally, integrating multiple external APIs (vision, reasoning, barcodes, maps) while staying within rate limits and keeping latency low required careful caching and optimistic UI strategies.

Accomplishments that we’re proud of

We’re proud that Oteka can convert a casual phone photo plus a hand reference into a physics-aware estimate of what you actually ate, rather than relying on manual text search. The hybrid vision pipeline (Gemini 3.0 server-first, Meta SAM 3 fallback) and depth-based volume calculation felt like a genuine step beyond traditional food logging UX. We’re also proud of the WASM NSGA-II planner and self-healing constraint system, which can juggle up to 16 competing objectives without freezing the UI.

On the data side, the entropy-based pantry model—with decay coefficients per food category and a “ghost check” review lane—turns pantry cleanup into a low-friction background process instead of constant notifications. The stateful workflows table and logs give us an auditable trail of what the advisor did and why, which is important for user trust when you’re suggesting changes to their diet.

What we learned

We learned that building a truly helpful food and metabolic assistant is less about a single “smart” model and more about carefully choreographing multiple specialized components. Vision has to respect physics, pantry data has to respect entropy and uncertainty, and metabolic advice has to respect explicit rules and safety boundaries. We also learned the importance of progressive degradation: users should get something useful whether they are online with full GPU, throttled on mobile data, or completely offline.

On the engineering side, we gained experience with WebGPU, onnxruntime-web, and running complex optimization algorithms in Web Workers without hurting the user experience. We also deepened our understanding of how to design JSON-first schemas for metabolic phenomena and conditions so that LLM reasoning is grounded rather than improvisational.

What’s next for Oteka!

Next, we want to make Oteka more collaborative and more personalized over time. That includes household-level pantry graphs, shared entropy-aware shopping lists, and better social features around streaks and progress. On the intelligence side, we plan to expand the metabolic phenomena library, refine density and decay coefficients with real-world feedback, and experiment with lightweight on-device models that can handle more of the pipeline privately.

We also want to invest in explainability: surfacing “why this recommendation” in plain language and giving users control over constraints they want Oteka to relax or enforce. Finally, we’re exploring integrations with wearables and glucose/HRV data to allow the advisor to adapt plans not just to meals and pantry state, but to how your body is actually responding.