Create a plan
My Plans
Insights
Suggest Coach
GAP-AI Hardware
Plan Outcome Tracking
System Architecture
Voice Interaction Workflow

Inspiration

People make plans every day, but what they intend to do and what actually happens are often very different. To-do apps record tasks, but they rarely help us understand our real behavior.

We built GAP-AI around one question: What is the gap between our intentions and reality?

What it does

GAP-AI is a proactive AI personal life assistant. You tell it your plan in everyday language, by typing or by voice, and it turns that into a structured plan. You then record what really happened: done, postponed, or skipped. Over time, GAP-AI measures your adherence, finds patterns in your behavior, and gives you coaching and recommendations.

Natural-language and voice plan creation (web and ESP32-S3 device)
Outcome tracking and behavioral analytics
Personalized onboarding and preferences
Three kinds of recommendations:
- Behavior-based: derived from your recorded plans and outcomes.
- Contextual: based on goals, deadlines, availability, upcoming plans, and weather.
- Discovery: a separate catalog of new ideas. These are not generated from your past behavior, and we never present them as if they were.

How we built it

GAP-AI is a full-stack system: a React web app, a FastAPI backend, a Supabase/PostgreSQL database, hosted AI inference, and an ESP32-S3 hardware prototype.

NVIDIA Nemotron on Nebius Token Factory

We did not rent GPUs or host a model ourselves. Our backend calls NVIDIA Nemotron through Nebius Token Factory's hosted, OpenAI-compatible API (nvidia/Nemotron-3_5-Lightning, Chat Completions, using our initial credits and free allowance). Nemotron does three jobs:

Understands natural language. "I want to study calculus tomorrow afternoon for two hours" becomes intent, title, date, start time, duration, and category.
Produces structured output. Every model result is validated against our JSON schema (ai/schema.json) before it can become a record. The backend can tell a valid plan from a request that needs clarification or is unsupported.
Writes coaching narratives. It explains metrics in human language and suggests a practical next step.

Our key design decision: Python calculates, AI explains

Nemotron never computes our official statistics and never touches the database. Our Python backend calculates the numbers:

Adherence rate: done / (done + postponed + skipped), so it only covers plans with a reported outcome. Plans without a reported outcome are counted separately, and we do not present the rate as "the completion rate of all plans".
Counts of completed, postponed, skipped, and unreported plans.
Patterns by weekday, hour, and category, plus postponement patterns.

Nemotron only receives these verified numbers and turns them into advice. The language model is never the source of truth for analytics, so the results stay predictable and trustworthy.

Voice, backend, and data

Speech recognition: Groq whisper-large-v3. The ESP32 records mono, 16 kHz, 16-bit audio (up to 10 seconds) and the transcript enters the same workflow as typed text.
Spoken responses: Groq Orpheus TTS, with TTS.ai (Kokoro) as a fallback. If speech fails, the plan is still saved.
Backend: Python 3.11, FastAPI, Pydantic, HTTPX, with bounded timeouts and deterministic fallbacks, so an unavailable AI provider does not break the app.
Database and login: Plan and activity data flow through our backend to Supabase (PostgreSQL). The web app also uses the Supabase Auth client for Google sign-in when it is configured. The AI model has no direct database access. We also built isolated demo sessions so reviewers can try the app without touching anyone's data.
Frontend: React 19, TypeScript, Vite. Users review every AI-generated plan before saving it.
Weather: Open-Meteo. Our MVP uses a fixed default location (Üsküdar) instead of claiming to know the user's live location.

The hardware

The ESP32-S3 prototype (ESP-IDF, firmware in C) uses an INMP441 microphone, a MAX98357A amplifier with speaker, an SSD1306 OLED display, and physical buttons. It connects to Wi-Fi, records audio, talks to the backend, and shows the result. The web app and the device share the same backend voice contract.

Built entirely on free tiers

Netlify (frontend), Render (backend), Supabase (database), Nebius and Groq (AI) all run on free tiers or initial credits. Note: the Render free backend sleeps when idle, so the first request may take a little while to wake up.

Challenges we ran into

Five independent systems had to work together: a web app, a backend, external AI providers, a database, and embedded hardware. We solved this with a shared API contract and JSON schemas, so the frontend, backend, AI, and firmware all spoke the same language.

Two lessons stayed with us:

AI should not be the source of truth for numbers. That is why analytics live in deterministic backend code.
A successful HTTP response does not prove the hardware works. An API call can return 200 while the speaker stays silent. We learned to test the API, the firmware, and the physical audio separately.

What we learned

Building GAP-AI taught us how to bring different engineering disciplines together. We learned to integrate hosted AI models, design reliable APIs, validate structured data, work with embedded systems, and troubleshoot real-world hardware and software problems.

We also learned that building a useful AI product is not just about choosing a powerful model. It requires reliable architecture, clear data flow, careful testing, and thoughtful decisions about what AI should and should not do.

Our Journey as a Team

We are 10 first-year Electrical and Electronics Engineering students, and GAP-AI is one of our first major attempts to build a complete system that connects software, AI, and physical hardware.

It was not easy. We had incompatible code, API integration problems, and unexpected hardware behavior. Sometimes solving one bug revealed another, and a "simple" fix took hours late at night. Getting the backend, AI services, web app, and ESP32 to communicate reliably took repeated testing, patience, and teamwork.

But every error taught us something. Along the way we learned how to divide responsibilities, integrate different technologies, and turn an ambitious idea into a working prototype.

For us, GAP-AI is more than a project. It is the result of learning, experimenting, failing, and building together.

What's next

Comparing adherence against previous periods to show real trends over time
Better physical device verification and richer embedded interactions
A more proactive, personalized coaching experience
Our long-term vision: moving beyond task management to help people build a realistic understanding of their own habits

GAP-AI turns plans into measurable outcomes, and outcomes into better decisions.

Try it: https://gap-ai-demo.netlify.app

Built With

c
esp-idf
esp32-s3
fastapi
groq
i2s-audio
nebius-token-factory
netlify
nvidia-nemotron
open-meteo
postgresql
python
react
render
rest-api
speech-to-text
supabase
text-to-speech
tts.ai
typescript
vite
whisper

Submitted to

Nebius x NVIDIA Global AI Hackathon

Created by

I contributed to the AI and backend development of GAP-AI, focusing on integrating NVIDIA Nemotron through Nebius Token Factory.

My responsibilities included developing the LLM client, implementing error handling and timeout mechanisms, and working on prompt engineering, evaluation, and testing to improve the accuracy and reliability of AI-generated responses.

I also worked on a contextual recommendation system that considers weather conditions and users' past planning behavior to provide more relevant suggestions.

Throughout the project, I gained hands-on experience in AI integration, API development, debugging, and collaborative software development. This experience helped me strengthen my technical skills, improve my problem-solving abilities, and gain a better understanding of how different technologies work together in a real-world application.

ulkusena uzun
I worked on developing audio recording and transmission systems for the ESP32-S3. I implemented audio recording using the INMP441 microphone and I2S protocol, storing recordings in WAV format. I also worked on uploading audio to the server via HTTPS and transferring transcription results to the display module. Additionally, I contributed to improving microphone gain, noise handling, and recording stability.

Eren Tepe
I contributed to the development of GAP-AI, focusing on backend, AI integration, and hardware communication.

During this project, I gained hands-on experience with FastAPI, PostgreSQL, AI model integration, ESP32 development, and voice-based interaction systems. I learned how to connect different technologies and turn separate components into a working product.

One of my biggest learning experiences was working with hardware-software integration, especially connecting the ESP32 device with our backend and implementing voice communication.

Beyond coding, I also improved my teamwork, problem-solving, project management, and technical communication skills.

This project taught me a lot about building real-world AI applications and working as part of an engineering team.

Hamza Emin
I contributed to the frontend development and web deployment for the GAP-AI project. My key contributions include building the interactive Insights Analytics Dashboard to visualize adherence metrics, day/hour distributions, and postponed targets. I ensured full mobile responsiveness down to 375px screens and implemented robust UI state management, including Loading Skeletons, Error Boundaries, and Empty States. Additionally, I configured the Netlify continuous deployment pipeline with SPA fallback routing, integrated the staging API environment, and recorded the final web demo clips for the project showcase.

sevdebetulkutlu41-code
I worked on the backend, the database, the ESP32 firmware and project coordination.

Backend and database
Built suggestion storage: a Supabase migration for a suggestions table (one row per plan), with row-level security, owner checks and cascade deletes. This makes GET /plans/{id}/suggestion return the same suggestion every time. I also added a SQL verification script and wired both into CI.
Built the POST /plans/{id}/outcome endpoint.
Wrote an admin token generator script for the database.
ESP32 firmware
Set up the first firmware skeleton with Wi-Fi and HTTPS, and documented how to reach a local backend on Linux (firewall setup).
Fixed a bug where audio_record stopped recordings after about 20 ms.
AI configuration: Pointed .env.example at a working Nemotron model and updated the LLM tests to match.
Docs and project setup: Wrote CONTRIBUTING.md, sorted the architecture, plan and task docs, and added the API and AI schema specs.

niko3x
I contributed to the development of GAP-AI, focusing on frontend development and user interface implementation.

During this project, I gained hands-on experience with React, TypeScript, Vite, API integration, and frontend application architecture. I worked on building the web interface and connecting it with the backend to create a functional and user-friendly experience.

My main contributions included setting up the React application, implementing authentication and demo session management, developing the natural language plan creation workflow, and building the plan listing and outcome reporting interfaces.

One of my biggest learning experiences was connecting the frontend with backend APIs and translating application requirements into functional user interfaces. I also worked on handling different response states, such as clarification requests, unsupported inputs, and successful plan creation.

Beyond coding, I improved my problem-solving, teamwork, debugging, and collaborative development skills through Git and GitHub.

This project taught me a lot about building real-world web applications, integrating frontend and backend systems, and contributing to a collaborative AI engineering project.

Taylan Burmalı
I contributed to the development of GAP-AI, focusing primarily on backend development, AI integration, and voice processing systems.

During this project, I gained hands-on experience with Python, FastAPI, Groq Whisper, API development, and backend testing. I worked on implementing speech-to-text functionality and developing voice-based communication between the ESP32 device and the backend.

One of my main learning experiences was developing the voice processing system, especially integrating Groq Whisper for speech recognition, implementing API endpoints, and troubleshooting audio transcription and timeout issues.

Beyond coding, I improved my problem-solving, debugging, teamwork, version control, and technical communication skills through Git and GitHub collaboration.

This project gave me valuable experience in developing real-world AI-powered applications and understanding how different software and hardware components work together in an engineering project.

Muhammet Burak VAROL
HFY61-coder
Mustafa Kerem Beriş