Toyota Lifestyle Co-Pilot

Inspiration:

When we were looking at how people actually shop for cars, we realized how limiting most car websites feel. You’re stuck clicking through drop-downs and sliders that never really capture what your life looks like — whether that’s sitting in traffic every morning, driving through rough city roads, or trying to fit a stroller in the back.

We wanted to build something that feels more human. Instead of checking boxes, what if you could just describe your day and have an AI understand what kind of car would fit best? That’s where the idea for Toyota Lifestyle Co-Pilot came from — using AI to bridge that gap between real life and car specs.

What it does:

Users can describe their habits naturally — for example, “I commute 40 miles a day through city potholes and take road trips on weekends with my toddler.” NVidia's Nemotron model interprets that free-form description, builds a structured profile, and orchestrates calls to: - The Catalog Service, which scores all Toyota models using specs like MPG, drivetrain, seating, and safety - The Finance Service, which estimates affordability from the user’s income, credit, and down payment
Users can view recommended cars and compare details between 2 or more models - Users can generate a car report, including or excluding certain details such as pricing & costs, performance & fuel economy, safety ratings, and dimensions.
View different Toyota car models in a 3D Augmented Reality to get a better understanding of it's exterior and interior.

How we built it:

We built a full-stack system combining agentic AI with structured data services:

Frontend (UI): a React web app where users type lifestyle prompts or adjust sliders for priorities like budget and eco-friendliness.
Nemotron (backend AI): NVIDIA Nemotron-Nano 9B orchestrates multi-step workflows — extracting user intent, invoking the scoring and finance APIs, and generating natural-language explanations.
Catalog Service: a Python microservice that reads a cars.json dataset containing full Toyota specs and computes preference-based scores.
Finance Service: another backend that runs lease and loan calculations to evaluate affordability.
Integration: Nemotron acts as the reasoning layer, planning which service to call, merging their outputs, and explaining final results.

Challenges we ran into:

Internal Reasoning and Rate Limits Nemotron sometimes spent so much time “thinking” internally that it hit token limits before giving any real answer. We had to reduce its reasoning depth and build a Python fallback that generated recommendations when the model returned blank responses.

Keeping Nemotron as the True Orchestrator At first, it was easy to let our Python functions do all the logic and just have Nemotron summarize. We had to rework the prompts and tool design so Nemotron actually made the decisions — when to call scoring, when to adjust filters, and when to ask follow-ups.

Syncing Chat and Smart Search Results The hardest part was keeping the car cards in the UI consistent with what Nemotron mentioned in the chat. We eventually built a shared data layer so both the conversational agent and the smart search displayed the same recommendations.

Accomplishments that we're proud of

Built a working multi-agent pipeline where Nemotron autonomously orchestrates catalog and finance tools instead of acting like a simple chatbot.
Designed a comprehensive Toyota dataset schema supporting detailed lifestyle scoring (ground clearance, child-seat fit, safety features, etc.).
Implemented explainable AI outputs that show users why each vehicle was recommended.
Created a clean, extensible architecture ready for future add-ons such as dealership appointment booking and image-based road-condition recognition.

What we learned

We learned that building an “agentic” system is very different from just plugging an LLM into an app. Early on, Nemotron felt more like a passive interpreter because it would describe the data we gave it instead of actually coordinating actions. Once we gave it clear rules and boundaries to utilize real car catalog, defined tools, and a well-designed feedback loop, Nemotron shined. By the end, we’d learned how to make an AI agent that doesn’t just talk but thinks, plans, and acts in a way that feels grounded and useful.
Outside of Nemotron, we also learned a lot from using agentic AI tools to help us code. We relied on it to debug tricky backend flow issues, refactor modules, and reason about the architecture when things broke. But we quickly found that we couldn’t just “ask it to fix things”, we had to fully understand each issue ourselves and know the general solution to drive its implementation. It was like pair programming with something that could help write code with you, but only if you knew exactly what you wanted.

What's next for Toyota Lifestyle Co-Pilot

The first is adding vision capabilities using the multimodal Nemotron 12B VL model. Right now, users describe their driving environment through text, but in the future, they’ll be able to upload a photo or short clip of the roads or parking space they deal with every day. The vision model will interpret those images like detecting things like gravel roads, potholes, tight parking spots, or steep driveways and automatically fill in the corresponding environment parameters for Nemotron’s reasoning loop. Instead of guessing what “rough roads” means, the system will see it, quantify it, and adjust car recommendations based on that context.

The second feature focuses on closing the loop from recommendation to action. Once Nemotron suggests a shortlist of cars, it will be able to use location data to find nearby Toyota dealerships, check which ones have that model in stock, and book a test-drive appointment within a user’s preferred time range. The model will fill in all the required booking details from the user —contact info, dealer ID, and car selection—then confirm the appointment directly in chat.