DownShift

Hardware
Hardware
Hardware

Inspiration

Buying a car is one of the biggest financial decisions many people make in their lives, yet understanding the true cost of ownership is rarely straightforward. Affordable vehicles are increasingly difficult to find over the last few years, interest rates can significantly change monthly payments, and unexpected changes in income can turn a manageable loan into a heavy financial burden.

We built DownShift to help people explore these decisions before committing to them. Instead of simply displaying financing offers, DownShift puts customers in realistic situations and asks a more important question: What happens when life changes after you sign? Our goal is to make Toyota's buying, leasing, and financing experience more transparent, interactive, and approachable for customers.

What it does

DownShift is an AI-powered automotive financing simulator combining a web application with an interactive hardware prototype. The web application lets users explore Toyota vehicles, compare financing options, practice negotiating, and understand how financial decisions affect their simulated banking health. Users can explore 20 predefined customer scenarios aproved by Toyota Financial Services or describe their circumstances through text and voice. For example:

"I just got a new job, and my credit score has improved. Am I still getting a good deal on my auto loan?"

Using NVIDIA Nemotron, DownShift interprets the customer's situation and provides personalized explanations of their financing options, assumptions, and trade-offs. Capital One's Nessie API supplies simulated banking information to help users understand how vehicle payments interact with their broader financial situation.

We also built a physical prototype powered by an ESP32, featuring an OLED display, rotary encoder, and RFID reader. Scanning RFID event cards introduces financial changes, such as reduced income or lower interest rates, with more to come in the future. Users can adjust the intensity of these events and explore how their decisions change under different circumstances. Basically, the web application represents DownShift's immediate customer-facing experience, while the hardware prototype explores a more tangible, gamified future for financial education.

How we built it

We combined automotive data, financial simulation, conversational AI, and embedded hardware into one system. Toyota vehicle data provides the foundation for exploring vehicle selections and financing scenarios. Nessie provides synthetic banking records for modeling customer finances without requiring actual banking credentials. Nemotron powers personalized financial explanations and supports conversational interaction through speech-to-speech AI. The ESP32 hardware prototype uses an RTOS-based architecture to coordinate inputs from an RFID reader and rotary encoder while updating an OLED display. We also used Claude, Gemini, and Codex throughout development for rapid prototyping and accelerating implementation and integration of systems.

Challenges we ran into

One of our biggest challenges was integrating NVIDIA Nemotron's real-time Speech-to-Speech AI Model (11BVoice) into DownShift. What initially appeared to be a straightforward integration quickly turned into hours of troubleshooting Python dependencies, PyTorch compatibility, and NVIDIA Brev environment configurations. Even after resolving several setup issues, we encountered inference limitations, including S2SPipeline: maximum concurrent streams reached, which disrupted real-time voice interactions. Working through these issues forced us to rethink how we managed inference resources, handled errors, and maintained a reliable user experience.

Another major challenge was balancing ambitious features with a 24-hour development window. For some, it's their first or second hackathon. For others, we had the benefit of an additional 12 hours of hacking. We envisioned DownShift as an interactive, gamified automotive financing experience combining real-world financial simulations and conversational AI into the physical world. Integrating these components required careful prioritization to ensure each technology contributed meaningfully to the customer experience.

We focused on building a strong simulation foundation while bringing gamification to life through our ESP32-powered hardware prototype, complete with RFID-triggered financial events and rotary encoder controls. This allowed us to demonstrate both an immediately practical financial tool and a vision for a more engaging, interactive future.

Accomplishments that we're proud of

We're proud of bringing together three traditionally separate experiences: automotive shopping, financial education, and physical interaction. Rather than simply building another chatbot that simply recommends a car, we developed a system that lets customers explore how financial decisions change under different circumstances. Our biggest accomplishment is making complex financial trade-offs easier to understand through interactive scenarios, personalized explanations, and a tangible embedded prototype.

What we learned

One of our biggest learning experiences was integrating NVIDIA Nemotron's real-time Speech-to-Speech capabilities into DownShift. What initially seemed like a straightforward AI integration became a significant engineering challenge involving Python environment dependencies, PyTorch compatibility, cloud infrastructure configuration on NVIDIA Brev, and inference errors such as S2SPipeline: maximum concurrent streams reached. These challenges taught us that deploying real-time conversational AI requires much more than connecting to a model. Managing inference resources, handling concurrent requests, and designing reliable fallback mechanisms are just as important as the AI itself. We also learned how to combine Nemotron with structured financial data from the Capital One Nessie API to provide context-aware automotive financing explanations.

On the hardware side, working with the ESP32 taught us how to coordinate multiple peripherals using an RTOS-based architecture. Most importantly, we discovered how significant changes in income, credit scores, and interest rates can affect the affordability of buying, leasing, or refinancing a vehicle. DownShift helped us understand how interactive simulation can make these financial trade-offs more accessible to everyday customers.

What's next for DownShift

Youth Gamification and Controller Support: Expand the prototype into a more complete game with controller integration, interactive challenges, and financial education designed for younger drivers.

Proactive Financial Alerts: Notify customers when changing circumstances create opportunities to reconsider their financing arrangements, such as potentially benefiting from refinancing after an improvement in creditworthiness.

Total Cost of Ownership: Incorporate insurance, depreciation, maintenance, repairs, and long-term operating costs into vehicle comparisons.

Early Warning and Specialist Handoff: Identify simulated indicators of financial hardship and explore how customers could transition from self-service guidance to assistance from a Toyota Financial Services specialist.

Our long-term vision is for DownShift to become a financial companion that helps customers make informed decisions not just when purchasing a Toyota, but throughout their ownership journey.