Questions
Scanner

Toyota Next Move

Toyota Next Move is our HackTX 2026 prototype for Toyota Financial Services. We wanted to help a customer answer a simple question: what should I do next with my Toyota?

The idea is a guided vehicle walkaround. The app helps someone collect useful camera evidence, shows what was covered, and explains what is still unknown. It then uses the scan results with sample customer, vehicle, loan, and lease data to suggest one next step.

What the app does

The user opens the camera and taps Start Scan. From there, the app guides the walkaround and checks each frame for quality and useful evidence.

The scan is split into 12 regions. A region can be covered, partial, blocked, or unseen. The next prompt changes as the user moves around the car. Low quality frames do not count. If the app does not have enough evidence, it leaves the region unknown instead of treating it as clean.

After the scan, the user sees a vehicle view and any visible findings. The longer term goal is to place findings on real vehicle surfaces. That surface anchor work is still on our roadmap.

A normal material state only means that we did not see a visible issue in the captured evidence. It does not mean that the vehicle is mechanically healthy.

How we built it

The iOS prototype uses SwiftUI and AVFoundation. ARKit can provide tracking and pose information. The capture layer produces ScanSession and CoverageRegion records. The perception layer turns evidence into findings. The product layer is planned to combine those findings with a sample customer and agreement profile and produce one Recommendation.

We support two inputs:

Live camera capture, with checks for blur, exposure, motion, and evidence quality.
A deterministic replay made from Toyota video frames and fixed JSON fixtures.

The replay is important for development and judging because the team can run the same scan every time. It includes covered, partial, and unseen regions, so it also tests the uncertainty behavior.

The parts share JSON contracts for scan sessions, regions, findings, vehicle profiles, agreements, and recommendations. This keeps the capture, perception, product, and QA work connected without making them depend on each other's implementation details.

Where the model fits

There is an optional private backend for visual perception. It samples a JPEG at most once every two seconds and sends it to a vision model using structured output. The response is limited to vehicle presence, view usability, recognizable regions, and short guidance.

The app only records a region after two different consecutive frames identify it with confidence of at least 0.7. The requested region is not sent as a hint. Unclear views, duplicate frames, and contradictory responses cannot create coverage.

This service does not diagnose hidden mechanical or safety problems. We plan to evaluate it on labeled fixtures by tracking wrong region accepts, unknown rates, and response time. The OpenAI key stays on the backend and is never shipped in the iOS app.

The vehicle reveal

The vehicle starts as a neutral gray model. As evidence is collected, observed areas can reveal their normal material. Partial coverage stays partial. Unseen and uncertain areas stay neutral or muted. A finding can sit on top as a small visual accent.

The current geometry is temporary. A cleaned Toyota like asset, named surface anchors, and a reveal that follows the mesh are still future work.

What we learned

The replay path was one of the best decisions we made. It gave us a reliable end to end demo while live capture and the final 3D asset were still changing.

We also learned that uncertainty needs to be visible. A partial scan is useful information. It tells the customer what to do next and gives a service workflow a clear place to continue.

Finally, the recommendation has to stay connected to evidence. A hard coded answer can make a demo look finished, but it does not prove that scan findings and customer context are actually joined.

What is next

Verify continuous live progression on the current iPhone build.
Replace the temporary geometry with a cleaned Toyota like asset.
Attach one evidence backed finding to a real surface anchor.
Connect scan findings to sample customer, vehicle, and loan or lease scenarios.
Run the full demo from Start Scan through the final personalized next step.

Toyota Next Move is a prototype, not a remote diagnostic system. It is our attempt to make a vehicle walkaround more useful while being honest about what the camera can and cannot tell us.

Demo notes

For a repeatable demo, use the deterministic replay. It includes covered, partial, and unseen regions. The model backed perception bridge is for private development use. Keep its backend secrets on the server.

Short description

Toyota Next Move is an iPhone prototype that guides a vehicle walkaround, makes coverage and uncertainty visible, and connects visible findings to one clear next action.