Inspiration

Minutes after a minor car accident, most people are shaken, standing on the side of a road, and trying to remember what to do. Move the car or not? Who to call? What to photograph? What should I not say? The answers exist, but they are scattered across insurance leaflets and forum posts, and nobody reads them while their hands are shaking. [Add a sentence about the team's own experience or someone you know.]

We wanted something that works like a calm passenger: it tells you one thing at a time, out loud, and gets out of the way.

What it does

ScenePilot is a mobile-friendly web app that guides a driver through the first minutes after a minor accident in the US.

Always-visible 911 button. The first thing on screen, at every step.
Two safety questions. "Is anyone hurt, or do you see smoke, fire, or fuel?" and "Can your vehicle be moved out of traffic?" Answers can be spoken or tapped. If the person is unsure, we treat it as yes.
One short action list, read aloud, instead of a long checklist.
Photo coaching. The user photographs both vehicles, the other car's plate, the road and signs, and the other driver's insurance card. A vision model checks each photo and, if something is missing, asks for a retake with a specific reason.
Conversation watch. While the user talks to the other driver, the app listens in the browser and plays a short reminder when it hears risky phrases, for example admitting fault or agreeing to settle in cash. Nothing is recorded or stored.
State notes and a plain summary of what was collected, with each item labeled by where it came from (user's statement, AI observation, or unconfirmed).
Free questions. An agent answers follow-up questions and never overrides the safety rules.

How we built it

Rules decide, AI observes. The safety level, the order of steps and every spoken instruction come from plain code and fixed sentences that we can read and test. The models only look at photos and answer questions. We chose this because a language model should not be the one deciding whether someone needs 911.

NVIDIA open models on Nebius Token Factory

Where	Model	Used for
Photo analysis	NVIDIA Nemotron vision-language model	Checks each photo against what the step asked for and reports what is missing
Free questions	NVIDIA Nemotron 3 Super (tool-calling agent)	Answers questions with tools, behind a guard that keeps safety-critical answers on the rule path
Spoken guidance	NVIDIA Magpie TTS	Every fixed instruction is synthesized ahead of time into audio, so the key sentences play instantly and never depend on a live call

All model calls run through Nebius Token Factory.

Stack. Python and FastAPI with an in-memory session state machine, a vanilla JavaScript front end (no framework), and an Android wrapper. Speech recognition uses the browser's built-in SpeechRecognition. The conversation watch is client-side only: rules are served by the API and matched in the browser.

Safety and cost controls. Per-IP rate limits protect the photo, question and session endpoints. The automated test suite (203 tests) covers the safety rules, the session flow, the audio manifest, the watch matcher and API behavior.

Challenges

Too long to be usable. Our first flow had many questions and long instructions. A stressed person will not sit through it. We cut it down to two safety questions and one list, moved 911 to a permanent button, and let people answer while the voice is still reading.
Speed vs. naturalness. Reading faster seemed like an easy fix, but 1.3x speech sounded unnatural. We went back to normal speed and shortened the content instead.
Voice in a noisy place. Hearing a command over the voice that is currently speaking is hard. We only cut the reading short on a clearly understood command and ignore other speech while it plays.
Staying honest about what the AI knows. We label every item in the summary by its source and keep the AI out of safety decisions.
Design. Our first UI looked generic. We redesigned it as a calm road-sign style (navy and amber, large type, one menu, full-screen questions).

What we learned

With a frightened user, less is better: every sentence has to earn its place.
Pre-synthesizing fixed speech makes the most important moments fast and reliable, and leaves the models for the places where they add something.
A small, readable rule set is easier to test and to trust than a prompt.

Limitations and what's next

ScenePilot is not legal or medical advice. It is built for minor accidents in the US, and the state-by-state notes are summaries that have not yet been reviewed by a professional. The conversation watch uses simple phrase rules, not understanding. We have not yet run it in a user study. Next we want to review the state notes with an expert, test with real drivers on real phones, and add more languages.