Aloud

• 

  One press, and it tells you what's in front of you. And when it isn't sure.

• 

  On a real phone: hazards first, with clock positions, and it says when it isn't sure.

• 

  Reading a handwritten note word for word, and saying when the page is cut off.

• 

  Where are my keys? The camera confirms each placement, and the memory stays on the phone.

• 

  Numbers are spoken only when two frames agree. Otherwise Nemotron 3 Ultra compares the reads, or Aloud abstains.

• 

  The evaluation reports every rate with its numerator and denominator, on dev and held-out splits.

• 

  Four Nemotron models on Nebius Token Factory, one job each. Code, not the model, decides what is said.

Inspiration

About 2.2 billion people live with a vision impairment (WHO). Camera assistants already exist, but they share one habit that matters more than anything else for a blind user: when the picture is bad, they guess, and they say the guess in the same confident voice as a fact. If you can't see the label, you can't catch the mistake. A wrong price, a misread appointment time or an invented object is worse than no answer.

So we built Aloud around one rule: say what you see, say how sure you are, and when you can't tell, say how to get a better picture.

What it does

Aloud is a phone web app (PWA) with one big button. Wear the phone on your chest or hold it, and press.

• Describe: a few short sentences, hazards first, with clock positions and rough distance ("Careful: I think there's a white mug, possibly hot, at 5 o'clock, within reach").
• Read: text exactly as written, in reading order. A number is spoken as certain only when two frames agree. If the page is cut off, it tells you which way to move.
• What is this: the main object you're holding and its label.
• Remember and find: "I'm putting my keys on the table." The camera checks they really are your keys, and the placement is stored on the phone only. Later: "Where are my keys?" "Your keys: on the table, confirmed today at 8:55." It works offline too.
• Appliance helper: reads the model number, finds the manual with Tavily, and gives one step at a time, checking each step with the camera.
• Simple mode for people with cognitive disabilities: sentences of at most 15 words, one step per turn, encouragement, and "Did that work?"
• Look harder: re-reads the last picture with the vision model's reasoning turned on.

When the picture is too dark, blurry or cut off, Aloud doesn't guess. It never says a road is safe to cross, never identifies people, and never gives dosing advice. It has a screen-reader mode for VoiceOver and TalkBack, works with a Bluetooth clicker or a keyboard, and anyone can try every skill on a laptop with the bundled sample photos.

How we built it

Four NVIDIA Nemotron models on Nebius Token Factory, each with one job:

The model never writes what Aloud says about an image. Code decides what is safe to say:

• Quality gate: if the model can't answer, Aloud speaks only fixed camera guidance ("Too dark. Try a light, or move toward the window.").
• Confidence wording: at 0.8 or above, said plainly; 0.5 to 0.8 becomes "I think …, but I'm not sure"; below 0.5 is dropped and counted.
• Two-frame numbers: Read takes two frames 400 ms apart. If they disagree, Ultra judges between the two structured reads, and code rejects any answer that neither frame produced. Otherwise Aloud abstains: "I see a number but can't confirm it. Move closer."
• Forbidden claims (people, traffic safety, medicine) are checked in code on every request and every output.
• Memory lives in IndexedDB on the phone. A placement becomes trusted only when the camera confirms the object.

Where Token Factory helped: one OpenAI-compatible API for all four models, so changing a model's role is changing an id. Strict json_schema and tool calling on the same endpoint. reasoning_effort control to keep vision fast. Usage in every response powers the per-answer cost shown in the app, the daily budget cap and the cost-per-hour in our evaluation.

Stack: React + Vite PWA, a stateless Hono API on Vercel, Tesseract.js for offline reading, Tavily for manuals, Vitest (229 tests) and an evaluation harness that prints every rate with its numerator and denominator, on a dev split and a held-out split.

Challenges we ran into

• Latency from West Africa. A TLS handshake to Token Factory took about 2.4 s from our connection, more than the vision read itself. On the deployed server, Describe takes 1.3 to 1.8 s server-side.
• Super VL thinks by default, spending 1.2K to 1.9K hidden tokens per call. Turning reasoning off made every press faster, and we kept reasoning as a deliberate "Look harder".
• Models add their own formatting. Lightning sometimes wrote /1/ or !\ in front of steps, which a screen reader reads aloud. We now clean every step in code, with tests.
• Honest abstention is hard to tune. Too strict and Aloud never answers; too loose and it guesses. The two-frame rule for numbers, with Ultra as a judge that can't invent a third answer, was the balance we found.

Accomplishments that we're proud of

• It works on a real phone: the demo video shows it describing a table, reading a handwritten note and finding keys by voice.
• In our evaluation, Read spoke zero words or numbers that weren't on the page, and there were zero forbidden-claim violations, on both splits. Perception costs about $0.10 per hour of use (120 presses).
• We publish the weak spots too: Describe struggles with abstract drawings, and every set is small and synthetic, so the numbers show the pipeline works, not that it's reliable.

What we learned

• A smaller set of things said with the right confidence is more useful to a blind user than a longer, confident description.
• Safety rules belong in code, not only in prompts: the model proposes, code decides.
• Giving each model one job made the system cheaper, faster and easier to test.

What's next for Aloud

1. Train a Nemotron 3 Nano 4B fast lane on Nebius Serverless AI. The teacher data (949 examples, checked by code) and the job and endpoint scripts are ready; the training run hasn't happened yet.
2. Test with blind and low-vision users and change Aloud based on what they say.
3. A real-photo evaluation set of at least 40 phone photos.
4. Run the distilled model on the phone, so routing and simple-mode plans work offline.
5. More languages. Aloud is English only today.

Built With

• hono
• indexeddb
• nebius-token-factory
• nvidia-nemotron
• playwright
• pwa
• react
• tavily
• tesseract.js
• typescript
• vercel
• vite
• vitest