Artlog: Museum Art Scanner

Point your iPhone at a painting. Artlog names it in about a second, then tells its story with sources and an audio guide.

Comment

Inspiration

Wall labels are small and say little about the work. Audio guides cover a handful of highlights and often cost extra. And the photos we take of paintings end up in the camera roll with no name attached.

We wanted the phone already in your pocket to be a guide for whatever painting is in front of you, not just the famous ones, and to remember the visit for you.

What it does

Point your iPhone at a painting and Artlog names it in about a second: title, artist, date, medium and museum, out of about 400,000 artworks from open museum collections and Wikimedia Commons. Then:

  • About this work: three or four sentences, each linked to the page its fact came from.
  • Listen: a 40- to 90-second audio guide with a transcript, written from the same research.
  • Wall-label fallback: if Artlog doesn't know the work, photograph the label. Artlog reads it and checks it against the museum's records, and the next visitor gets an instant match.
  • Journal: saved works land on the visit where you found them. Hang favorites on your own wall and share visits with friends.
  • Explore: current and upcoming exhibitions near you, read from each museum's own website.

How we built it

  • iPhone app: Expo and React Native. NVIDIA's open C-RADIOv3-B vision model runs on the iPhone's Neural Engine and turns each painting into a compact fingerprint, so a scan sends a fingerprint, never the photo.
  • Backend: Convex matches the fingerprint against the catalog of about 400,000 artworks, using Pinecone vector search plus a geometric check, and makes every model call. The phone never calls a model.
  • NVIDIA Nemotron on Nebius Token Factory handles every language task, each on the size the job needs:
    • Nemotron 3 Ultra writes the audio guide, the hardest writing job.
    • Nemotron 3 Super reads wall labels, pulls quoted facts from museum pages and lists exhibitions.
    • Nemotron 3.5 Lightning writes the short "About this work" and screens shared text.
  • Where Token Factory helped: one OpenAI-compatible API serves every Nemotron size, so moving a job to another model means changing one model ID, and we ran the same evals across candidates. Strict JSON-schema output returns answers in a fixed shape the backend can check against its sources.
  • Also used: Tavily finds museum pages, and OpenAI's gpt-4o-mini-tts voices the guide. Token Factory is the only Nebius service Artlog uses.

Challenges we ran into

1. The crop, not the model, capped recognition. We replayed 937 real camera photos from Wikimedia Commons through the app's crop path. In 278 of them, the cropper picked a rectangle covering under 10% of the photo, a detail inside a painting that filled the frame, and only 12 of those found the right work. Trying the whole frame first when the detected rectangle covers under 30% of the photo raised confident correct answers from 345 to 630 in that replay. Confident wrong answers rose from 7 to 15, mostly copies of the same composition.

2. A guide that wrote only whitespace. One production guide request used 12,000 tokens in 26 seconds and returned no usable script. Structured output enforces the schema token by token. Our schema required at least six story sentences, but the research supported only three. After the third, the grammar forbade closing the array, so the model kept choosing the only other legal token: whitespace. Replaying the request stalled 6 times out of 20. The fix is in the app: a 3,000-token cap, plus one retry with a relaxed schema in which every section may stop after one sentence. The relaxed schema passed 10 of 10 replays.

3. Real citations don't make a sentence true. With strict schemas and valid fact IDs, a story sentence still invented an exhibition venue while citing real facts. So every guide sentence now carries the exact words of its evidence, and the backend checks them against the research before anyone hears the guide. A failed script is retried once, with its problems named. Getting "exact words" right took its own round: the model joined fragments with ellipses, dropped catalog cross-references such as "(see cat. 45)" and dropped accents, so the check now compares clause by clause with accents folded.

Accomplishments that we're proud of

  • It's live. Artlog is free on the App Store and open source under AGPL-3.0. Judges without an iPhone can try it in a browser on a streamed cloud iPhone.
  • Identifying a painting sends no photo. Recognition runs on the phone and sends a 20 KB fingerprint. Wall-label photos stay on the phone too; only the text is sent.
  • Every fact points to its source. Each "About this work" sentence links to the page its fact came from, and each audio-guide sentence's evidence is checked word for word before it's spoken.
  • One model family, sized per job. Six jobs run on three Nemotron sizes through one Token Factory API, each matched to how hard the job is.
  • A catalog anyone can rebuild. About 400,000 artworks from The Met, the Art Institute of Chicago, the Cleveland Museum of Art, the Finnish National Gallery and Wikidata/Commons, all from open data.

What we learned

  • Match the model size to the job. Token Factory serves every Nemotron size behind one OpenAI-compatible API, so moving a job between Ultra, Super and Lightning means changing one model ID. We ran the same eval sets against each candidate. On 25 real wall labels, Nemotron 3 Super got both title and artist right on 21. Two misses came from text-recognition errors and two from labels naming several artists.
  • Schemas guarantee shape, not truth. Any fact a user sees needs a check in code: a citation that must exist, a quote that must appear in the source, a date that must be on the page.
  • Every minimum in a schema is a promise the input has to keep. If the research can't support six sentences, don't require six.
  • Reasoning has a cost. For writing from a fixed fact sheet, thinking off was about 6 times faster and no worse in our evals.
  • Test on real photos. A synthetic holdout scored 0.990 confident precision, yet real camera photos exposed crop failures the holdout never showed.

What's next for Artlog: Museum Art Scanner

  • Tell originals from copies. Most remaining wrong answers are copies or versions of the photographed composition, such as a "Last Supper (after Leonardo)". Recognition should prefer the original when it's in the catalog.
  • More museums. Add more open collections to the catalog, and more museums' exhibition pages to Explore.
  • More languages. Stories and audio guides are written in English today. The same cited research can be written up in a visitor's own language.

Built With

Share this project:

Updates

Submission history