Inspiration

Cortical visual impairment (CVI) is a brain-based vision impairment where the eyes work, but the brain struggles to sort what they see. It is reported as the leading cause of childhood visual impairment in developed countries, and a 2021 Bristol screening of 2,298 primary-school children found that about 3 in 100 may have CVI-related vision problems.

Children with CVI recognize objects much more easily when they see one thing at a time, with little clutter, strong contrast and often a favorite color. A picture-book page is the opposite. Today a teacher of students with visual impairments (TVI) rebuilds pages by hand with scissors and black card, so each child gets only a few adapted books. We found no tool that adapts real books automatically, so we built one: one book, different ways to see it.

What it does

A teacher uploads a picture-book PDF, and Miraai:

  1. Finds the subject. A model reads each page's picture and text together and suggests up to five subjects.
  2. Cuts it from the original art. Another model traces the subject into a mask. Nothing is ever drawn - every pixel is copied from the page.
  3. Lets the teacher decide. The teacher confirms a suggestion, or combines several suggestions. If the model fails, the teacher can have the model re-extract the mask, or trace the object by hand.
  4. Restyles it per child. Each child profile sets a level (Tinted, Cut-out, Faded page or Outlined), preferred colors, background, size and position. Switching child or level re-renders the saved cut-out with no new AI calls.
  5. Requires approval. For each child and level, the teacher reviews a contact sheet with automatic checks (contrast, clutter, size, and a second AI "look" that names the cut-out). The checks are measurements, and a child can only see a page after the teacher approves it for that child and level.
  6. Reads calmly. A full-screen reader shows one approved page at a time, with no buttons, animation or auto-advance, and ends by crediting the book.

Our demo uses Animal Homes (Pratham Books, StoryWeaver, CC BY 4.0) and two fictional child profiles.

How we built it

We planned the PRD and Technical requirements first, then created a plan to build, splitting the work up into four modules we could work on separately with as little overlap as possible. We wrote stand-in data for the modules we needed that weren't built, and worked on our own branches, with frequent commits and PRs that had to be approved by other group members.

  • Web: React 18, Vite, TypeScript, Tailwind, Atkinson Hyperlegible.
  • API: FastAPI and Pydantic v2 on Python 3.11, with MongoDB Atlas.
  • Pipeline: PyMuPDF renders pages to 3,000 px masters; four pages process in parallel, and each gets its own state, so one bad page never blocks a book.
  • AI, as two cascades: our own small models answer first, and a larger model is called only when ours is unsure.
    • Choosing the subject: our local finder reads the page first and scores its confidence in each subject. It keeps its answer only when exactly one subject clears the confidence bar, the page text names that subject, and the text names nothing else. Otherwise the page goes to Gemini Flash (JSON-schema output, temperature 0, fixed seed).
    • Cutting it out: our own U-Net traces the mask first and grades its own work with five checks (edge sureness, fill, one piece, no background in the box edge, and agreement with the mirrored page). Any box scoring under 0.75 goes to SAM 2.1 small. This cut-out cascade is what runs on the live server.
  • Display and checks: plain NumPy, OpenCV and Pillow, so the same input always gives the same picture.
  • Hosting: one CPU-only Vultr instance with Docker Compose and Caddy. A systemd timer rebuilds from main every minute, so merging was deploying.

The slow AI work happens once per book; everything after is cheap compositing from saved masks.

We trained the cascade's first-pass models from random weights during the event: a 7.8M-parameter U-Net for cut-outs, and a local subject finder (a ResNet-18 namer plus a whole-page U-Net that proposes boxes). Both cascades are built into the pipeline and switched on with settings (SEGMENTER=cascade, SUBJECT_FINDER=cascade), and every routing decision is logged, so each page's route can be audited. The cut-out cascade is live on the server. The teacher still approves everything either way.

Challenges we ran into

  • The AI is often unsure. On our first run Gemini marked 4 of 5 pages as having several subjects. Instead of forcing an answer, we let teachers choose, to ensure we didn't have AI remove key elements of the story.
  • Speed on a CPU server. Gemini takes about 11 s per page and SAM about 4 s to encode one on a 4-thread CPU, so we parallelize and cache every mask. A 5-page book with real Gemini calls processed in about 30 s on the live server.
  • Our models vs. real book art. Our cut-out model came within 0.009 mean IoU of SAM 2.1 on held-out synthetic drawings (0.897 vs. 0.906) at about 5x the speed on a GPU (0.02 s vs. 0.10 s a page). Those pages resemble its training data, and on our CPU-only server the speed-up shrinks to 1.03x, so the cascade's job there is routing, not raw speed. Our local finder also fell short of Gemini.

This means the local finder only answers when it is nearly certain it can identify the subject type and where it is. We generated our own test dataset by labeling various picture books with the main subjects, and tested our models on it. The local finder confidently identified the subject in around 20% of the data, and didn't give any false positives (where the model is confidently wrong). On simple, clearly defined objects, its accuracy rises to about 90%. It also ran in about 0.07 s on a laptop CPU versus 11 s for a Gemini API call, over 150 times faster.

Accomplishments that we're proud of

  • Training two model families from scratch in 36 hours and wiring them into a working two-stage cascade: our models take the pages they are sure about, Gemini and SAM 2.1 take the rest, and the gate that decides is tuned to be wrong on zero pages rather than to answer more.
  • A working end-to-end product on our own domain, from upload to reader.
  • Promises enforced in code: a unit test confirms cut-out pixels match the source page, and the server checks approval on every page a child sees.
  • A hand-drawn outline path that works with every AI service turned off.
  • Reporting where our models lose, instead of hiding it.
  • About 550 Python and 190 web test functions across four parallel lanes.

What we learned

  • Admitting uncertainty is a feature. "I'm not sure, you pick" is safer for a child than a confident guess. Gemini's confidence is uncalibrated, so we use it only to screen out weak guesses.
  • Fresh data keeps you honest. We re-tested on two books our models had never seen; some gains held and some didn't.
  • Recall and precision pull apart. Tricks that made our finder suggest more cut-outs also pushed more wrong pages to Ready.
  • Version control is how four people build one product. Each of us owned our own folders and worked on our own branch, committing often and merging through PRs another teammate reviewed. Because merging to main redeployed the live site within a minute, every merge had to keep the whole app working. We talked through every PR that touched an interface between our parts, and across over 250 commits nobody overwrote anyone else's work.

Built With

Share this project:

Updates

Submission history