MnemoLens

Point to one unfamiliar word in a real book. MnemoLens understands which word you mean, reads its sentence, explains that meaning, and turns the encounter into something you can recall.

Comment

Inspiration

I'm currently at the stage of learning Polish where I already understand most everyday words, but reading books is still slow because unfamiliar words appear constantly.

The idea for MnemoLens came while reading my first Polish book in the University of Warsaw Library (BUW), famous for its rooftop gardens.

My reading process quickly became frustrating. Every time I encountered an unfamiliar word, I underlined it, opened Google Translate, pointed the camera at the page, waited for OCR, and then searched through the translated paragraph to find the one word I actually cared about.

After about four pages, I realized I probably wouldn't have enough patience to finish the book this way.

Around the same time GPT-5.6 became available.

Instead of continuing to read, I decided to build the tool I wished already existed.

That was the beginning of MnemoLens.

What it does

MnemoLens lets you point at a word in a physical book and immediately understand that exact word in its original context.

Instead of translating the entire page, it identifies the selected word, extracts the surrounding sentence, determines the intended meaning, and generates a contextual translation.

If the user wants to remember the word, MnemoLens builds a personalized learning card.

The card can include:

  • contextual translation
  • the original sentence
  • pronunciation
  • mnemonic associations
  • relationships with languages the user already knows
  • generated visual mnemonics
  • verified etymology and historical background whenever reliable sources are available

For example, because I also speak Ukrainian, many Polish words become much easier to remember when I see a common root or a similar pronunciation. MnemoLens can use this information to build language-specific memory connections unique to every learner.

The goal is not simply to translate words.

The goal is to help people actually remember them.

How we built it

At first I thought the project would be relatively straightforward.

I assumed OCR would be the difficult part.

It wasn't.

The real challenge turned out to be determining exactly which word the user intended to select.

Books move.

Hands move.

The finger often covers the word itself.

Sometimes everything moves at once.

To solve this problem I built a complete recognition pipeline that tracks the page, follows finger movement, maps pointer coordinates back onto the OCR text map, and continuously determines the intended word.

To improve accuracy systematically, I recorded seventeen real-world reading scenarios using physical books. These scenarios became automatic regression tests for every improvement.

Codex was given measurable engineering goals rather than isolated coding tasks.

It continuously maintained an engineering diary describing every experiment, every successful improvement, every failed hypothesis, and the next objective.

This made it possible to hand development from one Codex session to another without losing accumulated knowledge.

Once the word has been identified, GPT-5.6 is used for the project's central reasoning task: generating personalized memory strategies.

Instead of immediately generating an image, the system first evaluates multiple mnemonic approaches, verifies them, builds a minimal visual specification, generates the image only if the concept passes quality checks, and finally performs an independent vision QA on the generated result.

Challenges we ran into

The biggest surprise was that OCR was never the hardest problem.

The hardest problem was understanding human intention.

A person naturally points slightly below a word, rotates the book, partially covers the sentence, and constantly changes the viewing angle.

Detecting the finger was only the beginning.

Understanding what the user actually meant required combining computer vision, geometry, OCR, tracking, and language understanding into one stable pipeline.

Another unexpected challenge was mnemonic image generation.

Image generators are very good at drawing objects.

They are much less capable of understanding why an image should help someone remember a word.

We eventually built a separate concept-validation framework that evaluates the mnemonic itself before allowing image generation.

Accomplishments that we're proud of

We're proud that MnemoLens grew from a personal reading problem into a complete learning system.

We're especially proud that:

  • the application works with real physical books rather than prepared demo text
  • every word is translated in its actual context
  • learning continues even without an Internet connection
  • memory techniques are personalized instead of generic
  • every generated mnemonic passes multiple quality gates before reaching the user
  • the system is allowed to reject image generation completely when no useful visual mnemonic exists

Most importantly, we believe we've shown that AI can improve paper books without replacing them.

What we learned

Building MnemoLens taught us that successful AI products are rarely about choosing the strongest model.

Most of the difficult work happens between models.

Good OCR does not automatically create a good reading experience.

A correct translation does not automatically create learning.

A beautiful image does not automatically become a useful mnemonic.

The real challenge is designing the interaction between humans, AI, computer vision, and memory.

We also learned how effective Codex becomes when it is treated like an engineering collaborator with measurable goals, automated evaluations, and long-term project memory rather than simply a code generator.

What's next for MnemoLens

The current version uses the phone camera.

The next stage removes that requirement.

I have already purchased a XIAO ESP32S3 Sense development board together with an upgraded 5 MP OV5640 autofocus camera module.

The idea is to mount this tiny Wi-Fi camera above a book, for example on a desk lamp, allowing it to continuously observe the page while the phone performs all AI processing.

The reader simply points at a word.

The camera identifies the location.

The phone immediately displays the contextual explanation.

Eventually I would like this kind of camera to become as common in libraries as reading lamps.

Interestingly, the entire idea for MnemoLens was born while reading in a library, so it feels natural that one day the same environment could provide intelligent reading assistance for anyone studying printed books.

We're also preparing support for smart glasses. The connection layer has already been implemented, but we haven't yet had access to compatible hardware for real-world testing.

Our long-term vision is to make printed books as interactive as digital documents while preserving everything that makes physical reading enjoyable.

Built With

Share this project:

Updates