Inspiration
Language-learning apps usually teach someone else’s vocabulary list. Kantu starts with the world directly in front of the learner: a breakfast, a hotel room, a market stall, a street sign, or anything else they are curious about.
What it does
Take a picture or choose one from the photo library. Kantu uses multimodal scene understanding to identify useful objects, ingredients, materials, visible text, and small details, then places translated labels directly over the original image.
Instead of returning generic guesses, Kantu aims for the most specific description supported by visible evidence. Learners can tap a label to:
- hear its pronunciation in the language they are learning;
- read a practical example sentence;
- inspect the evidence and certainty behind the identification; and
- save the word to their private dictionary.
Kantu supports 18 learning languages. Switching languages reuses the original spatial grounding and translates it on demand, so the photo is not analyzed again and the change does not consume another picture allowance.
How we built it
The mobile app is built with React Native, Expo, and TypeScript. A Fastify gateway preprocesses images, removes metadata, performs multimodal grounding with Qwen on Alibaba Cloud, validates structured responses, and maps normalized coordinates back onto the displayed image.
Firebase Authentication supports Apple and Google sign-in. Firestore stores the server-authoritative monthly picture allowance so it survives reinstalls and device changes. Idempotent reservations prevent retries from consuming multiple pictures.
RevenueCat powers Kantu’s Basic and Premium subscriptions. Entitlements are mirrored to the backend through authenticated webhooks:
- Free: 5 pictures per month
- Basic: 120 pictures per month
- Premium: 350 pictures per month
The allowance resets monthly. Failed scans and language changes do not consume pictures; the optional “More detail” pass starts a new grounding analysis and clearly communicates that it uses one additional picture.
Challenges we ran into
The hardest product problem was balancing spatial accuracy, translation quality, latency, and sustainable unit economics. Generating every language eagerly produced good coverage but unnecessary cost and delay. We separated visual grounding from translation, return the selected language first, then reuse and cache the grounding result for any additional language.
We also moved quota enforcement away from the phone. The backend now resolves the active RevenueCat-backed plan and increments usage atomically, while the device keeps a local snapshot for a responsive interface.
Accomplishments that we're proud of
- Precise labels stay anchored correctly across different image and screen aspect ratios.
- Language changes do not rescan the photo.
- Pronunciation and useful example sentences turn identification into an actual learning loop.
- Subscription state and picture allowances persist across devices.
- The experience is localized for 12 storefront and interface languages.
- Kantu has a distinctive, accessible visual system and localized marketplace assets.
What we learned
A compelling AI product needs more than a model response. The useful experience comes from grounding every word in visible evidence, handling uncertainty honestly, making latency understandable, and designing monetization around the real cost of successful work.
What's next
During Shipaton we are focused on store approval, real-user onboarding, retention measurement, and a small daily real-world vocabulary mission that gives learners a reason to return without turning Kantu into another generic flashcard app.
Built With
- alibaba-cloud
- expo.io
- fastify
- firebase
- node.js
- openai-codex
- qwen
- react-native
- revenuecat
- typescript