WordsCam - えいごカメラ

WordsCam_Screenshots

Inspiration

The inspiration for Eigo Camera came from a simple, universal experience: the struggle to connect textbook vocabulary with the real world. Traditional language learning often feels abstract and disconnected. We memorize lists of words like "desk," "chair," or "tree" from a book, but the link between the word and the physical object is weak.

I noticed that the most memorable learning happens through direct experience. When you travel to a new country, you learn the word for "coffee" by ordering it, the word for "station" by navigating the city. The context makes the vocabulary stick. I asked myself: What if we could bring that immersive, contextual learning experience into our everyday lives, without needing to travel?

The recent advancements in on-device computer vision and large language models (LLMs) provided the answer. By combining the power of a smartphone camera with AI, we could create a tool that instantly bridges the gap between the world around us and the English language. That's how the idea for Eigo Camera was born: a camera that doesn't just take pictures, but teaches you the words for what it sees.

What it Does

Eigo Camera is an AI-powered mobile app that transforms your environment into an interactive English classroom. Here’s the core functionality:

Snap & Identify: Point the camera at any object or scene. The app uses a computer vision model to identify what's in the photo.
Learn Instantly: The app overlays the corresponding English word on the image, complete with native speaker pronunciation. For example, point it at your breakfast, and it might identify "coffee," "toast," and "plate."
Build Your Visual Dictionary: Every photo you take is saved as a beautiful, interactive "flashcard" in your personal collection. This creates a highly personalized vocabulary book based on your life and your surroundings.
Smart Review: A built-in review system uses a simple swipe-based interface to help you practice your saved words, prioritizing the ones you struggle with.

How We Built It

Building Eigo Camera involved integrating several cutting-edge technologies into a seamless user experience. Our tech stack is primarily focused on mobile and AI.

Frontend: The app was built natively for iOS using Swift and SwiftUI. This allowed us to create a clean, modern, and highly responsive user interface. We leveraged Apple's AVFoundation framework for camera access and Vision framework for initial on-device object detection.
Core AI - Object Recognition: For the primary object recognition, we fine-tuned a lightweight, mobile-optimized vision with Gemini. The goal was to achieve a good balance between speed and accuracy directly on the user's device.
Core AI - Contextual Learning: Once an object is identified, its name is sent to a server-side Large Language Model (LLM). We use Gemini API. The LLM's task is twofold:
- Provide a concise, accurate definition of the word.
- Generate a contextually relevant example sentence.
Backend & Database: A lightweight backend using Firebase handles user authentication and data storage. Firestore is used to save the user's collection of photo-word pairs.
Text-to-Speech (TTS): To provide audio pronunciation, we integrated Apple's native AVSpeechSynthesizer, which offers high-quality, natural-sounding voices directly on the device.

Challenges We Faced

Accuracy vs. Speed in Object Recognition: Our biggest technical hurdle was finding the right balance for the computer vision model. A highly complex model would be more accurate but would drain the battery and feel sluggish. We spent weeks testing different models and ended up using a hybrid approach.
Managing Ambiguity: The real world is messy. Is that a "cup," a "mug," or a "container"? The AI had to be smart enough to provide the most common and useful term. We solved this by programming the LLM to prioritize the most frequent and practical vocabulary word.
Creating a "Learning" Experience, Not Just a "Translation" Tool: Our challenge was to build a compelling learning loop. This required significant effort in UI/UX design to make the process feel rewarding and game-like, rather than like a chore.
API Cost Management: Constant calls to a powerful LLM API can get expensive. We had to implement smart caching strategies to reduce redundant API calls and keep operational costs manageable.

What We Learned

This project was a profound learning experience, both technically and conceptually.

The Power of Context: We confirmed our initial hypothesis: learning is dramatically more effective when it's personal and contextual.
UI/UX is King for EdTech: For an educational app, the user interface isn't just chrome; it's part of the pedagogy. The satisfaction of building a visual collection was just as important as the AI's accuracy.
Iterative AI Development: Building with AI is not a one-and-done process. We are constantly monitoring the model's performance and using that data to fine-tune it for future updates.
The Future is Multimodal: The project solidified our belief that the future of learning tools lies in multimodal interfaces that engage multiple senses—sight (the photo), hearing (pronunciation), and text (the word and sentence). Eigo Camera is our first step in that direction.