Tellr

It tells you what you can't see. Tellr is your AI shopping companion that describes products, catches hidden fees, and helps you shop with confidence.

Comment

Inspiration

Have you ever bought something online, only to realize it looked completely different from what you expected? Or reached checkout and discovered that the final price was higher than what you originally agreed to pay?

Now imagine trying to navigate that entire experience without being able to see the screen.

For blind and visually impaired shoppers, online shopping can mean navigating hundreds of product listings, listening to repetitive screen-reader announcements, and relying on descriptions that often leave out the details that matter most. A screen reader might tell you that a product is a “black leather handbag,” but it can’t necessarily explain its shape, how long the strap is, or how it visually compares to another bag.

And even after finding the right product, shoppers still have to navigate complicated checkout pages, unexpected fees, subscriptions, and optional add-ons.

We realized that the problem wasn’t simply that shopping websites needed to be read aloud. People needed a way to actually understand what they were buying.

That inspired one question: What if you could hear the whole picture?

We built Tellr, a voice-first AI shopping companion that transforms online shopping into a conversation, helping users understand products, compare options, and make confident purchasing decisions.

What it does

Tellr transforms online shopping into a conversational experience. Instead of scrolling through endless listings or navigating complicated interfaces, users can simply tell Tellr what they’re looking for and let it guide them through the shopping process.

Built as a Chrome extension, Tellr works alongside existing shopping websites, allowing users to interact with products through natural voice commands.

Voice-Powered Product Discovery: Users describe what they want in everyday language, including their budget, preferred color, style, or other requirements. Tellr interprets these preferences, searches available listings, and recommends products that match.

AI-Powered Visual Descriptions: Tellr uses multimodal AI to examine product images and explain details that traditional text descriptions might miss. For example, instead of simply reading “black handbag,” Tellr can describe its shape, straps, hardware, and overall appearance based on the product’s photos.

Evidence-Based Product Analysis: Tellr is designed to distinguish between information supported by product photos, claims made by sellers, details that appear contradictory, and information that cannot be verified. Rather than guessing about a product’s features, the goal is to help shoppers understand what the available evidence actually supports.

Intelligent Product Comparisons: Users can ask Tellr to compare multiple products based on appearance, price, and available listing details. Rather than opening several tabs and piecing together information, shoppers can hear the differences explained in one conversation.

Conversational Shopping Navigation: Tellr remembers which products it has recommended, allowing users to say things like “Tell me more about the second one” or “Choose option three.” It connects these commands to the correct product on the webpage and can help locate the appropriate Add to Cart button.

Transparent Checkout: Tellr’s checkout guardian is designed to review the full cost of a purchase, including quantities, shipping, subscriptions, optional extras, and unexpected charges. Instead of relying on AI to calculate totals, checkout rules use deterministic code to compare the order against the shopper’s requirements. The system is designed to distinguish between verified details, potential problems, and information that still needs review.

For our hackathon prototype, checkout confirmation is simulated. No real payments are processed, and Tellr does not handle credit card information.

Tellr doesn’t just read shopping websites aloud. It helps users understand what they’re buying, why a product might be right for them, and what they’re actually paying for.

How we built it

We built Tellr by combining multimodal AI, real-time voice interaction, and browser automation into a single shopping assistant.

Our architecture consists of several core components:

Chrome Extension — JavaScript, HTML, CSS

We developed Tellr as a Chrome extension that interacts directly with existing e-commerce websites. It extracts product listings, images, prices, and available shopping actions, allowing users to navigate online stores through voice commands rather than relying entirely on visual interfaces.

AI Reasoning and Visual Understanding — Anthropic Claude

Claude serves as Tellr’s reasoning and product-analysis engine. It processes shopping preferences, evaluates product information, and analyzes images to generate descriptions and comparisons.

Its multimodal capabilities allow Tellr to examine real listing photos against a shopper’s requirements. Our evidence-based approach distinguishes between four types of findings: supported, seller-claimed, contradicted, and unknown.

This distinction is especially important for accessibility. If a product’s photos don’t clearly show a particular feature, Tellr shouldn’t confidently claim that the feature exists. Instead, the system is designed to communicate uncertainty and identify the evidence behind its findings.

Voice Interaction — Cartesia Ink & Sonic

We integrated Cartesia’s voice technology to make Tellr a conversational shopping companion.

Cartesia Ink converts spoken shopping requests, questions, and commands into text through a push-to-talk interface. Cartesia Sonic converts Tellr’s responses into natural-sounding speech, reading product descriptions, recommendations, warnings, and shopping information aloud.

We also built microphone and playback controls to coordinate voice commands and reduce overlapping responses. Tellr’s speech can be muted so it doesn’t compete with existing assistive technologies such as VoiceOver.

Backend — Next.js and Node.js

Our backend connects the Chrome extension with AI services, processes product-inspection requests, and handles communication between the shopping website and Tellr’s conversational interface.

Product Identification and Context Tracking

One of our biggest technical challenges was ensuring that the product Tellr described was the same product it selected.

We developed a system that identifies individual listings using platform-specific product IDs and tracks recommendations across conversations. This allows Tellr to maintain consistent product references even when listings appear multiple times or recommended products are displayed in a different order from their position on the webpage.

Multimodal Image Processing

To make product descriptions more accurate, we built an image extraction pipeline that identifies a product’s actual photo gallery, retrieves high-resolution images, and filters out unrelated content such as logos, color swatches, and advertisements.

Development Tools — Mana & Notability

We used Mana as an AI coding teammate connected to our GitHub repository to develop Tellr's checkout guardian and strengthen its verification logic.

Before implementation, Mana reviewed our shared types and API contract and identified two missing pieces: a way to verify the shopper's selected item and a field for subscription acceptance. Mana then helped implement the deterministic checkout checker, input validation, review and confirmation routes, and 58 automated tests. We reviewed the resulting pull request and iterated on specification mismatches.

Mana also supported our broader development process through Chrome extension review, investigation of product-image understanding issues, and refinement of our presentation to match Tellr's website design.

Our Mana Development Sessions

We also used Notability as our development notebook, documenting everything from our initial brainstorming and accessibility research to technical decisions and changes throughout the build.

Under the hood, Tellr follows a conversational pipeline:

Voice Input → Speech-to-Text → Intent Recognition → Product Discovery → Image & Listing Analysis → Claude Reasoning → Spoken Response → Browser Action

Challenges we ran into

Designing for accessibility, not just functionality: One of our biggest challenges was realizing that just because a feature worked technically didn’t mean it was actually accessible. Throughout development, we constantly had to rethink, change, and add features to make Tellr genuinely useful for blind and visually impaired shoppers. Something as simple as having Tellr describe a product wasn’t enough if users couldn’t interrupt it, ask follow-up questions, or confidently select the correct item. We quickly realized how much research, testing, and attention to detail goes into building technology for people with different accessibility needs. It pushed us to think beyond what we thought users needed and consider how someone would actually interact with Tellr from beginning to end.

Getting AI to understand real shopping websites: Online stores have complicated layouts, dynamic content, sponsored listings, and inconsistent product information. Extracting accurate data was much harder than we expected. We discovered that some websites caused the same product to be extracted multiple times, creating duplicate recommendations and incorrect numbering.

Making sure Tellr was talking about the right product: One of our most frustrating bugs happened when Tellr described a product as “Number 1,” but the webpage displayed a badge labeled “21.” Although Tellr was highlighting the correct product, duplicate listings were causing conflicting labels. We had to redesign how products were identified and tracked to keep spoken recommendations, visual indicators, and browser actions synchronized.

Teaching Tellr to look at the correct images: Our initial image extraction system sometimes collected color swatches, brand logos, or unrelated products instead of the actual item. This meant our AI could receive perfectly valid images that were completely irrelevant to the shopper’s question. We rebuilt the extraction process to identify the selected product’s own gallery and retrieve its high-resolution photos.

Managing real-time voice interactions: Getting Tellr to speak was relatively straightforward. Getting it to listen, respond, and avoid interrupting itself was much harder. We encountered overlapping voice commands, repeated product descriptions, and situations where Tellr accidentally interpreted its own speech as a new command. Solving these issues required careful management of microphone sessions, audio playback, and asynchronous requests.

Understanding how people naturally shop: Users don’t speak in perfectly structured commands. A request like “I don’t want to spend more than $40” initially caused Tellr to search for products costing exactly $40. We had to improve how Tellr interpreted shopping constraints so that users could communicate naturally without learning specific phrases.

Building trust into checkout: We realized that checking whether a purchase is safe requires more than asking AI to read the final total. We needed a system that could compare the actual order against what the shopper requested, identify unwanted extras, and avoid treating missing information as confirmation that everything was fine. This pushed us toward deterministic checkout rules, explicit confirmation, and a clear distinction between verified information and unknown details.

These challenges taught us that building an AI assistant isn’t just about generating intelligent responses. It’s about understanding the people you’re building for and making sure the technology works for them in practice, not just in theory.

Accomplishments that we’re proud of

Turning online shopping into a conversation: Our biggest accomplishment was building a working Chrome extension that brings voice-powered shopping assistance directly into existing e-commerce websites. Users can describe what they’re looking for, hear recommendations, and navigate products without relying on traditional browsing alone.

Giving products a voice: We successfully connected Claude’s multimodal capabilities with Cartesia’s speech synthesis to turn product images and listing information into spoken descriptions. Seeing Tellr analyze an actual product and explain its appearance aloud was one of our most exciting milestones.

Connecting AI recommendations to real browser actions: We built a system that keeps spoken recommendations connected to the correct products. When Tellr says “Number 2,” that number corresponds to an actual listing that the extension can identify, highlight, and navigate to.

Building a more reliable visual understanding system: We tested our improved image extraction pipeline against 18 live Amazon products across six different categories, successfully retrieving the correct high-resolution product photos without unrelated page images.

Developing a strong testing system: We created automated tests covering voice interactions, image processing, budget interpretation, product identification, recommendation numbering, and shopping actions. We also developed 58 tests for our deterministic checkout verification logic.

Making checkout transparency part of accessibility: We’re proud that Tellr addresses more than product discovery. Our checkout guardian is designed to identify unwanted add-ons, incorrect quantities, unexpected charges, and information that needs further review. By separating deterministic verification from AI-generated explanations, we’re working toward a shopping experience where users can understand the entire purchase before making a decision.

Building with accessibility at the center: Rather than taking a traditional shopping assistant and adding voice features afterward, we designed Tellr around the idea that users should be able to understand products and make decisions without depending on visual information.

What we learned

AI is only as reliable as the information it receives: One of our biggest lessons came from product image analysis. Even a powerful multimodal model can’t provide an accurate description if it’s given the wrong images. We learned that reliable data extraction and validation are just as important as the AI itself.

Voice interfaces are more complicated than they appear: We learned how difficult it is to coordinate microphone input, speech generation, asynchronous commands, and user interruptions. Even a small timing issue can cause repeated responses or unintended actions, making careful state management essential.

Context makes conversation possible: A command like “choose the second one” sounds simple, but it requires Tellr to remember which products it recommended, the order they were presented in, and which listing corresponds to that choice. Building this system taught us the importance of maintaining context across multiple interactions.

Accessibility requires trust: When someone relies on spoken information to make a purchasing decision, accuracy matters. An incorrect product description, mismatched selection, or misunderstood budget can have real consequences. We learned that designing for accessibility means prioritizing clarity, reliability, and user control from the beginning.

Not every decision should be made by AI: While AI is incredibly useful for understanding language and interpreting product images, we learned that some tasks require predictable, verifiable rules. Checkout calculations and purchase requirements are examples where deterministic code can provide a stronger foundation for reliability than relying entirely on a language model.

Building for real people requires constant research and iteration: One of our biggest takeaways was realizing how much we still had to learn about the experiences of blind and visually impaired shoppers. Features that seemed intuitive to us weren’t necessarily useful in a voice-first experience. We had to continually revisit our assumptions, research accessibility considerations, and rethink how users would interact with the product. It taught us that accessibility isn’t something you add at the end of development. It needs to shape every decision from the beginning.

Most importantly, building Tellr showed us how AI can do more than automate tasks. It can help make everyday experiences more independent and accessible.

What’s next for Tellr

Smarter Checkout Verification: Finding the right product is only half the shopping experience. We want Tellr to help users feel just as confident when they’re ready to make a purchase. We’re developing a checkout verification system that reviews the entire order before confirmation, including the final price, shipping fees, quantities, subscriptions, and any unexpected add-ons. Tellr would read out an itemized breakdown, compare the charges against the user’s original budget and preferences, and flag anything that seems incorrect or wasn’t intentionally selected. For example, if a user adds a $30 item to their cart but the total increases because of an automatically selected subscription or additional fee, Tellr would explain exactly where that extra charge came from. Our goal is to make checkout more transparent so users can review every detail, catch unwanted charges, and make the final purchasing decision themselves.

Personalized Shopping Experiences: We want Tellr to become a shopping companion that gets better at understanding each user’s preferences over time. Future versions could remember things like preferred brands, clothing sizes, budgets, favorite colors, and shopping habits, so users wouldn’t have to repeat the same information every time they shop. We also want Tellr to adapt to individual accessibility preferences, such as how detailed someone wants product descriptions to be, how quickly information is read aloud, and whether they prefer short recommendations or more thorough comparisons. For example, someone who regularly shops for clothing could have Tellr remember their preferred sizes, styles, and price range, making future searches faster and more relevant. By allowing users to control what Tellr remembers, we hope to create an experience that feels less like navigating a website and more like talking to a shopping companion who understands what you’re looking for.

Making Checkout Verification Production-Ready: Before supporting real purchases, we want to move beyond simulated checkout confirmation. This includes server-verified checkout reviews, where the reviewed order is stored or cryptographically signed rather than trusted entirely from the browser. We also want Tellr to inspect real checkout pages in a read-only manner, explain any issues it finds, and leave the final purchasing action to the shopper.

Measuring Real-World Impact: We want to evaluate Tellr based on more than whether its features technically work. Important measures would include how often shoppers complete purchases without sighted assistance, how many unwanted charges or checkout issues Tellr identifies, and how accurately it communicates uncertainty instead of making unsupported claims.

Real-World Accessibility Testing: We want to work directly with blind and visually impaired shoppers and accessibility organizations to understand their experiences, test Tellr in real shopping situations, and improve the product based on their feedback.

Our long-term vision is to make online shopping more independent, transparent, and accessible for everyone.

Built With

Share this project:

Updates

Submission history