Lantern

This image is currently processing. Please be patient...
Lantern Chrome extension.
Guides you step by step, in Spanish or any language.
Highlights the next thing to click.
Walks you through checkout.
Stops a scam before you pay.

Inspiration

Older adults and people who don't read English well get stuck online, and they're exactly who scammers target with "your power will be cut off, pay with gift cards" pages. Family isn't always there to help, but a guide in the browser can be. Introducing Lantern, a voice-first, multilingual guide that shows you what to click and stops scams.

What it does

Say one goal out loud in your own language ("ayúdame a pagar mi factura de luz"), and Lantern shows you where to click, one step at a time, on the real website. The page dims, a ring lights up the right button, and Lantern says the step out loud in your language. You do the clicking, and Lantern looks again after every action until the goal is done.

Voice first, any language. Speak your goal; every step comes back spoken in the language you used.
No chat window. One ring, one short caption, one spoken sentence.
Never clicks or types for you. It never asks for your password either. It points at the box.
Stops scams. Gift-card or crypto payment requests, "account suspended" threats, countdowns and fake web addresses turn the page red, and Lantern warns you out loud, even when you didn't ask it for help.
Push-to-talk. Press Option+L and speak.

How we built it

Chrome extension (Manifest V3): a service worker runs the loop (look, ask the model, point, wait for the click). An offscreen document records the mic and plays the voice.
Set-of-Marks prompting: each step, we screenshot the tab with a number drawn on every element and send the picture plus the numbered list. The model answers with one element number and one sentence, and the extension rings that exact element on the real page.
Gemini 3.5 Flash Lite picks each step with a strict JSON schema and the step history.
ElevenLabs both ways: Scribe v2 hears the goal and detects the language; Flash v2.5 speaks each step back, with a natural voice per language.
Scam rules in code: checks that set a minimum risk the model can raise but never lower. "High" needs two separate warning signs.
The overlay: React, TypeScript and Tailwind in a closed Shadow DOM, so no site's CSS can break it.
A Node backend keeps the API keys out of the extension.
A realistic demo bank (Sunrise Power) with a sign-in, a paperless popup, a bill-pay flow and a fake "50% off" offer that leads to a scam page.

Challenges we ran into

Accuracy vs. speed. We built a test from 11 real screens captured through the extension and ran each one in English and Spanish, 3 times. Gemini 3.5 Flash Lite picked the right element in 66 out of 66 calls at about 1.2 s median latency. The real bottleneck was the free-tier rate limit.
Scam checks that cried wolf. Our first rules flagged Amazon for selling gift cards. Now they look for a request to pay that way and need two warning signs before calling a page high risk.
Scams don't wait for a goal. At first Lantern only checked pages while it was guiding you, so a scam opened afterwards went unnoticed. Now it checks every page you open, using the language you last spoke.
Off-path users. Real people switch tabs and reload. The loop pauses in background tabs and resumes when you return.
The right voice. Our first Spanish voice had a Spain accent our users don't speak, so each language now gets its own voice.

Accomplishments that we're proud of

Made a guide that works on real sites (Amazon search to checkout), not just our demo.
100% next-step accuracy (66/66) on our English and Spanish test screens, at about 1.2 s median model latency.
Scam protection that's always on, warning out loud in the user's own language.
An accessible design: big ring, one short caption at a time, voice in and voice out.

What we learned

How to keep an AI in a loop with a real user, one screen at a time; that safety rules belong in code, where the model can raise the alarm but never lower it; and that for older users, fewer words and a familiar voice matter most.

What's next for Lantern

Need to handle three issues: 1) a hosted backend so families can install Lantern without running a server, 2) keeping banking and health pages private while doing so, and 3) a family circle that gives a trusted relative a heads-up when Lantern blocks a scam.

Team and code

Code: https://github.com/j-palacio/lantern

J (voice, lead) · Andrew (Gemini) · Juan (UI) · Xiangtao (scam rules)

Built With

chrome
css
elevenlabs
framer-motion
gemini
html
javascript
manifest-v3
node.js
react
tailwindcss
typescript
vite

Submitted to

Knight Hacks IX

Created by

I developed the Gemini brain: the prompt, a strict JSON schema, one retry on bad JSON or a made-up button number, back-off when Gemini is busy, all inside an 18-second budget. History-aware, so it knows what the user already did.

Andrew Barsoum
Implemented the Scam check logics. Created fake scam page for demo, and created a few test cases.

Xiangtao Wu
The in-page UI, rebuilt on React + TypeScript + Tailwind + Framer Motion. Vite compiles it into a single content script mounted in a closed Shadow DOM, so no website's styles can break Lantern's UI and Lantern's can't leak into the page. The rest of the extension stayed plain JS with no build step.

The highlight overlay — a glowing ring that glides onto its target, a caption placed where there's room, and a connector tying them together. The ring follows the target through scrolling and resizing. On a high-risk scam page it switches to a red, dimmed warning state with nothing highlighted.

The widget — three tiers sharing one animated shell: a launcher coin, a voice panel, and a task panel showing the goal and the current step. Covers idle, listening, thinking, guiding, done and error states, with Stop always available.

The voice orb — a Canvas visualizer of layered blurred gradient blobs that breathes when idle and reacts to live volume, leaning blue toward the user's voice and amber toward Lantern's. Fed by a level channel deliberately kept outside React, since volume updates arrive about 20 times a second.

The page scanner — finds every interactive element on the page, including inside open shadow roots and same-origin iframes, with coordinates translated into top-level viewport space. A hit test confirms each element is actually reachable, so Lantern never points at something buried under a cookie banner or a sticky header.

Set-of-Marks placement — the numbered tags drawn on the screenshot the model reads. Placement is a cost-weighted search across candidate positions, tuned so a tag never covers a neighbouring element or another tag, since the number is the model's only spatial reference.

Background-tab handling — screenshots always capture the active tab, so a turn that reaches that point while the guided tab is in the background now pauses and resumes when the user returns, surviving a service worker restart.

Multilingual support — added Filipino to the language picker, and fixed two ways it silently failed: the text-to-speech model rejecting the language code, and speech detection misreading short Filipino phrases as other languages.

A dev harness that drives every UI state with a faked extension runtime — no extension, no server, no API keys — so UI work didn't need a full reload cycle each time.

juanlopolicaarpio Policarpio
Team lead and voice. Built the voice pipeline end to end: push-to-talk mic capture, ElevenLabs Scribe v2 speech-to-text with language detection, and Flash v2.5 text-to-speech with a voice per language. Designed the architecture and step loop, built the Sunrise Power demo site, added always-on scam warnings, ran the accuracy test (66/66, about 1.2 s), reviewed and merged every PR, and made the demo video.

J Palacio