WallyWare

Inspiration

Try handing your grandmother a 1040 the night before taxes are due. The task itself is usually simple. Knowing what each box means, and spotting the scams along the way, is the hard part. Americans over 60 reported losing $7.7 billion to online fraud in 2025 (FBI), and there are fewer family members around to help. A chatbot could just do it for them, but then they never learn. We wanted a helper that points and explains instead.

What it does

WallyWare is a web browser with a walrus guide. Tell Wally what you need, and he boxes the exact field or button to use next and explains it in one plain sentence. You do the clicking; he never does it for you. He checks every page for scam red flags first (gift cards, urgency, "don't tell anyone") and steers you to Report or Delete. With the Wally Scan phone app, you scan a paper bill or form and Wally walks you through the photo line by line. Private numbers like SSNs are masked before anything reaches the AI.

How we built it

Electron and SvelteKit for the browser. Grok (xAI) and Gemini with a strict JSON schema for Wally's steps and the scam check. An Expo app plus an iOS Shortcut for scanning. A Flask server on Fly.io running Tesseract OCR, privacy masking and MongoDB.

Challenges we ran into

OCR. Tesseract returns words, not lines, and a real IRS W-4 exposed bugs our test images never did: duplicated words, dropped short words, blank lines. Getting Apple's scanner into Expo Go took a Shortcut-and-clipboard workaround. We also had to find the smallest scan size that still reads fine print, which turned out to be 1440 px wide.

Accomplishments that we're proud of

A paper document goes from a phone scan to Wally pointing at the right line, with private numbers masked on the way. And Wally won't walk you into a scam: he explains the red flags and helps you report it.

What we learned

Pointing beats explaining. Test on real documents early. When something has to be right every time, like the scam check going first, enforce it in code instead of trusting the model.

What's next for WallyWare

Checking your answer when you press Next, remembering values across documents (like your W-2 when you're on the tax form), multi-page scans, and a Chrome extension.