Formly

Confirmation Page
Questions and General Interface

Presentation Link : link

Inspiration

Formly started with my grandparents, sitting in front of a government portal, stuck on a field labeled something like "Declaration of Beneficiary Nominee Status." The question underneath it was simple. The wording wasn't. They speak limited English, and forms like that assume a fluency and confidence they don't have, especially when money or legal status is on the line. They'd call me over to sit with them and translate. Formly is basically what I am to my grandparents, turned into something anyone can open on their own.

What it does

Formly is a voice-controlled Chrome extension that turns any confusing form into a conversation. It reads the real fields on the page, translates each one into a plain-English question, and asks it out loud. You answer by voice. It fills that answer into the real form and moves to the next question. If new fields appear partway through, like when checking a box reveals three more questions, it can rescan and pick those up too. Sensitive fields like SSNs or passwords never get sent anywhere; they're handled locally. And nothing gets submitted behind your back: Formly always has a confirmation screen for you to review!

How we built it

The extension is a Manifest V3 Chrome extension that scans the page's DOM for form fields, using label, aria-label, and placeholder text to figure out what each one is asking. Those fields get sent to a FastAPI backend, which calls Groq's Llama 3.3 with structured JSON-schema output to turn them into plain conversational questions. Voice input and output run on the browser's native Web Speech API. Answers get filled into the real page using native DOM event dispatching so it works even on framework-driven sites, and everything routes through a summary-and-confirm screen before a single field is actually written.

Challenges we ran into

The biggest one was that modern forms aren't static. Sites built with React or Angular can reveal entirely new fields after a checkbox is clicked, and our first version was blind to anything that appeared after the initial scan, so we built a rescan mechanism that detects and folds in new fields without a reload. Another was getting Formly to recognize when 'uh, I don't get it' means explain, not answer, was the hardest part of making it feel like a real conversation instead of a voice-to-text box.

Accomplishments that we're proud of

The thing we're proudest of isn't any single feature, it's the trust architecture underneath the whole product. Sensitive fields are filtered out locally before they ever reach an LLM. Nothing is ever auto-submitted. Every answer gets read back for confirmation before it touches the real form. For a tool meant to help people through high-stakes paperwork, that restraint felt like the actual product, not just a safety feature bolted on.

What we learned

Building this made it very clear how much of "making AI useful" is really about constraining it. Getting an LLM to translate a field into a plain question was the easy part; getting it to reliably return structured JSON, correctly handle dropdowns and conditional logic, and know when to stay quiet on a sensitive field took far more iteration than the initial idea suggested. We also learned a lot about the messiness of real forms: inconsistent labeling, framework quirks, and fields that reveal more fields, none of which show up until you test against something real instead of a clean mockup.

What's next for FormGuide

Multi-language support is the most obvious next step, since the accessibility gap is often a language gap first. We'd also like to add save-and-resume for long forms, since these applications are rarely filled out in one sitting, and expand testing to a wider range of real benefits and government forms to see where our field-detection logic breaks. Longer term, we'd want to get this in front of the people it was actually built for, starting with communities like my grandparents', and let their feedback shape what comes next instead of guessing at it ourselves.