SwiftFillAI

Inspiration

At DragonHacks, we noticed how tedious online forms have become — from banking forms to job applications to sign-ups — users waste countless minutes manually entering repetitive details. We wondered: what if an AI could “see” your form, understand it, and fill it out intelligently, in seconds? Not just simple fields — but even dropdowns, radio buttons, and checkboxes — dynamically!

That’s where SwiftFillAI was born.

What it does

SwiftFillAI is a Chrome extension that: • Lets users capture a view of their current screen • Uses Gemini AI to analyze the form fields visually • Generates realistic, safe autofill values based on the context • Autofills everything — including text inputs, dropdowns, radios, and checkboxes • Allows follow-up questions about the webpage (like “What does this form do?” or “What fields are missing?”) • All through a beautiful, dark-themed, lightweight popup interface!

How We Built It

• Frontend: We built a Chrome extension popup using plain HTML, CSS (dark theme + Gemini branding), and vanilla JavaScript.
• Backend: A Node.js + Express server that communicates with Gemini 2.0 Flash models via Google’s API.
• Screen Analysis: We capture the visible page using getDisplayMedia and send screenshots to Gemini for analysis.
• Form Autofill: We collect field metadata (names, labels, placeholders) and prompt Gemini to generate a smart JSON of autofill values.
• Dynamic Injection: Autofill scripts dynamically fill fields (even tricky ones like credit cards, SSNs, dropdowns, and gender radios) without manual hardcoding.

Everything is powered by live AI responses — no preprogrammed guessing.

Challenges we ran into

• Extension popups closing during screen-sharing initiation (Chrome security restriction)
• Parsing Gemini responses when they came wrapped in Markdown instead of raw JSON
• Handling different input types — text fields were easy, but dropdowns, radios, and checkboxes required much smarter logic
• Maintaining UI/UX — keeping the extension clean, intuitive, and mobile-responsive

Accomplishments that we're proud of

Built a full AI-powered autofill system — not just dumb templates! • Made it look professional and polished with custom CSS (matching Google’s dark UI trends) • Successfully handled edge cases like credit card fields, address fields, and missing values • Learned how to deeply integrate Gemini AI into Chrome extensions in real-time

What we learned

How Chrome extensions work under the hood (manifest V3, service workers, popup management) • How to clean and parse AI-generated JSON safely • How to trust AI to do dynamic field mapping instead of manual coding • How small UX touches (like better button gradients and fonts) massively improve user feel • Why organizing project folders cleanly matters when scaling fast

What's next for SwiftFillAI

Smarter validation of generated autofill (detect invalid emails or missing fields automatically) • Let users customize what kind of profile they want to autofill with (professional, casual, anonymous) • Save user history and allow multiple “personas” • Integrate vision-based field detection (even better screen analysis using Gemini Vision models) • Publish SwiftFillAI to the Chrome Web Store so anyone can install it!

Built With

• and
• chrome
• chrome-desktopcapture-api
• chrome-extensions-api-(manifest-v3
• css3
• desktopcapture)
• dotenv
• express.js
• firefox
• gemini
• gemini-2.0-flash-(generative-ai-model)
• gitlab
• google
• google-cloud-(gemini-api-access)
• google-gemini-api-(generativelanguage.googleapis.com)
• html5
• javascript
• javascript-?-node.js-?-html5-?-css3-?-express.js-?-node-fetch-?-dotenv-?-chrome-extensions-api-(manifest-v3
• messaging
• node-fetch
• node.js
• scripting
• storage
• tabs
• visual
• visual-studio-code