Inspiration
Filling out forms online is still a tedious task, especially when traveling or applying for various services like visas. The repetitive process of inputting the same information for each form, especially when it involves multiple family members, can be time-consuming and prone to errors.
This inspired me to build a solution that automates and simplifies this process using AI. The aim is to save time, reduce errors, and improve the overall user experience when filling out online forms.
What it does
Smart Form Filler is a Chrome extension that uses AI technologies to automatically detect form fields, match them to user input data, and fill them out accurately.
By leveraging on-device AI, the extension ensures that personal information is kept private and secure, without the need to send it to external servers. The extension intelligently identifies fillable fields on a webpage, maps the user's information (such as name, address, and travel details), and fills the form in a seamless manner.
This significantly speeds up the process, reduces manual errors, and enhances convenience for users, especially when dealing with repetitive tasks like booking travel or shopping.
How we built it
The extension was built using the WTX extension library, React, and Chakra UI for the user interface. To interact with AI APIs, I used the PromptAPI to perform key functions:
- Identify fillable form elements and generate the correct selectors.
- Map user information to the form fields, even when different websites request different formats (e.g., full name vs. first, last, and middle names).
- Use chat interfaces to gather user information or confirm details that need to be autofilled.
The combination of these tools enabled us to create an efficient and intuitive user experience. The interface uses the Chakra UI and llm-ui libraries to present a clean, user-friendly design for managing and interacting with the data.
Challenges we ran into
While building the extension, I encountered several challenges:
- Accessing the SDK: Initially, getting access to the AI API was tricky. However, after some research and discussions in the community Discord, I managed to resolve the issue.
- Handling large DOM structures: The PromptAPI threw cryptic errors when parsing large, complex DOMs. To mitigate this, I had to clean up the HTML input before sending it for processing.
- Parsing inconsistent responses: The data returned by the API didn’t always adhere to the expected JSON format, which made it difficult to parse programmatically. I had to implement workarounds to handle such inconsistencies.
- Hallucinations of selectors and user information: Occasionally, the AI would produce inaccurate form selectors or user data, which required manual intervention and fine-tuning to improve accuracy.
Accomplishments that we're proud of
I am particularly proud of:
- Building an extension that I might actually use!
- Getting the extension to a working state, despite the challenges faced along the way.
- The innovative chat feature that helps users input their information quickly, which also serves as an interactive method for confirming or modifying data.
- Building an AI-driven solution that operates fully on-device, ensuring users' personal information remains private and secure.
- The seamless integration of AI to identify form fields and accurately map user data to them, reducing the time spent filling out forms.
What we learned
Throughout this project, Ilearned a great deal about working with new AI technologies, especially those that operate locally on the device.
This was my first experience developing AI-powered applications, and it was an exciting journey. I gained a deeper understanding of how AI can be used to solve real-world problems, such as improving the online form-filling experience. Additionally, I learned how to overcome challenges related to data parsing, working with APIs, and ensuring data privacy when using AI.
What's next for Smart Form Filler
There are several exciting features I plan to implement in the future:
- SummaryAPI Integration to get user data: I plan to integrate the Summary API to extract relevant information from emails, such as travel details or reservation confirmations, and automatically populate form fields.
- Contextual Layers: I plan on adding a layer of context, such as a "Sri Lanka Trip," where users can store common details like flight information or hotel addresses. This would allow users to automatically populate relevant details across multiple forms, providing even more convenience.
These enhancements will make the Smart Form Filler even more powerful, efficient, and useful for users.
Built With
- chakra-ui
- react
- typescript
- wxt
