Inspiration
Seeing many incredible builders build a great backend, not knowing what could be friction for a user in the frontend. Like you have many different use cases, but the user is not able to find them on your website. I had that happening in a previous hackathon and thought of a solution for this before deployment.
What it does
It takes the given URL or Screenshots, crawles through the pages and captures the DOM and the visual of the page. Then this is fed to an llm (I chose a GPT model) with precise instructions on how to review the given data. After first review, it goes back to the llm with the task to flag issues on the first review based on several friction points. The flagged review goes back to the llm and just the flagged issues get overworked again. Afterwards the review gets showcased on the screen, consisting of friction points, accessibility and user stories (feeling towards this action and system response). On click the user can now compare an older audit to the just created one, showcasing improvements apon changes on the website.
How I built it
I predefined the requirements and the structure I wanted to have for this project. Then I started with a base layer of functionality (working product just not yet really accurately), consisting of OpenAI integration, Playwright integration, prompt engineering, pipelining and routing. After this structure was done I moved from single screenshots to multiple (crawling the webpages via Playwright and adding this to the prompt) and added a description function for the user. I also added later on a login and subscription system. With adding the memory for the user, I could also add a history of audits, to come back to and integrate a comparison function.
Challenges I ran into
At the beginning I forgot to add a file consisting my api key for the OpenAI model to the gitignore and accidently published that. Therefore I had to rotate the key and delete the github commit history, after resolving this problem I could continue quite smoothly.
Accomplishments that I am proud of
Adding the memory layer to the existing framework and therefore adding login, subscription, history and comparison easily to it. I feel like this feature brings the "app" furthermore to life.
What I learned
Look into what is going to be pushed to github, before pushing it. For next time I can save me that hassle. Other than that I solo builded for the first time in a hackathon and had to structure everything for myself and had to limit some ideas, as the time was too little at the end.
What's next for UI Copilot
I wanted to add the feature of navigating through the submitted webpages after the review, with the friction points marked and on click described within the pages. So this will be something for the future.
Built With
- bcrypt
- css
- fastapi
- gpt
- jwt
- playwright
- python
- react
- sqlite
- typescript
- vite
