Design Review

Inspiration

In the recent advent of very capable Generative AI models I became quite inspired to build things, to solve problems for myself and others. Two things came together with this technological breakthrough as a designer and a novice programmer:

1. I could now build things that weren't possible or were very difficult for a single person to build before.
2. I can build them much faster and easier with little technical knowledge.

So, I started building many AI tools to solve my problems or that of others.

I have been thinking of how to use AI to improve my design processes. Once I heard about the Gemini 1.5 pro, its 1 million token context limit and its multimodal features, I waited eagerly to have access so I can see what it was capable of and what new problems it can solve for me and others. Not long after I got access and was exploring the models capabilities, I heard about this hackathon and I decided to work on this idea for the hackathon.

What it does

This is a Figma plugin or website (I created a web version in case people want to test or use outside of Figma) where you give it your UI design and reviews it and gives you detailed feedback on how to improve the design.

How I built it and Challenges I ran into

I first of all prototyped it using Google AI Studio to see if it was possible. When I saw it worked, I worked on improving the prompting. Next I copied the code to test it on a Notebook but it didn't work correctly. I started playing around with the Vertex AI platform and google console. It was really difficult to understand but with trial and error I was able to figure it out. I prototyped it again in Vertex AI and copied the code to Google colab and it also didn't work at first. Some code were missing which I had to write and many errors which I resolved after lot of trial and error and research. I then moved the code to Vscode on my computer and setup a python server that will receive requests from the client. I moved on to create the Figma plugin so it can be used within Figma directly. I spent time reading Figma plugin documentation and then setup the plugin. The plugin detects the frame or image selected by the designer and when submitted it would convert to base64 and send it to the server. The server receives it converts it back to an image/PNG and sends it alongside the prompt to Gemini 1.5 pro model API, when the response is gotten the server sends it back to the client(plugin) and it is displayed to the user. One challenge that I had and I am still figuring out is accurately displaying the response as a formatted text, methods I have tried haven't worked accurately, I will solve this after the competition. After creating the plugin and it worked locally, I decide to start deploying them. I deployed the server on the Render platform, here I faced multiple challenges because my code was no longer working - most of the challenges had to do with Google cloud services. But after days of figuring things out I was able to resolve them. One of the solutions had to do with setting up a service account, another setting up a credential json file and the other major one was assign a role of Vertex AI Admin to a service account. The server was up and running, I published the plugin to Figma and it was approved. Everything was finally working fine. I decided to create a web version so those who didn't have access to Figma can use the website to upload their designs and get detailed feedback.

While developing this solution, I prioritized Privacy & Security by not storing users' data, specifically their designs. In my approach, I emphasized Accessibility as a key area for providing feedback to designers, ensuring that their creations are usable by everyone. Additionally, I made sure to accompany each recommendation with a clear rationale to enhance user understanding. However, it's important to note that feedback may not always be entirely accurate. To maintain transparency, I included a disclaimer at the bottom to inform users of this possibility.

Accomplishments that I am proud of

I am really proud that was able to build a useful product not just for me but for others while having basic programming knowledge.

What I learned

1. I learned perseverance in the face of challenges.
2. How to use Google cloud services
3. How to use Gemini API

What's next for Design Review

1. Added a knowledge base file with design knowledge where the model can get information on how to improve designs.
2. Be able to not just provide feedback but to show the user examples or implementation of the feedback.

Built With

• figma
• flask
• gemini
• google-cloud
• python
• render
• typescript
• vertex
• vertex-ai