Gemini Fashion Advisor

• 

  Architectural Diagram

• 

  Website

Inspiration

I spent a lot of time considering some large issues that Gemini could be applied to, ranging from time series prediction integrations to federal budget modeling. While considering these, I realized that every day I was standing in front of a mirror frustrated, trying to figure out what I was going to wear that day.

What It Does

Gemini Fashion Advisor is a website that allows users to enter photographs and ask questions about fashion or style.

• User uploads a photograph of clothing, hair, makeup, accessories, etc.
• The user can ask a question using natural language about the assistance they need with the photographed item(s).
• Gemini reviews the photograph and question and provides friendly, positive advice on fashion and style.
• Gemini also provides personalized shopping suggestions that link to the photograph and user's style/concerns.

How I Built It

I used the Gemini API with Python and Streamlit to create a fully functional, interactive website with an appealing front-end.

• After the user submits a photograph and question, the photograph is processed and prepared for sending to Gemini along with the user's question.
• Gemini Pro Vision receives the data and uses a detailed prompt to create output in markdown format, which is sent back to the Streamlit front end for display.
• Gemini Pro Vision also sends their response to the Gemini Pro model which parses the response and using another detailed prompt, generates a search query based on the Gemini Pro Vision suggestions.
• The search query is sent to the Ionic Shopping API which returns customized shopping suggestions for the user that are displayed via the Streamlit front end.

Challenges I Ran Into

I wanted to use an entirely Google product, so looked for ways to integrate Google Shopping via API. Unfortunately, I was unable to find a way to do so, as Custom Search Engine does not return Google Shopping results correctly even when restricted to this site, and the Shopping APIs are focused on Merchant accounts and providing access to a merchant's specific products instead of displaying products from across the Shopping ecosystem as search results.

Also, the documentation of Gemini API vs Vertex API aren't especially well documented. There is a lot of cross-over of functionality (functions/tools), but these use differently formatted code syntax. In the end, I opted to use only Gemini API to streamline and eliminate some of the issues I ran into when trying to determine the syntax for both.

Accomplishments I'm Proud Of

I'm proud that I was able to use minimal coding to create a website with impressive functionality. The combination of Streamlit with Google Gemini API had a high ease of use, with the potential to expand and customize this product in multiple ways.

What I Learned

I learned that the Gemini API is very easy to work with, and the models are quite impressive, especially when it comes to recognizing what is in an image. The Gemini Pro Vision model was able to accurately determine clothing, color, style, and multiple other subtleties in an image and provide great advice.

What's Next for Gemini Fashion Advisor

My next steps include:

• Learning Android development to create a phone app for Gemini Fashion Advisor.
• Including more ways to get personalized recommendations.
• Using Gemini Pro 1.5
• Expanding the advice to a chat session and allowing multiple photographs
• Custom training a Gemini model using street fashion sites and daily updates to keep it on the cutting edge of style.

Built With

• google-gemini
• google-generative-ai
• ionic-shopping
• python
• streamlit