Drip or Drown

Elevate your visual aesthetic with your personal AI stylist. 💦

Comment 1

What it does

Our site allows a user to take a picture of their outfit and receive AI-generated feedback on their style, color palette, and cohesiveness. While we are not your typical education tool, we think that dressing well has a massive boost on your confidence, your presence, and your ability to talk about yourself! Our goal is to give everyone a chance to be their most styling self.

In addition to hyping up the best of your outfit, we recommend relevant pieces to incorporate into your style as you explore defining your visual voice. Discovering fashion should be exciting, not intimidating -- and with the help of LLM's, Drip or Drown does exactly that!

How we built it

During the course of the weekend, we experimented with many different approaches but ultimately settled on the following architecture (click here for full res): architecture. Here's some of the salient elements:

  1. Depth Perception When an image is uploaded, we use computer vision to remove noise. We use the MIDAS depth perception model to separate the background from the foreground, create a separate image only of the person and their fit!

  2. Visual Transformation Model (Q&A): We used the ViLT QnA model to query the de-noised image about attributes about the outfit — extracting information about what they're wearing and details in a recursive fashion.

  3. In-Context Learning Fashion Critique Model Finally, we use Large Language Models to generate in-depth feedback for your fit using our description. We also use a model to rate it on a scale of 1-10 and another to categorize it into one of many "aura"s, so that you can have a well-rounded understanding of your own style!

Challenges we ran into

  1. Figuring out deployment was particularly hard. Given the many moving parts, our project is hosted on multiple platforms that all interact with each other.
  2. Using non-production research models like ViLT meant that our performance for the API calls was quite abysmal, and we had to get pretty clever about parallelization and early-stopping algorithms within our call structure.
  3. We spent a lot of time adding juice to the project! Making it fun to use was a big goal of ours, which was often easier said than done :)

Accomplishments that we're proud of

  1. The UI! We think it looks pretty great – one of our teammates mocked it all up in Figma, and then we spent most of Sunday night making components together. We really wanted to make our project fun to use, and I think we accomplished that.
  2. Using Multi-modal AI! It's one of the biggest unsolved problems in AI right now. How do you use multiple forms of input — image and text — together? I think we came up with a pretty clever solution that works quite well, and is pretty interpretable as well!

What we learned

  1. A lot about many technologies! Like Flask, PythonAnywhere, AWS, Heroku, Vercel, generating particles, CSS, Visual Transformer models, GPT fine-tuning, image progressing, classification algorithms, and more! This project spans many different domains, and it was pretty fun to pick up skills along the way.
  2. The need for patience! For the longest time, we would have "blocker" bugs that would prevent us from deploying or developing further. We pushed ahead, and every time we handled those, the emergent abilities of the system surprised us as well.
  3. And of course, that having fun once in a while is important. We did some of our best work when we were all singing to pop songs together at 11PM.

What's next for Drip or Drown

  1. Improving suggestion quality: We think we can push this even further! While our current image -> text algorithm is clever, we think we could make it even smarter by using a shared embedding space between images and text. This could capture attributes of the image our QnA model could not!
  2. Follow-ups and conversation: We'd love for you to be able to ask the model questions about your fit! "Why does a white belt work better?" "What do you think about this leather jacket with that shirt?"
  3. Suggestions! Finally, from all the feedback, we'd love for the model to be able to suggest fits as well. "You'd look great in a green croptop for this casual event!" We hope that Drip is the AI assistant to help you achieve your most fashionable self.

Built With

Share this project:

Updates