AutoRoaster

Friend or foe? Yay or no? Send a compliment to someone to tip your hat, or a sly roast for a nice laugh. All powered and automated by the latest deep-learning innovations in vision-language!

Comment

Inspiration

This project was inspired the recent debates around the ethics of artificial intelligence, and our desire to weight the balance between the positives and negatives of artificial intelligence. In that sense, we wanted to create a project which deliberately displayed both of these sides in an engaging manner. Still, we wanted to keep the project on the light-hearted side, allowing friends to roast and compliment each other for a laugh with the help of artificial intelligence.

What it does

This project links together several APIs, so that the user can take a saved/webcam picture and generate a deep-learning powered compliment or roast. The program runs through Python and takes a picture of the user, which then gets saved onto the User's local server. Afterwards, the project runs the picture through an open-source image-to-text AI, which auto-generates a caption that describes the picture. We then feed the caption with a prompt prior through another open-source sequence-to-sequence model, which generates either a roast or a compliment depending on the user's preference.

How we built it

We used the Replicate AI and OpenAI APIs to access the image-to-text and sequence models. We built this project using Python, HuggingFace, and OpenAI; the website draft is currently under construction using HTML, Javascript, and Python Flask as a server.

Challenges we ran into

The initial task of linking and parsing the APIs was challenging, and took collaboration and lots of research to successfully provide a functional program. Also fine-tuning the models and parameters to make them better match our task was difficult, but it helped us improve our technical understanding. Our ultimate goal for this project was to create a website on Flask server that took a picture of the user and ran the Python program. Although our index.html and roaster.js files successfully created a local website that took a picture of the user, we ran into a challenge while successfully bridging the website and our Python program through Flask. We explored different options such as building everything on Python rather than Javascript and creating a simple Flask server to upload it.

Accomplishments that we're proud of

We were able to create a program that takes a picture of the user, stores it locally, and auto-generate messages depending on the user. We definitely enjoyed playing around with the program, and we hope that others can too!

What we learned

We improved our understanding of OpenAI and Replicate's APIs, and also improved our experience with Computer Vision, Deep Learning, and the ethical side of AI. We were also able to work on a hands-on project where we could see how easy it is to use artificial intelligence to compromise values like privacy, accountability, etc.

What's next for AutoRoaster

We will create an end-to-end website that processes webcam images and returns roasts or compliments in real-time, so that everyone can access it from a public domain.

Built With

Share this project:

Updates