Inspiration

  • In 2022, my (phuthai) mother passed away as I was graduating RPI (now doing masters). Kelly and I were inspired to use AI to help re-live interactions with past (or current) loved ones that are no longer near. Keeping with "Retro v Modern", my mother and many people in our age group have parents who were living it up "retro" 80s. We put a happy spin and and theme on this interaction, and hopefully bring joy/entertainment to anyone who wants a retro version/conversation with a person.

  • NOTE: IF YOU VISIT THE URL, YOU MUST BE ON RPI WIFI. I BLOCKED SUBNETS 4 SECURITY

What it does

  • Upload an image of anyone, and watch them go 80s!

  • Pre-Prompt the interaction, and carry on unforgettable conversations in a variety of voices!

  • Render an interaction, and watch as the image comes to life!

How we built it

  • The infrastructure is built in VULTR cloud, deployed via Terraform

  • We used Text->Text and Text+Image->Image models from Gemini API

  • We audio/voices from ElevenLabs

  • Build on a simple Python/Flask, HTML, CSS, JS tech stack

  • Used a custom Wave2Lips LipSyncing model to bring images to life!

Challenges we ran into

  • Because we ran our own local machine learning model to create lip-syncing images, the Python required heavy CUDA/CPU usage. We had to use the more expensive VULTR VMs to continue running remotely

  • Because of computational requirements, I neglected to perfect the Kubernetes deployment. The Cluster is up, but the app is not functional on Kubernetes

  • Web Cam doesn't work when run on the server, only locally

  • There are 2 versions of Google Gemini's APIs, that aren't compatible. Using AI to learn this API causes issues since even Gemini mixes up the APIs. (google-genai vs google-generativeai)

  • Repo is private, and we'll offer on request or make it Public after the Hackathon ends. We did not do a good job cleaning up the API keys from the repo + repo history

  • We both hit CoPilot prompting limits

Accomplishments that we're proud of

  • Most things worked! The coolest parts of the project worked

What we learned

  • A lot more about APIs and reading docs, trusting it over AI recommendations

What's next for Echo Of You

  • Find a permanent home outside of the unique temporary IP address it currently lives on

Built With

Share this project:

Updates

Submission history