Inspiration
The Museum theme of the hackathon and of course the movie, Night At The Museum helped us find this idea of having the artifacts of the museum "speak" for itself.
What it does
This web app takes an image input from the user's phone, like a portrait, or a statue or the photo of a human being and leverages AI to animate the historical figure to teach history and about themselves.
How we built it
We built the frontend using React + Tailwind CSS while the backend uses FastAPI with Python. We used multiple AI/ML models like Wav2Lip, FOMM, TTS-1, and OpenAI to help with the animation and lip synchronization of the image. Additionally, we used AWS S3 to store our input image and the final video output for faster replay.
Challenges we ran into
Our main challenge was testing each of the models because we never worked with them before and see if they were compatible with each other and finally the hard task of integrating the different components.
Accomplishments that we're proud of
Have a decent functioning prototype ready even earlier than expected. We initially assumed this idea was out of scope and would not even be ready for demo.
What we learned
We learned about new and interesting models like FOMM for superimposing motion onto a 2d image and Wav2Lip for lip synchronization for the phoneme pronunciation to match the audio.
What's next for Anima
1) Multiple language support. 2) Real-time animation instead of uploading a photo. 3) Faster process of image and final video output. 4) Integration into educational systems to make learning not only history but any subject a lot more interesting.
Log in or sign up for Devpost to join the conversation.