Inspiration
My friend Ren from MIT is a PhD major in material science. He realized that lob logs are always missing and people are having trouble to reproduce or debug their experiments.
What it does
It coverts the video streaming from the web cam into logs that describe the event and has a timestamp. People can ask query about historical events or ask it to send alert on certain event.
In a simple scenario, you can put a camera in your garage, if your garage is open for more than 10 minutes, call an API to close the garage door.
How we built it
We used React to serve user, and then periodically call llava to interpret the key frames. These logs are store in Vectara. We then built an agent using Fetch ai to handle the alert. In our demo, we will show a banana rebalancer that holds the worker information and coordinate who should be refill the bananas from the micro kitchen.
Challenges we ran into
It took us some time to figure out how to prompt the Llava model so that it can generate the raw text for the log.
Accomplishments that we're proud of
We made it work end to end:)
What we learned
How to use Groq api, how to store in the vector db in vectara.
What's next for Video to logs
Look for more use cases and see if we can make it into production.
Log in or sign up for Devpost to join the conversation.