PetPOV

Inspiration

We often dream of communicating with our pets, what if we actually could? Imagine getting texts from your cat while you're at school, or being able to text and check in with your dog at any time in the day. In Turing City, AI enables interaction across species.

What it does

An AI Agent living within iMessage that links to a live feed from a camera on your pet's collar, sending proactive texts about your pet's day, images included, when your pet has something to show you allowing you to send messages back and forth with your pet depending on what your pet is up to, just like texting your best friend.

How we built it

We used a camera to live stream video to Overshoot, allowing for live monitoring from your pet's POV on their day. Taking that stream from Overshoot, we monitor your pet's activities and automatically send texts and images via a Photon iMessage agent to the owners when Overshoot detects certain activities. Moreover, we cache snapshots of recent images taken from the camera live stream so when the user reaches out to the pet, a message will be sent taking into account the pet's most recent activities and sights, modifying it's tone, energy, and response.

Challenges we ran into

One of the largest challenges we ran into was supporting and processing the livestream footage. Originally, we had planned to stream from a 0v7670 connected to an ESP32 via wifi. Firstly we faced issues with getting data from the camera in the first place, especially with the 0v7670 lacking onboard memory. To get data, we decided to write custom libraries to use the ESP32's I2S DMA peripheral to read data fast enough to support the 0v7670 module. However, despite getting valid sync signals, we were unable to get pixel data from the camera reliably, most likely due to hardware issues. Thus, we decided to pivot to streaming from iPhone where we faced more challenges. The Overshoot SDK runs in browser and requires a real MediaStream from the camera API, meaning it can't accept HTTP image endpoints or video files directly. As such, we had to design a different architecture where the iPhone runs the web app with Overshoot (accessing camera natively via Safari), then forwards results to the Mac backend via WebSocket and we need to manually capture/transmit JPEG snapshots for the photo attachments since Overshoot only returns text descriptions, not images.

Accomplishments that we're proud of

I am proud of the UX, especially the livestream and constant feed that allows for proactive updates from pets, something I think that resembles real human interaction and allows the AI pet to feel extremely human. Moreover, by taking in camera data and modifying the mood and based on the recent video feed I feel like also does a really good job of allowing for realistic and human-feeling interactions with the AI agent.

What we learned

I think one of the biggest things we learned was the complexity of building embedded systems and trying to exploit sensor inputs to augment our responses. While we faced many challenges with the hardware component, I think having the real-time video inputs is what allows for the magical experience that PetPOV offers, allowing for more tailored AI personality and proactive messaging based on real-time updates from the camera inputs.

What's next for PetPOV

We plan to partner with smart-collar providers such as PetPin to offer our software as an additional feature to their product offerings. While most smart collars or camera collars are used for just tracking of pets or for content creation from pet-influencers, I think by adding PetPOV software we can create a cool and fun interaction for all pet owners with their pets, greatly expanding the market of smart collars.

Built With

• overshoot
• photon
• typescript