Point Smart

Shell Hacks Hackathon 2026

Presented by:

Madden Deal & Sarah Funderberk (Mississippi State University)

Gabriel Lozano & Claribel Hernandez (Florida International University)

Inspiration

Our group assembled through the ShellHacks Discord channel. We started as strangers and through getting to know one another we identified a common interest - accessibility. Each of us has personal experience benefitting from accessibility tools and missing out due to lack of accommodation.

That shared experience led us to think about accessibility in the home- the smart home to be more specific.

What does Point Smart do?

Point Smart is a smart camera and companion web app that recognizes when a person points at a Google Home enabled device to toggle it, eliminating the need for verbal activation of Google Home speakers or walking to a device to interact with it.

For example...

💡 A Deaf woman wants to turn a lamp on in her home, so she points at the light and it turns on!

📺 A wheelchair bound man wants to turn on the TV, but his phone and remote are out of arms reach, so he points at the TV and it turns on!

How Point Smart was built

The smart camera was created by combining an ordinary webcam and a Raspberry Pi using Google MediaPipe and Yolo World to recognize when a user points to a Google Home enabled device.

When the pointing gesture is made, a vector "shoots" out from the pointer finger. When that vector intersects with a hit box surrounding a device, the smart camera communicates with the companion web app.

On the companion web app, the device ID is recieved and passed through Google Home API asking to toggle that device.

Gesture -> Camera -> Web App -> Google Home -> 💡 📺 🔒

Challenges during the build

Privacy Considerations: In a world of shrinking privacy, we wanted to find an appropriate balance between convenience and privacy especially because our product is a camera in a home. Therefore, we decided to keep all the processing on the smart camera. Neither the web app nor the Google Cloud see footage from the camera unless the user navigates to the live feed.
Additionally, the user is able to set "Do Not Disturb" mode on for a duration of their choosing as well as turn the camera on and off through the web app.
Gesture Recognition Through the Camera: The training of the AI model was particularly challenging because of the latency of the camera feed and diminishing frame rate limiting the ability to reliably recognize the gestures. As a result, this portion of the project needed to be redone up to 4 times.
Google Home API: This was our team's first time working with Google Home API and that meant we had to learn about the package library and which would work best for our program. The biggest challenge was authentication and working around the traditional audio method of triggering Google Home.

Accomplishments

Gesture Recognition: The big breakthrough was simplifying the 3D capture field to a 2D. The joint calculations were reduced to the necessary shapes instead of excessive detailed joints.
Google Home API: We were able to successfully authorize, not one, but two separate gmail accounts with our API. For the demo, we were able to set up a "Shell Hack House" Google Home in a classroom, complete with two lamps and a dehumidifier, to test real cases (to much success!).
Team development: This project was the first team build for all of the members and the dynamic was harmonious. We stepped away from the computers periodically to have honest team meetings. Each member's voice was heard no matter their opinion and we were all clear and aligned at each step.
As a group, we mapped out the development plan and delegated according to our interests, strengths, and weaknesses.

+Alive and (mostly) well! Two team members were ill during this hackathon weekend - but lack of sleep, a cold, or a stomach bug won't stop this team!

Lessons Learned

On the technical side...

Pay close attention to versions of packages and languages as they are critical to frictionless communication between layers.

On the personal side...

It's important to be honest about progress and personal bandwidth.
Don't underestimate what you can learn.

Next Steps for Point Smart

The possibilities for Point Smart expand far past what we accomplished at Shell Hacks 2026...
Smarter gesture recognition for more commands (adjust volumes, temperatures, etc.)
Multiple camera network for expanded range