Fluffy

Fluffy dimming the lights for a person who notes that 'it is too bright.'
Fluffy!
Initial mock-up draft design thing.

Inspiration

Our original idea was a physical home assistant with the Perceptron track's SO-101 robotic arm. When we were not selected, we pivoted to a more general home assistant, this time focussing on instinctive queries, something we took inspiration from Muse, Meta's virtual assistant. We decided to specialize on elderly care, since that is something we both see in our daily lives.

What it does

Through steps of labelling, inference, tool call, and execution, Fluffy aims to better the user's experience the best way that it can. Fluffy first labels each object and person in the view with respect to a reference frame (right door, left wall, etc.), then it assigns ownership and relative position based on its labelling with Perceptron's Mk 1.5 VLM. Next, Fluffy infers the next step to do based on the context of the situation both verbally and visually. Ex: person scratching his throat and coughing -> he probably wants water. From this inference, the model makes a tool call depending on the previous inference. We had a set number of tool calls like text, play song, dim lights, open windows, notify caretaker, etc. Execution is the final step where Fluffy executes its series of tool calls and waits for a response for the user. Since this is a continuous stream of data, Fluffy is mostly in the waiting stage, always searching for the correct context to help the user.

How we built it

We built our entire codebase using only Cursor. Our planning was mostly based on research papers of cascading instinctive models and how they behave as well as how modern smart homes execute their tasks. We also made use of the APIs provided in the DubHacks Discord server and Hacker Guide (Spotify API and Photon API) to play music and send emergency texts. Since we were vying for the Perceptron robotics track, we already had the Mk 1.5 in our vision of an ideal Fluffy, but the lack of arm changed our direction from grasping objects for delivery to just an agent meant to assist elderly folks with Perceptron's VLM capabilities.

Challenges we ran into

Cursor was an unfamiliar agent to both of us, and we were unfamiliar with the application before Saturday. Additionally, because SpaceX was unable to provide their free $25 Grok voice credits, our original plan of Mk 1.5 scene description + Grok voice -> Grok reasoning/inference -> API tool calls & execution changed to a Mk 1.5 inference, which gave us less computational awareness, since Mk 1.5 is a lighter model. Additionally, the Mk 1.5 transcribing ability was extremely slow for our purposes, so we had to switch to a Chrome speech-to-text program. Also, one of our team members got his GitHub account banned half-way through coding (oopsies) and the other ran out of Cursor credits so our programming capabilities were slightly impaired around 2AM Sunday, but we persevered.

Accomplishments that we're proud of

Being relatively new to the scene of hackathons, we are both very proud that we managed to stay up the whole night and devote our utmost focus onto this project. Moreover, we were able efficiently utilize the hackathon tools through planning and scaffolding our project. This is something that we struggled with in our past hackathons (and just projects in general), and finally being able to realize 'planning' became a very integral part of our hackathon journey at DubHacks.

What we learned

We learned how to utilize with agentic workflow through the persistent use of cursor. During the night, we had, at times, 8 agents running simultaneously. With our planning, we were able to properly designate tasks to these agents, helping us work efficiently and successfully.