Inspiration
As two experienced hackathon-goers, we have experienced many a messy table, where papers, junk, and laptops, obscure important parts, technology and flyers, causing increased friction. During the development of a hackathon project, everyone's time spent searching for lost things can mean the difference between a successful project and a hardcoded presentation.
This problem isn't just unique to hackathons though. At home, you would be lying if you say you haven't spent hours searching for something small only to find that it was right under your nose, barely obscured by a stray notebook.
To help solve this problem, we built Palace.
What it does
Made possible only by modern, multimodal AI models like Qwen3.5-Flash, Palace uses an overhead camera to scan the visible desk area whenever it detects some form of movement. It then catalogues every item that it sees in its spacial memory. When an object is placed such that another object is obscured, Palace doesn't forget about it like a newborn playing peek-a-boo, but rather demonstrates the object permanence skills of an 8-month old!
Later, when the user calls the wake word "Palace", it uses Qwen's audio and natural language processing to understand what the user is looking for, matches it to its catalogue, and accurately words its position so that it can be found by the user.
How we built it
Palace is built using the Qwen3.5-Flash model through yibuAPI credits provided by Huawei for the OMNI Live multimodal AI prize track. It also uses DynoV3 as a local ML model in order to detect object movement whilst avoiding the high latency of a cloud model, only using it when required.
Challenges we ran into
One challenge that we ran into was the lack of documentation in the yibuAPI service, which made building our own software around it more difficult. Apart from that, the integration of all the independent systems such as the video and audio processing, as well as the front end system was difficult as unsuspected problems always found a way to pop up.
Accomplishments that we're proud of
The live scanning and spatial memory was very cool, and it was a lot of fun to watch it track moving objects around the screen, even when they are hidden or obscured.
What we learned
When it comes to computer vision, depth processing, shadows, and proper camera focus are some of the biggest factors to get a clear reading. We had to find creative ways to combine multiple data sources to obtain the highest accuracy.
What's next for Palace
The biggest next steps for Palace is definitely reliability. While the proof of concept is there, all the data doesn't always end up combining properly. When it works it is highly successful, but often times in loud rooms, varying lighting conditions, and in the presence of similarly shaped and coloured objects, it can end up getting confused and not producing a reliable result.
Log in or sign up for Devpost to join the conversation.