THEIA

• 

  output interface pointing out where a small necklace is located on a messy desk

• 

  simple user interface greets the user after logging in to allow for ease of use

• 

  reassuring message during processing to ensure the user that progress is happening

• 

  simple but functional login page with database memory for future sign ins

Inspiration

People living with dementia, and people who are just forgetful, lose track of ordinary things. The glasses are on the desk. The earbud case is by the keyboard. Asking the group chat, or opening another app, does not put a point on the object.

That is why we built Theia, a tabletop turret. You ask for the object. It looks around the room, circles it on screen, and gives the angle a laser can point to.

What it does

You say what you are looking for, or type it. When you finish speaking, the search starts. A phone camera takes seven photos. Theia picks the best match, draws a mark on it, and shows azimuth and elevation in degrees. If the object shows up in more than one photo, it still circles the strongest match and shows that angle. If nothing matches, it says so.

How we built it

The page is React. The server is Flask. Voice uses the browser’s speech recognition and submits when you stop talking. The camera is a phone running DroidCam. Each search saves a new set of photos and sends them to a vision model. The model returns a box. We turn the center of that box into an aim angle and draw it on the photo.

Challenges we ran into

The same object appears in several overlapping photos. At first that counted as multiple matches, so the page drew a box and left the angle blank. We also do not have the motors connected yet, so every photo is tagged with pan and tilt at zero, and the same object looks like it moved. A small object, like an earbud case, got a box that sat too low, on the keyboard instead of the case. Voice submit also broke once and had to be put back.

We also faced significant issues connecting the hardware with our software as it was difficult for the Arduino to communicate with the software.

Accomplishments that we're proud of

You can ask for an object and get one circled photo with a real angle. On the earbud-case test, the mark sits on the case, not on the mouse next to it. Several photos of the same thing no longer block the result. The search stays around twelve seconds.

What we learned

Most of the wait is the vision call and the camera sweep, not our own code. We also learned that a finder has to pick one point. Telling the user the scene was ambiguous is the wrong result when they only need the object pointed out.

What's next for Theia

Connect the pan-tilt motors and the laser so the angle becomes a dot in the room. Measure the camera’s real field of view. Keep the same question for home, a caregiver, or a clinic closet: find this.

Built With

• arduino
• cad
• computervision
• css
• cursor
• html5
• javascript
• python
• ui
• webcam