Real Time Hand Interface

A webcam is all you need to experience futuristic controls

Comment

Inspiration

I watched the Iron man movie, and I thought it was so cool. Tony Stark was pinching 3D model that was projected in the air, throwing it, and turning it around. I think every boys would be attracted by that. Finally, I realized that I can make it in real life. I also watched the Youtube video about Goldman Sachs, and analyst in that video was using 6 monitor. When I realized, my hands was already working on that project.

What it does

Real Time Hand Interface provides futuristic and innovative user experience. In the demo that I made, you can control 3D market dashboard on website. All you need is webcam that can see your hand. If you have good GPU, you may can see smooth movement. (It can be working without GPU) The data on each panels are not real-time. This stock market thing is just an example of this interface.

Here's some rules to control it

  1. When you pinch with your finger, you can rotate it.
  2. When you open your hand, you can stop control it.
  3. When you point only with index finger, you can summon the cursor.
  4. With your index finger, you can control the cursor.
  5. If your cursor is pointing out any panels for a couple seconds, that panel would be bigger mode.
  6. If you want to finish bigger mode, you can open your hand for couple seconds.
  7. You can change some settings by hitting S key on your keyboard.

How we built it

After I got inspiration, I put it into words, and I made specifications document. I put that document into Claude Code, try it, and fix it with Claude Code. I repeated this loop so many times.

Challenges we ran into

The hardest challenge was the limitation of hardware. I only have laptop without GPU, so it was always lagging when I test it. The most frustrating thing was that I couldn’t observe for myself, right up until the very end, just how smoothly I could actually move. The other challenge that I ran into was how to make the control method that moving your hands in the air, something no one is used to yet feel comfortable. I tried every my idea. As a result, this demo is totally different and so comfortable compared to the first prototype.

Accomplishments that we're proud of

I'm so proud of the accomplishment that I show the new interface possibility without expensive machines. I'm pretty sure that every tech people would be excited to try this demo. In addition, I'm proud of the fact that I was able to create something I was satisfied with in a short time.

What we learned

I have learned a lot of things in this project. Firstly, the various strategies I devised to improve usability with unfamiliar controls taught me that “the solutions that survive after trying a large number of patterns in practice are more effective than the correct answers I came up with in my head.” This experience would affect my other projects in the future. Secondly, I learned how to put the abstract concept of “user experience” into words and communicate it to an AI. This translation skill have been improved so much by this project.

What's next for Real Time Hand Interface

If you put multiple webcam in the user's room and get hand movement as a 3D movement, you can get better accuracy, and you can do more detailed control. Also, if projecting technology improve and it can project video in the air, you can built exactly same work environment as Iron man.

Finally, this finance console is just an example of use case. You may can improve immersive VR game experience, or there might be so many other ways to use this technology. I can't wait to see them.

Built With

Share this project:

Updates

Submission history