Home Dashboard
Media page
Navigation page
Phone page
Call Simulation
Night Mode with Ambient Lighting
Gesture Tutorial

GESTURA — Gesture-Based User Recognition for Automobiles

Inspiration

Modern vehicles are equipped with increasingly sophisticated infotainment systems, but interacting with touchscreens while driving can distract the driver from the road. We wanted to explore a more natural and intuitive way to interact with in-car technology.

That idea inspired GESTURA, a gesture-controlled car infotainment system that brings computer vision into the driving experience. Our goal was to create an interface where simple hand gestures can control music, navigate menus, and interact with simulated phone calls, reducing dependence on conventional touch-based controls.

What it does

GESTURA is a browser-based automotive infotainment prototype that combines hand tracking, gesture recognition, and an automotive-inspired interface.

Its key features include:

Hand gesture control: Track hand movements and recognize gestures to interact with the interface.
Media player: Play locally selected audio files and control playback and volume using supported gestures.
Navigation: Search for destinations, view map locations, and preview routes using OpenStreetMap-based services.
Simulated phone calls: Demonstrate incoming calls, accepting or rejecting calls, active-call controls, and call history.
Vehicle settings: Interact with simulated vehicle controls and customize the interface with ambient lighting.
Voice-controlled navigation: Use speech recognition where supported by the browser.

How we built it

We built GESTURA using HTML, CSS, and JavaScript, with a focus on keeping the application lightweight and suitable for static web hosting.

MediaPipe Tasks Vision powers hand landmark detection. It tracks 21 landmarks on a detected hand, allowing our JavaScript gesture engine to interpret hand poses and movements. The gesture-processing logic uses thresholds, smoothing, dwell times, cooldowns, and state management to help prevent accidental interactions.

OpenCV.js provides camera-frame quality diagnostics, including luminance and contrast measurements. These diagnostics help identify lighting conditions that may affect the experience.

We used Leaflet and OpenStreetMap-based services for interactive maps and destination search, with OSRM for route previews. The media player uses browser-based audio playback, while the phone and vehicle controls simulate infotainment functionality without controlling a real vehicle.

The application is designed for static deployment, including AWS Amplify Hosting.

Challenges we faced

One of our main challenges was making gesture interaction reliable enough to feel natural. Hand landmarks can fluctuate as the hand moves, and similar hand poses can be confused with one another. We addressed these challenges through gesture thresholds, cooldowns, and state management.

Another challenge was coordinating real-time camera processing with responsive interface interactions. We had to consider camera permissions, model initialization, pointer alignment, and performance while keeping the interface usable with a mouse or keyboard as well.

Navigation introduced additional challenges involving external search and routing services, network availability, and displaying route information correctly. We also worked on making the interface feel closer to a modern automotive infotainment system rather than a conventional web dashboard.

What we learned

Building GESTURA helped us understand how computer vision can be integrated with interactive web applications. We learned how hand landmarks can drive user-interface actions, why gesture recognition needs careful state management, and how external mapping services can be combined with browser-based interfaces.

We also learned that a polished interface is only one part of the experience: responsiveness, accessibility, graceful error handling, and reliable interaction are equally important.

What's next

Our next priorities are improving gesture reliability under different lighting conditions, evaluating recognition performance, refining voice-controlled navigation, and expanding the system's functionality.

We also aim to strengthen the OpenCV image-analysis pipeline and document its performance through reproducible evaluation.

Responsible use

GESTURA is an experimental infotainment prototype, not a certified automotive system. Its phone calls and vehicle settings are simulated, and it does not directly control a real vehicle. It is intended for demonstrations and development in safe, stationary environments—not for use while driving.