Handsense

Our v1
Our v0 beta
The simulator scanning screen
The lunar landscape
The simulator scanning screen for the rover
The drill press
The rover
Initial Drill Press Screen when starting
Drilling into soft rock
Drilling into hard rock

Inspiration

Remote exploration asks people to understand unfamiliar environments through screens and telemetry. We wanted to bring another sense into that experience: touch.

Our idea was to build an affordable wearable interface that lets operators control equipment through hand gestures and receive physical feedback about what that equipment encounters. With HackHarvard’s space theme, lunar exploration became our testing ground.

We also wanted the interface to adapt to the person and the task. Driving a rover and operating a drill require different controls and convey different information. One glove should be able to support both.

What it does

Handsense connects a physical haptic glove to an interactive lunar exploration simulator.

The glove has five vibration motors positioned around the base knuckles of the fingers. An ESP32 controls the motors, with the electronics mounted on the wrist and powered by three AA batteries. A phone worn as a head camera captures the operator’s hand, and computer vision translates gestures into equipment controls.

On the laptop, users explore terrain reconstructed from NASA’s Lunar Orbiter Laser Altimeter data near Shackleton crater. They can drive a simulated rover and operate a drill while receiving feedback through the glove.

The feedback changes with the task. Driving produces rumble, directional terrain feedback, impacts, and slope warnings. Drilling produces different sensations for free spin, soft material, hard material, and a stalled bit. These are vibration cues representing simulated equipment conditions.

Each glove also has an NFC tag. Scanning it identifies the wearer, loads their personal feedback settings, and activates nearby equipment in the simulator.

An AI mission-control interface adds spoken interaction. Users can ask about equipment status, adjust haptic sensitivity, or ask why their glove is vibrating. A Mission Assistant connects those questions to equipment telemetry, terrain information, and the current haptic state. Personalized mission patches add a visual identity for each operator.

How we built it

We built the wearable around an ESP32, five vibration motors, PWM motor control, and a battery-powered wrist assembly. The laptop sends motor commands over Bluetooth Low Energy.

A Python FastAPI hub connects the glove, NFC reader, phone tracker, simulator, and AI services. Its haptic engine converts equipment telemetry into five motor intensity values, combining continuous sensations with distinct event and warning patterns.

The simulator uses TypeScript and Three.js. Our terrain pipeline processes NASA LOLA elevation, slope, coverage, and uncertainty data into a navigable environment. Fine terrain details and material hardness are simulated properties.

The hand-tracking pipeline supports MediaPipe on the phone and an optional AnyHand-based tracker on the laptop. Hand landmarks become finger-curl and palm-roll measurements for grabbing, throttle, steering, braking, and drilling.

For AI, we implemented Grok Voice tool calls, and Grok Imagine mission-patch generation. Voice actions pass through application-side validation. The assistant includes structured responses, grounding checks, and a local rules fallback.

Challenges we ran into

Tracking a gloved hand reliably was a major challenge. Gloves obscure visual features, and unstable predictions can turn into unintended controls. We added calibration, filtering, gesture thresholds, and an alternative tracking pipeline.

Another challenge was making five small vibration motors communicate useful information. Stronger vibration alone was not enough: we developed different spatial and temporal patterns for terrain, drilling, impacts, and warnings.

Finally, we had to coordinate camera frames, simulator updates, NFC scans, Bluetooth commands, and AI responses. We designed the system so the core haptic feedback continues independently of slower AI requests.

Accomplishments that we're proud of

We connected a physical wearable to a complete interaction loop: hand gesture, simulated equipment action, telemetry, and tactile feedback.

We’re proud that personalization changes how the system behaves. NFC profiles load different sensitivities and feedback mappings, while equipment modes change both gesture controls and tactile cues.

We also gave the AI a concrete role: helping the operator understand the sensations and equipment state, with access to the same context that drives the glove.

What we learned

We learned that haptic feedback needs a vocabulary. Location, intensity, timing, and context all affect whether a vibration communicates something useful.

We also learned the value of separating time-sensitive control from conversational assistance. Equipment feedback must remain responsive while an assistant interprets a question.

Working with lunar terrain data taught us to distinguish source measurements, interpolation, and synthetic detail—and to communicate those boundaries clearly.

What's next for Untitled

We want to test whether users can reliably distinguish our haptic cues and whether those cues improve task performance.

Next steps include comparing operation with and without feedback, measuring end-to-end latency, improving tracking across users and lighting conditions, and replacing the breadboard with a compact wearable enclosure.

We also want to connect the interface to a physical rover or instrumented tool. Our longer-term goal is a configurable interface for exploration, training, and equipment operation in hazardous environments, along with having a system to continuously monitor astronauts surroundings.