TactileLens

Inspiration TactileLens — Feel the World Through Your Camera

What if you could feel a texture just by looking at it? TactileLens is an on-device AI app that turns your smartphone camera into a sense of touch. Point it at any surface, crumpled paper, rough concrete, smooth glass, and the app analyzes the visual microtexture in real time, translating it into synchronized haptic vibrations and audio feedback. You don't just see the world. You feel it.

What it does

Use Case: For low-vision and blind users, surfaces carry critical safety information, such as the difference between a smooth tile and a wet floor, or between gravel and grass. TactileLens makes that information accessible through touch, instantly, with no internet connection required. And for everyone else, it unlocks a new way to interact with digital content, "feeling" a product texture before you buy it, or touching a photo as if you were there.

How we built it

We deploy a custom AI texture model directly onto the Snapdragon Hexagon NPU using LiteRT and the Qualcomm QNN Delegate, achieving under 10ms inference with zero cloud dependency. The model outputs four continuous physical axes: roughness, hardness, friction, and density, which drive Android's haptic composition API and a low-latency audio engine simultaneously. The entire loop, from pixel to vibration, runs in under 20ms. Built for the edge. Built for humans.

Challenges we ran into

The hardest challenge was stabilizing the on-device AI inference pipeline. When we migrated to LiteRT 2.x's new CompiledModel API, native SIGSEGV crashes deep inside libQnnHtp.so made the entire app unusable. Diagnosing a crash in a closed-source native library with only Logcat stack traces took significant effort. We also discovered that passing flat FloatArray buffers to the TFLite Interpreter caused silent dimensionality mismatches for 4D tensor models (U2Net's [1, 320, 320, 3] input), which the runtime rejected with a cryptic internal error rather than a useful message. Finally, the material classification was consistently wrong; every surface was classified as Glass because the Kotlin LinearHead class's stored normalization statistics were calibrated on a different EfficientNet checkpoint than the one the model we were actually running used. The weights looked correct on paper, but produced garbage in practice.

Accomplishments that we're proud of

We got the full end-to-end pipeline running on real NPU hardware: U2Net foreground segmentation, texture feature extraction, material classification, and synchronized haptic + audio rendering all at FP16 precision on the Snapdragon Hexagon HTP, completing in under 20ms. No cloud. No NNAPI fallback. Pure NPU inference. We're also proud of the zero-copy ByteBuffer I/O approach we engineered: inference buffers are pre-allocated once, reused across every camera frame, and passed directly to the TFLite runtime, eliminating garbage collection pressure on the inference thread entirely.

What we learned

LiteRT 2. x's CompiledModel API requires pre-compiled QNN context binaries, not standard .tflite files, a nuance not clearly documented at the time. Interpreter.run(FloatArray, FloatArray) implicitly reinterprets the array as a 1D tensor regardless of the model's actual input shape. ByteBuffer I/O bypasses all shape validation and is the correct path for non-1D models. Model weights and the model they were calibrated on are inseparable. Swapping one without the other produces confidently wrong outputs, a lesson that applies far beyond mobile AI. Empirical centroid calibration on the actual device matters far more than theoretical training-set centroids. Real-world lighting, camera sensor characteristics, and JPEG compression all shift the model's output distribution meaningfully.

What's next for TactileLens Broader material vocabulary:

We currently classify 6 canonical materials. We want to expand to 20+ through additional on-device data collection and active learning. Live camera mode: The current flow is photo-capture-then-analyze. The next step is continuous real-time texture streaming, where haptics update as you pan across a surface. Wearable output: Integrating with Bluetooth haptic gloves or wristbands to deliver texture feedback without holding the phone. Accessibility partnerships: Working directly with low-vision communities to validate that the haptic signatures are genuinely useful and intuitive, not just technically impressive.