Talk to the Hand-Voice to Robotic Amazing Hand Sign Language

A real-time multilingual voice-to-sign-language system — Amazon Nova 2 Sonic captures speech, a Strands Agents AI agent on NVIDIA Jetson Thor drives a physical robotic hand that fingerspells ASL.

Comment

Talk to the Hand: Real-Time Voice-to-Sign Language Translation with AWS AI and a Robotic Hand

Inspiration

I've spent the last 2 decades building software — web apps, cloud architectures, the usual. But around five years ago, I got hooked on IoT. It started with a smart cat feeder for my pets, grew into a 4x4 telemetry system strapped to my Suzuki Jimny, and before I knew it, I was flashing firmware over-the-air with Strands Agents. Each project pulled me a little further from pure software and closer to the physical world.

I tend to build new solutions by combining building blocks from past projects with the next new thing I learn. I'd recently learned about Amazon Nova 2 Sonic and its real-time voice capabilities, and around the same time I got my hands on an NVIDIA Jetson Thor and a Pollen Robotics Amazing Hand — an open-source, 3D-printable robotic hand with 4 servo-driven fingers. The Amazing Hand was new territory, Nova 2 Sonic was fresh in my mind, and I wanted to see what would happen if I connected the two. The hand sat on my desk for a few days, and then the idea hit me:

What if you could speak into a browser, and a robotic hand would fingerspell your words in American Sign Language — in real-time?

The concept brought together everything I'd been learning: real-time voice AI, cloud-to-edge IoT messaging, agentic tool use, and physical hardware control. It was the kind of project my past self would never have attempted — and that's exactly why I wanted to build it.

What I Built

Talk to the Hand is an end-to-end system that translates spoken words into ASL fingerspelling on a physical robotic hand, with a synchronized 3D animation in the browser. Because Nova 2 Sonic handles the speech processing, the system supports all languages that Nova 2 Sonic understands — someone can speak in Japanese, Spanish, French, or any other supported language, and the system automatically translates their speech to English and fingerspells it in ASL. The system spans three repositories and three blog posts:

Part 1 — Voice Capture & AI Processing (Blog · Repo)

A React web app captures speech from the browser microphone, streams it to Amazon Nova 2 Sonic via bidirectional streaming on Amazon Bedrock, and publishes the cleaned sentence text to AWS IoT Core via MQTT. The app also renders a procedurally-generated Three.js 3D hand that mirrors the physical hand's servo angles in real-time via GraphQL subscriptions.

Part 2 — Cloud Infrastructure (Blog · Repo)

An AWS CDK stack that routes hand state data from IoT Core through Lambda to AWS AppSync, completing the real-time feedback loop. The IoT Rules Engine extracts the device name from the MQTT topic path, and a Lambda function flattens nested finger angle payloads into a flat GraphQL schema for DynamoDB persistence and subscription broadcasting.

Part 3 — Edge AI Agent on NVIDIA Jetson (Blog · Repo)

A Strands Agents application running on an NVIDIA Jetson Thor receives MQTT commands, invokes Amazon Nova 2 Lite to select the fingerspell tool, and drives 8 Feetech SCS0009 servos to spell each letter in ASL. A background thread records the hand via OpenCV, encodes to H.264 MP4, and uploads to S3 with a presigned URL.

The Data Flow

$$ \text{Speech} \xrightarrow{\text{Nova 2 Sonic}} \text{Cleaned Text} \xrightarrow{\text{MQTT}} \text{Strands Agent} \xrightarrow{\text{Nova 2 Lite}} \text{Servo Commands} \xrightarrow{\text{MQTT}} \text{3D Animation} $$

How I Built It

Audio Pipeline (Browser → Nova 2 Sonic)

The browser's Web Audio API captures microphone input via an AudioWorklet running in a separate thread. The worklet accumulates 2048 samples per buffer, then the main thread resamples from the device's native rate (typically 48 kHz) down to 16 kHz using linear interpolation, converts Float32 to PCM16, and base64-encodes the result for JSON transmission to Bedrock.

One key design decision: I configured Nova 2 Sonic not as a chatbot, but as a "dumb speech-to-text relay pipe." Using forced tool use (toolChoice: { any: {} }) and a carefully crafted system prompt, the model cleans up grammar, removes filler words, translates non-English to English, and invokes a send_text tool with the cleaned sentence — every single time, without adding commentary or refusing messages.

Cloud Glue (IoT Core → AppSync)

The CDK stack is intentionally minimal (~74 lines of TypeScript). An IoT Rule with the SQL query:

SELECT gesture, letter, ts, fingers, video_url, topic(3) AS device_name
FROM 'the-project/robotic-hand/+/state'

extracts the device name from the MQTT topic's third segment and routes to a Lambda function that flattens the nested finger angle JSON (fingers.index.angle_1 → indexAngle1) and calls the AppSync createHandState mutation. The CI/CD pipeline auto-discovers the Amplify-managed AppSync configuration from SSM Parameter Store and CloudFormation outputs — no hardcoded values.

Edge Agent (Strands Agents + Amazing Hand)

The Strands Agent creates a fresh instance per MQTT message to prevent conversation history bloat. Each invocation sends the sentence to Nova 2 Lite, which selects the fingerspell tool. The tool maps each letter to a set of 8 servo angles from a hand-tuned ASL lookup table and drives the servos via the rustypot library at 1,000,000 baud.

The threading model keeps everything responsive:

Thread Purpose
MQTT event loop AWS CRT callback thread — never blocked
Agent executor Single-threaded pool for LLM reasoning
Publish executor Non-blocking MQTT state publishing
Video capture Daemon thread for OpenCV frame capture

What I Learned

1. Nova 2 Sonic is remarkably flexible. I expected a conversational model to resist being used as a "relay pipe," but with forced tool use and the right system prompt, it reliably transcribes, cleans, and forwards every utterance without hallucinating or adding commentary.

2. Fresh agents beat persistent ones for high-throughput edge workloads. My first approach maintained a single Strands Agent instance across all MQTT messages. Token counts grew unboundedly, latency increased, and the LLM started referencing prior commands in unexpected ways. Creating a fresh agent per message solved all three problems and made token usage predictable.

3. Hardware has no mercy on timing. Software retries are cheap; sending servo commands too fast over a serial bus causes missed commands and erratic finger movements. I learned to insert precise delays — 0.2ms between speed writes, 5ms after goal positions — to give the Feetech protocol time to process.

4. The Amazing Hand only has 4 fingers. Several ASL letters (I, J, Y) require a pinky. I had to approximate these letters using the ring finger instead, which meant hand-tuning 26 sets of servo angles to look correct with one fewer finger than the standard ASL alphabet assumes.

5. Blending software and physical hardware is a whole different world. As someone who's spent their career in software, controlling physical actuators from cloud-delivered AI commands was humbling. This project is only the tip of the iceberg — I'd love to someday design my own robot, and now I have a much deeper appreciation for just how much there is to learn. Mechanical design, electronics, kinematics, real-time control — it's a vast field, and this project gave me my first real taste of it.

Challenges I Faced

Challenge 1: Making an LLM Not Be an LLM

Nova 2 Sonic is designed to be conversational. My system needed it to be the opposite — a silent, obedient text relay. Getting it to never add commentary, never refuse a message, and always invoke the send_text tool required both prompt engineering ("You are a dumb speech-to-text relay pipe") and API-level enforcement (toolChoice: { any: {} }). Without either one alone, the model would occasionally try to be helpful and respond conversationally instead of relaying.

Challenge 2: Bridging Two Worlds of Infrastructure

The AppSync API and DynamoDB table are managed by Amplify Gen 2 in one repository, but the IoT Rules Engine and Lambda live in a separate CDK stack in another repository. Connecting them required the CI/CD pipeline to dynamically discover the AppSync API URL, API key, and API ID at deploy time by querying SSM Parameter Store and CloudFormation stack outputs. This was more plumbing than expected but keeps both repositories independently deployable.

Challenge 3: Preventing Agent Runaway Loops

During testing, the Strands Agent occasionally entered loops — calling fingerspell, then deciding the text needed modification, then calling it again. I implemented a custom MaxToolCallsHook that counts tool invocations per agent run and hard-cancels any call beyond 3. This simple guardrail eliminated the problem entirely.

Challenge 4: Real-Time 3D Synchronization

The Three.js hand in the browser needs to mirror the physical hand's servo angles with minimal delay. The pipeline — servo movement → MQTT publish → IoT Rule → Lambda → AppSync mutation → GraphQL subscription → React state update → Three.js render — adds latency at every hop. Keeping each step non-blocking (dedicated publish threads, QoS 1 MQTT, efficient Lambda) brings the total feedback loop to approximately 20ms from IoT publish to UI update, which feels instantaneous to the user.

Challenge 5: From Web Developer to Hardware Tinkerer

This was my first project where an AI agent directly reasoned about and controlled physical actuators in real-time. The learning curve was steep: understanding serial bus protocols, servo calibration offsets, baud rates, the mechanical constraints of a 4-finger hand. It taught me to be more careful, more deliberate, and more respectful of the physical world.

Tech Stack Summary

Layer Technology
Frontend React 19, Vite 7, TypeScript, Three.js
Voice AI Amazon Nova 2 Sonic (Bedrock bidirectional streaming)
Auth Amazon Cognito (Amplify Gen 2)
Messaging AWS IoT Core (MQTT)
Infrastructure AWS CDK, Lambda, AppSync, DynamoDB
Edge AI Strands Agents + Amazon Nova 2 Lite (Bedrock)
Hardware NVIDIA Jetson Thor, Pollen Robotics Amazing Hand
Servos 8× Feetech SCS0009 via rustypot at 1M baud
Video OpenCV capture, imageio-ffmpeg H.264, Amazon S3

Built With

Share this project:

Updates

Submission history