TouchlessOS for Education

Platform for gesture-controlled experiences for neurodiverse and neurotypical students

Comment

Inspiration

Sirica Therapeutics runs cognitive sensorimotor therapy for autistic kids in Burlingame. Their setup is tablets tethered to dedicated hardware trainers: expensive, fiddly, one child at a time. They saw TouchlessOS and asked a simple question. Can you run the whole thing on one big screen, with just a camera, for less?

That question was our founding thesis pointed somewhere we hadn't pointed it. We built TouchlessOS so any display plus a commodity camera becomes an interactive surface. In a bar that's a photo booth. In a therapy center it's a motor-function game a kid controls with their whole body. In a classroom it's a lesson thirty students can reach without anyone touching anything.

But the thing that actually redirected the company came later, and from the customer. Our first paying school didn't want our content library. They wanted to make their own, for their own students, and then for the autistic kids at Sirica. That is when it stopped being a product and started being a platform.

Because the real shortage in accessible educational software has never been ideas. Teachers and therapists have more ideas than anyone. The shortage is that shipping one has always required a computer vision engineer, and there are perhaps a few thousand people on earth who can build a gesture interface that works on a child with atypical motor control. There are millions who know exactly what that child needs.

What it does

TouchlessOS for Education has two halves.

The runtime. Any TV becomes a gesture-controlled learning surface. An iPad running our app sits under the display and does the seeing, the TV shows the experience. No controller, no touch, no headset, no wearable, nothing for a child to hold or drop or share. Larger installs run the same stack on a Mac Mini. What's live today includes motor-function games built around therapy protocols (reaching, crossing midline, bilateral coordination, tracked as play rather than exercise), a touchless letterboard for air-spelled communication with context-aware word prediction, a Creative Painter where a child's air-drawn sketch is transformed into a finished portrait by Gemini's image model, and an AI guide built on the Gemini Live API that watches the screen and coaches the student through whatever they're doing.

Staxel Studio. This is where Gemini does the heavy lifting. A teacher, a therapist, or a parent describes the experience they want in plain language. Gemini generates it against our gesture-native component library, and it runs on any TouchlessOS screen immediately. A spelling game tuned to one class's vocabulary list. A reaching exercise calibrated to one child's range of motion. A lesson in a language we've never shipped content in. No code, no computer vision, no waiting on our roadmap.

The person who knows what a specific child needs is almost never the person who can build software. Studio removes that gap. Every gesture experience we've built ourselves is a proof that the runtime works. Studio is the argument that it scales.

All gesture computation happens on the device. Camera frames are discarded after hand detection, so no child video is recorded, stored, or transmitted. That is what makes this deployable in a classroom or a clinic at all.

How we built it

The deployment unit is deliberately boring. An iPad runs our native app, connected to a TV by cable or AirPlay. The app wraps our web-based experiences and adds a gesture layer underneath them, so anything built for the browser becomes touchless without being rewritten. That single architectural decision is why Studio is possible: Gemini generates a web experience, and our layer makes it controllable by a hand across the room. Where an install needs more headroom or multi-person tracking, the same stack runs on a Mac Mini.

Hand tracking runs on-device through Google's MediaPipe, feeding a 60fps AVFoundation capture pipeline that drives the external display. Mac Mini deployments add RTMO for full-body pose with a ByteTrack-based tracker for multi-person scenes.

Gemini is in production in three places, all of them user-facing:

  • Studio generation. Gemini turns a plain-language description into a working experience, constrained to a component library where every interactive element is already sized, spaced, and dwell-timed for gesture control at two to three meters.
  • Creative Painter. The Gemini image model transforms a child's air-drawn sketch into a rendered portrait, on every drawing a student finishes.
  • The AI guide. The Gemini Live API with screen context, so the avatar responds to the actual state of the experience rather than reading from a script.

Studio's generation calls route through Vertex AI on Google Cloud, which also gives us the request logging and quota controls we need before letting non-engineers generate content that runs in front of children.

On pre-existing work. Staxel's core gesture layer and the iPad app shell predate this hackathon and are the foundation we built on. What we created inside the window is TouchlessOS for Education: Staxel Studio and its generation pipeline, the therapy and classroom experiences, the Gemini-powered painter and guide, the deployment model for schools and clinics, and the business around it. We'd rather state that plainly than have a judge find it.

Challenges we ran into

Generated interfaces are touch-shaped by default. Ask any model for a web experience and you get buttons sized for a fingertip on a phone. At two meters, controlled by a hand in the air, those targets are unhittable. We couldn't solve this in the prompt, and we stopped trying. Instead we built a constrained component library where target size, spacing, dwell time, and error tolerance are properties of the components themselves, and Gemini composes from that vocabulary rather than generating raw markup. The model decides what the experience is. The library guarantees a child can actually control it.

Atypical motion. Standard hand tracking assumes intentional, well-formed gestures. Many of the kids we build for don't move that way, and involuntary movement reads as input. Separating intent from noise turned out to be the core problem rather than a refinement, and pushed us toward gross arm and head gestures over fine landmark tracking.

One port, three jobs. The iPad has a single USB-C port, and the deployment wants it to charge, drive an external display, and in some configurations take an external camera. All day, unattended, in a room where nobody is going to troubleshoot it.

Thermals. Continuous inference on an iPad for a full school or clinic day means heat. We built inference hibernation and battery cycling so the device throttles gracefully during idle periods instead of degrading mid-session.

Accuracy in the lower half of the screen. Persistent tracking errors near the bottom of the display, solved through coordinate mapping and layout changes rather than moving the camera. "Please reposition the iPad" is not an instruction you can give a teacher.

Mounting. Unglamorous and still not fully solved: getting an iPad positioned under a TV, camera unobstructed, in a room that was not designed for it.

Accomplishments that we're proud of

A live pilot at a Sirica Therapeutics center with autistic children, free during validation period.

Our first paying education customer: an SF school subscribed to run and develop gesture experiences with their students. It means the accessibility product and the general-education product are the same product, which is the difference between a business and a grant-funded pilot.

And that school is not only a customer. Part of our agreement is that they build experiences for the Sirica center and for autistic students. A mainstream school, paying us, producing accessibility content for a therapy center it has no commercial relationship with, using a tool where Gemini does the engineering. That is the entire thesis demonstrated once, at small scale, with real money attached: when building a touchless experience is easy enough, the people closest to the children build it, and the supply of accessible software stops depending on us.

Gemini doing something load-bearing in front of a child rather than behind a dashboard. A drawing becomes a portrait. An avatar that can see the screen guides a student through an exercise without an adult narrating it. A teacher describes a lesson and it exists.

A free public letterboard at play.staxel.ai/letter-board, running entirely in-browser on any laptop or iPad with a webcam, all computer vision local.

A privacy architecture we can defend in a room full of parents. Landmarks on device, no child video anywhere.

All of it from a team of four.

What we learned

Our customers wanted to build, not just buy. We expected to ship a content library. The first school asked to make their own instead, and we wrote it into the contract. That single conversation reorganized the roadmap and produced Studio.

Therapy and classroom deployments are technically easier than public venues and commercially harder. Controlled lighting, one user at a time, a known standing distance. The vision problems that torment us in a bar mostly evaporate. What replaces them is a longer sales cycle and a far higher bar for safety and privacy.

Accessibility technology should be sold through education budgets, not medical reimbursement. School districts and facilities budgets are slow but navigable. Reimbursement pathways have killed better-funded companies than ours.

Constraint is what makes generation usable. Giving Gemini a narrower vocabulary produced better experiences than giving it a better prompt. The component library isn't a limitation on the model, it's what makes the model's output work for a five-year-old with limited motor control.

Wrapping the web was the right call. Putting the gesture layer under a webview rather than inside each experience means every new lesson is a web build. It is why four people can serve a therapy center and a school with different content on the same stack, and it is the only reason a platform was ever possible.

The camera being passive is an argument, not just an architecture. For letter-based communication, the long-standing clinical objection concerns facilitator influence. A camera-based system with no physical contact and timestamped selection logs doesn't fall under that critique. It answers it.

What's next for TouchlessOS for Education

Open Studio to everyone. Right now it's us and one school. The goal is that any teacher or therapist can describe an experience and have it running on their classroom screen before the end of a prep period, with a shared library so an exercise built for one autistic child in Burlingame is available to every center running TouchlessOS.

Convert one therapy center from free pilot to paid, then move on their five-year thesis of a hundred therapy centers.

Expand from one school to a district deployment.

Give the AI guide memory across sessions, so it knows what a student worked on last time and where they struggled. That's the difference between a pleasant avatar and something a therapist will rely on.

Ship intent classification for atypical motion as a first-class input mode: gross arm and head gestures, with intentional movement separated from involuntary. This is the technical wedge nobody else has a commercial reason to build, and it is the thing that decides whether a child with cerebral palsy can use a screen independently.

Publish on the touchless letterboard as a method of independent communication for autistic and nonspeaking people, so the clinical audience has something to cite.

Built with

iPad · Mac Mini · Swift · AVFoundation · MediaPipe · RTMO · ByteTrack · WebRTC · Gemini API · Gemini Live API · Gemini image generation · Vertex AI · Google Cloud

Share this project:

Updates