Project Into the Sky

Inspiration

Almost everyone remembers folding their first paper airplane: creasing it a little crooked, throwing it across a classroom or down a hallway, and watching to see how far it went. It's a small moment, but it's one of the first things kids make and launch completely on their own. For many kids who are blind, have low vision, or have limited use of their hands, that moment looks different. Instructions are visual, folding takes precise fingers, and you can't watch where your plane lands. Usually an adult ends up folding it for them, and the kid misses the part that matters most: doing it themselves. We wanted to change that. Our hackathon's theme was "Into the Skies," and we took it two ways. Literally, the plane goes into the sky. Figuratively, kids with disabilities hear "reach for the sky" all the time, usually as encouragement from someone else. We wanted to build something where they actually get there on their own: folding their own plane, throwing it, and hearing how it flew. In Project Into the Sky, the kid isn't the inspiration. The kid is the pilot.

What it does

Project Into the Sky is a web app that helps kids who are blind, have low vision, or have trouble using their hands fold and fly their own paper airplane, without someone doing it for them. The whole app works with one key: the Space bar. A friendly, natural voice from ElevenLabs reads every step and every choice out loud, like "Press Space 1 time to begin," so kids never need to see the screen. Before starting, a few quick yes/no questions set the app up for how each kid folds, whether they're blind, fold with one hand, or have shaky hands. While they fold, the camera watches their hands and guides them with short sounds and quick hints. After they throw, the app analyzes the flight. Then the kid hears their flight as a song, with the sound rising and falling as the plane did, and a narrator tells them how it went. As they get better, kids climb levels from Runway all the way to Orbit, so they can hear themselves growing from one flight to the next.

How we built it

We built Project Into the Sky as a web app in JavaScript, so it runs in a regular browser on a laptop. The voice comes from ElevenLabs. We picked one friendly narrator, Jessica, on the Eleven v4 model, and pre-recorded about 330 clips so the voice starts instantly and even works offline. To stay within our free credits, we built menus by stitching short recorded pieces together, like "Press Space" + "2 times to" + "say no." Flight results with real numbers are spoken live through our own small server using v4 Turbo, and if the internet drops, the browser's built-in voice takes over so the app is never silent. For the camera, we used MediaPipe to track 21 points on each hand right on the device, which powers the fold coach. For the throw, we recorded our own test flights and trained a small model on them that finds the plane, its release, and its landing, then draws the flight path. The Flight Song is made with the browser's audio tools, turning the plane's height and direction into sound. We finally developed a machine learning model that is able to track the flight path of thrown paper airplanes and collect data to share with the user (also provides a unique soundtrack of the paper airplane trajectory).

Challenges we ran into

Our biggest challenge was making sure the voice actually spoke every line. In our tests, the app said every menu was read out loud, but in real Chrome some lines were silently skipped and there were long pauses between sentences. It turned out our log was recording lines when they were sent, not when they were heard. We rebuilt how the audio plays, loaded all the voice clips ahead of time, and started testing in a real Chrome window instead of a fake one. Designing a one-key interface was harder than we expected. Kids have to count their presses, so we kept every menu to five choices or fewer, made every option a full sentence, and gave kids with shaky hands or a weak grip more time between presses. We also had to balance talking: too many hints while folding felt overwhelming, so we switched to short sounds for most feedback. The camera was tough too. Our first tracker looked for brightly colored paper and printed dots, so it couldn't find a plain white plane, especially against white walls. We recorded our own folding and throwing videos and trained a model on real footage. And with only a small amount of free ElevenLabs credits, we had to stitch short recorded pieces together instead of recording every sentence.

Accomplishments that we're proud of

We're proud that a kid can use our whole app with their eyes closed. With just the Space bar and a friendly voice, they can set up, fold their plane, throw it, and hear how it flew, all on their own. Getting every step spoken clearly and on time was hard, and seeing it finally work was our biggest win. We're also proud of the Flight Song and our flight analysis. We trained our own model to find the plane in our throw videos, and after a throw, kids can hear their plane climb, dip, and drift, so a blind kid can actually experience their own flight instead of just being told about it. Most of all, we're proud that kids are in charge. They pick their goals, fold their own plane, and level up from Runway to Orbit by getting better. We also kept it private, so the camera never saves or uploads any video.

What we learned

We learned that accessibility is about the details. Small things, like how long to wait between Space presses, whether every option is read out loud, or how much the app talks while you're folding, make the difference between something a kid can actually use and something that just looks good in a demo. We also learned to test in the real world. Our app passed hundreds of automated tests, but some voice lines were still silent in real Chrome, and our camera couldn't find a plain white plane until we tested with our own videos. Testing with real tools and real footage showed us problems we never would have found otherwise. Finally, we learned that a natural voice really matters. When the voice is the whole interface, it has to sound friendly and clear. ElevenLabs gave our app a voice kids would actually want to listen to.

What's next for Project Into the Sky

First, we want to test it with the kids it's built for. We plan to reach out to schools for the blind, special education teachers, and occupational therapists so real kids can try it and tell us what works and what doesn't. We'll also improve our flight model with throws from more rooms, make the fold camera work with any plain paper, and record every line in our ElevenLabs voice. Next, we want to bring it to more devices. We're starting an iPhone and Apple Watch version, so kids can feel their flight as vibrations on their wrist, which would also open the app up to kids who are deaf / blind. We'd also like to add more plane designs, more languages, and a Captain's Log where kids tell the story of their flights. Long term, we see this as a possible startup. Paper airplanes are just the beginning: the same one-key voice and camera coaching could guide kids through origami, crafts, and science experiments. We'd love to offer it to schools and therapy programs, so every kid gets the chance to build something and launch it on their own.