Voice2Motion

Inspiration

Everyday therapeutic work includes several challenges that can affect therapeutic success. One of them is the gap between treatment sessions, where the patient has to remember how to perform home exercises correctly and where no therapeutic adaptation is possible when something goes wrong. In addition, the time a therapist can spend on creating personalized digital guidance is limited.

My combined experience in software engineering and osteopathy gave me the idea to create simple processes that support therapists and patients between treatment sessions. Voice is an important factor that can support therapeutic continuity.

What it does

A fast way for the therapist to create a personalized exercise is to record the complete exercise in their own words and transcribe it. GPT-5.6 creates a structured exercise from the transcript, improves the exercise description, and generates questions about inconsistencies that need to be resolved.

The therapist reviews the generated exercise, changes the necessary parts, and approves it. The exercise can also be translated when needed. Currently, English and German exercises are supported. Audio guidance can be generated with text-to-speech for both languages. After a final consistency audit with GPT-5.6 and a review by the therapist, the exercise can be published for the patient.

The patient has a simple exercise view where they can read the exercise or run the guided exercise using text only, audio only, or both text and audio. Patients can send feedback to the therapist about how they felt and whether they experienced any problems. They can also record spoken feedback and have it transcribed.

The submitted feedback is shown to the therapist for review. GPT-5.6 can provide an analysis with questions to resolve and suggestions for possible changes. Updating the exercise and notifying the patient are the final steps for the therapist. Because no push notification service was implemented, the notification is currently shown as a change in the patient view.

It is important to note that AI only supports the workflow. The therapist remains responsible for reviewing and approving the exercise, translations, and suggested adaptations.

How we built it

I created the application with Next.js, React, TypeScript, and Tailwind CSS. For persistent data storage and file storage, I used Supabase, which is based on PostgreSQL. Zod was used to validate the structured JSON exchanged between the application, the database, and the OpenAI API.

The first implementation and the creation of the GPT-5.6 features were done with Codex. Cursor was used for the further development and refinement of the application. I used GPT-5.6 High for product reasoning, UX structuring, domain modelling, and the preparation of briefs for Codex and Cursor.

For speech-to-text, I used gpt-4o-mini-transcribe, and for text-to-speech, I used gpt-4o-mini-tts. GPT-5.6 was used for translations, data extraction, the creation of structured exercises, the analysis of patient feedback with suggestions for changes, and the consistency audit of exercises.

Challenges we ran into

There were several challenges to solve. I found the Build Week challenge rather late and started my project on July 18 at 08:00 CEST. The first challenge was to estimate what could realistically be done in four days.

This was my first hackathon experience, so I had no previous experience to compare it with. I already had some ideas from a digital health project of mine and decided to build a scalable vertical slice, but I did not yet have a concrete UX in mind.

I started with the first feature implementation and a basic UX. Through the working prototype, I gradually developed a clearer picture of how the final workflow and UX should look.

The development was faster than expected, but the use of Codex credits was also higher than expected. I therefore had to divide the work between Codex and Cursor. I used Cursor for more basic implementation tasks and Codex mainly for AI-related features and larger changes.

Another challenge was protecting enough time for the demo video and the documentation. Creating the application was only one part of the submission, and presenting the most important parts clearly in a video of less than three minutes required additional planning and editing.

Accomplishments that we're proud of

I am proud that my broader digital health project could generate a meaningful MVP for this challenge and that the workflow proved to be technically feasible.

I found several useful areas where AI can support the therapeutic workflow. The prototype also showed that my approach to extending therapeutic continuity between treatment sessions can be represented in a practical application.

Around 25 hours of AI-supported development resulted in a rather complex vertical slice with a feedback system and a complete closed loop: from the therapist recording an exercise, to patient guidance, to patient feedback, and finally to therapist adaptation.

I was surprised by how effective this style of development can be, especially when product reasoning, implementation, testing, and refinement are closely connected.

What we learned

It is a helpful approach to use AI to create a fast, interactive design probe. In this way, I could structure my thoughts about the UX and develop a clearer direction for the further design.

It will take several projects to estimate reliably how fast development can become with full AI support. I used GPT-5.6 for planning the whole project, and it helped that I already had a solid knowledge base for AI use from another project.

The drafts for the AI implementation briefs were also a great help. Creating a consistent and precise brief is time-consuming without AI support. Clear acceptance criteria are essential and help avoid extremely long prompts.

Another important lesson was that the right AI model should be selected for the right task. This is demonstrated by the separate use of speech-to-text, text-to-speech, and GPT-5.6 for reasoning and structured content.

What's next for Voice2Motion

One of the main next steps is to refine the processes and UX to create a clearer and simpler interface. This is an important requirement for helping therapists reach their goals with the minimum number of steps.

Another step toward testing the prototype with real therapists is to add authentication and review the requirements for production use in a healthcare context.

The current functionality provides a baseline that can later be extended to other exercise representations, such as images, videos, and animations. Support for additional languages is also a natural next step.

For the therapeutic context, the next step is to evaluate whether the process is time-efficient for the therapist and whether the guidance is clear for the patient.

Built With

• codex
• cursor
• eslint
• gpt-4o-mini-transcribe
• gpt-4o-mini-tts
• gpt-5.6
• json
• next.js
• npm
• openai-javascript-sdk
• react
• rest
• supabase
• tailwind-css
• typescript
• zod