AR rehabilitation app designed for UCSF children's hospital.

Designed to gamify the rehabilitation process, encouraging children to hit milestones while reducing the need for physical therapist oversight.

Spun off from a research project currently under the advisement of Professor Allen Yang and Kat Quigley.


Physical therapy is a repetitive and physically tasking branch of medical treatment. Often patients are required to do the same set of tasks dozens of times.

However, we were initially inspired to take action within the Oncology ward where patients undergoing cancer therapy are essentially locked in a hospital wing, prisoners of their compromised immune system. Under intensive chemotherapy, they are battling against all odds. In their weakened state, exercise within the cramped corridors of a hospital is often overlooked. However, engaging these patients would make a world of difference both to their mental wellbeing and their physical health. We hope to eventually extend this project's method of gamifying the rehabilitation process to engage with these patients.

What we learned

Swift! We came in with no knowledge of mobile app programming nor the Swift programming language. Basically everything was self taught in the whirlwind of the last 36 hours.

We also learned a lot about Storyboards and why developers can be frustrated about it. However, I think we all walked away with a decent understanding and appreciation for its ability to rapidly prototype application UI.

Under the hood

The AR application runs on Apple's ARKit and RealityKit framework. We loaded in all our models through RealityComposer and used RealityKit to further augment the functionality of the AR application.

Challenges Faced

Our biggest hurdle was learning a new programming language and the nature of mobile development. This was vastly different from the standard backend python programming we were used to. As a result through out our development we faced numerous bugs we had to design around or debug including a very persistent nil variable error originating from a faulty Storyboard sequence. (Here we understood why it could be extremely frustrating to have to deal with a GUI editor.)

We also had a fair amount of trouble accessing and augmenting our RealityComposer content through RealityKit and written lines of code. Apple does too good of a job wrapping everything into a nice package that modifying the package was a poorly documented API.

Built With

Share this project: