Journify

From Words to Worlds: Journify Your Journey

The strength of the parent-child relationship is critical to a child’s social and emotional development. And yet, today there is a breakdown in family communication. Over 150 million parents are anxious the bonds with their children are deteriorating. Today, 2 in 5 kids lack an emotional bond with their parents, and 50% of those kids suffer mental health challenges as a result.

Parents are spending upwards of $3,000 annually on the tech-forward tools they can engage together with their kids - multiplayer video games, AR/VR devices, creative apps, subscription services. However, none of these specifically target the parent-child relationship or incorporate social-backed Attachment-Based Family Therapy (ABFT) practices or the RULER framework, an evidence-based approach to building emotional intelligence and secure attachment, developed by the Yale Center for Emotional Intelligence. This is Journify's social impact. This is its value add and competitive advantage.

What it does

Our approach to spatial journaling incorporates GenAI and WebXR, meaning it uses speech-to-3D, is device agnostic, and multiplayer. It is a collaborative, gamified journaling experience. Not only do parents and children engage in meaningful conversation, but they co-create 3D assets that can be saved (and revisited) and visually represent their memories in XR.

Each journal prompt is based on the latest social-emotional research. Each cloud represents a month and has a new page for a journal entry hanging from it. Faded cubes represent past years. Once a new jounral page is pulled down, users can record their response to the prompt and a 3D asset is produced.

This is for children ages 8-12.

How we built it

Firstly, have the Unity + Needle Engine component which is responsible for the WebXR and all interactivity as well as realtime 3D rendering. Secondly, our Gen AI pipeline from Speech to Text to Image to 3D model is using a multitude of services and APIs. We experimented with different services and models.

Challenges we ran into

Use of the Needle plugin in combination with a microphone recorder. The installation of multiple plugins that require specific versions to interact with each other on the Needle server. Since we are using WebXR in a multiuser context, we had to ensure network replication and hosting of the generated files, which, for a short hackaton, was very challenging but also highly rewarding. Generative AI poses it´s own challenges when trying to create very specific outputs which we also encountered a lot. Latency between microphone recording and GenAI prompt. Workaround for wait between speech prompt and the 3D object appearance.

Accomplishments that we're proud of

It all works. Seriously, this is BIG. Combining multiple large AI models in a pipeline such that they take a speech input, transcribe it, generate a 2D image and eventually a 3D model.

What we learned

GenAI in WebXR is possible! It will work if you try hard enough.

What's next for Journify

Hopefully, pitching the business idea tomorrow in front of a large audience. We have a business model, go-to-market strategy, and a five year plan for R&D, including feature additions and market expansion.

Built With

• genai
• javascript
• needle
• openai
• spatialcomputing
• stability.ai
• stablediffusion
• unity
• webxr