On-device AI therapist

Inspiration

I am always stressed when my to-do list piles up. When I am stressed, I eat snacks and gain weight, which makes me even more stressed. When I messaged my friends, they were annoyed because they had heard it many times. Therefore, I need an on-device therapist who is willing to listen to my problem and provide advice. On-device solution is the best because I don't need my problem to be data of some training sets of a cloud model.

What it does

The user will start a therapy session and engage in a conversation to discuss his problem. When the conversation ends, AI summarizes the feedback it provides.

How I built it

I use the Angular framework to build a web application for on-device therapy.

Challenges I ran into

Implementing multimodal input for the Prompt API is difficult because I don't practice it often. Due to the time constraint, I did not have enough time to capture audio and use the Prompt API to transcribe the audio to text.

Moreover, I have not had time to implement photo capture to capture my facial expression so that the Prompt API can analyze my mood at that moment.

Accomplishments that I'm proud of

What I learned

• How to include @types/dom-chromium-ai in tsconfig.json in order to use the typing in the Angular project.
• Use the proofreader API to correct the mistakes in the user inputs before prompting the Prompt API to generate the response.

What's next for On-device AI therapist

• Support multiple translations where users can enter text in non-English. Then, the Translation API translate the text to English before using the Prompt API to generate a response
• Use Firebase hybrid on-device inference SDK to support both on-device AI and cloud-based AI. When the device's browser has the appropriate APIs to perform the AI tasks, they are used. Otherwise, the tasks are delegated to Gemini API or Vertex AI Gemini API to handle them.
• Use the Multimodal Input of the Prompt API to analyse the photo of the users to identify their mood. Then, Gemini Nano can ask the users why they feel a certain way to open the dialogue.
• Register origin trial so it can be used in Chrome stable.

Built With

• angular.js
• prompt-api
• proofreader
• summarization-api
• tailwind
• web-speech-api