My Kopi Kaki

My Kopi Kaki
Homepage
Calendar
Companion

Inspiration

Imagine you're elderly, with shaking limbs and an empty home. You want to do something, anything but where do you go? Who do you find? Constantly asking for help feels too much sometimes for a person of your age. One-size-fits-all schemes and initiatives may be helpful for some, but not for yourself. Bombarded with irrelevant activities that don't feel personal, you start showing up less and less. Current technologies addressing this might not appeal to elderlies who are not very tech savvy, overwhelmed by generic activities not tailored to them.

Introducing My Kopi Kaki, an app that you speak to as a friend, and it creates personalised responses to you. It focuses on reducing isolation among elderlies by encouraging interaction with others, either by simply greeting neighbours, or inviting seniors to go for community events they are actually interested in or makan sessions

What it does

The application utilizes conversational AI to allow seniors to directly converse with the app. It uses Whisper's Singlish Speech-to-Text (STT) model to convert the speech input into words, then uses Open AI to interpret and simplify this input, which is often in Singlish, into structured language our system can understand.

The app then prompts seniors by speech through Google's Text-to-Speech (TTS) model, asking questions related to meal, events, health. The app then provides tailored responses. For example, if a senior shares that they had an unhealthy breakfast (determined by the AI model), the AI model will generate a ranking with explanation and suggests healthier options for lunch with options that can be easily found in their neighbourhood. The output is then played back to the senior as a speech, just like a conversation.

The app also retains the information on events the senior has showed interest in previously and suggests related activities/events using AI focusing on the type (sports, crafts, exercise, etc.). This reduces the need for seniors to scroll through events unrelated to them.

The last feature our app has is a mental health prompt which acts like a friendly chat (powered by AI), which can toggle between 2 modes: referral and companion. Referral gives us the permission to provide helpful resources to offer professional help. Companion allows seniors to speak freely without judgement, acting like a listening ear. This allows seniors to get emotional support in a way that feels natural and comfortable by allowing them choice to choose how they want to engage. This feature is not meant to replace a therapist, but merely plays a therapeutic role and will offer professional help if warning signs are spotted.

How we built it

Python: Used to build the logic of the app handling user flow, user input, and conversational structure. It also allows the usage of the OpenAI model for engineering prompt and to simplify the users input into readable language for our system. Whisper's Singlish (STT) & GTTS: To facilitate the translation of input from speech to text, then the output from text to speech. This makes it much more convenient for elderlies to use our app, especially those who are not used to technological devices.

Challenges we ran into

Unpreparedness: Time spent on setting up the repo was not ideal. We could've done it before the hackathon started. Difficulty narrowing to a solution: We spend a considerable amount of time brainstorming due to the wide range of issues faced by seniors, making it difficult to focus on a specific solution.

Accomplishments that we're proud of

We managed to stay up late and persevered to finish the project. It's also the first hackathon for 3 of our members.

What we learned

We learned how to implement STT and TTS and how frustrating it was to use FFMPEG. We also learned to use OpenAI for engineering prompt in Python.

What's next for My Kopi Kaki

We aim to collaborate with government agencies and community partners to provide anonymized insights from the app. This allows them to host events/interest groups that are more tailored to the residents of each town.