REMedy

Inspiration

I was inspired to create this app when I started to realize how hard it was to search the web without getting distracted by social media and other stuff. Additionally, I noticed a lot of limitations to the internet that couldn't be solved with a simple Google search. As a result, I wanted a fun, friendly anime character that could speak to me in the most anime way possible.

What it does

The app is a desktop assistant that is supposed to be like a way for users to interact with a fun anime character and solve problems as well as engage in mentally soothing activities like playing games or having fun. Named after the character from the anime "Zero", REMedy is your new remedy for being lonely or forgetful.

Within the non-anime version, I had implemented a couple routes for a few simple tasks like setting reminders, making to-do lists, and etc. However, this list is way too tiny and isn't comprehensive enough. Plus, while Whisper API was able to do the job, it just didn't feel personable enough and the functionality of the app was very limited.

Hence, I implemented a new version of the app that relies on ChatGPT to make a personable AI assistant.

How we built it

I used the OpenAI API for API calls to the AI. This primarily was used for running the backend of the application. I used the Twilio API for API calls to phone a number. If a user asked to phone a number, they would have the ability to call or text a number once they made a call from a Virtual Privacy Network Phone Number issued by Twilio. I embedded the system on Tkinter.py for Desktop applications. This was a smarter choice as often the web is a very dangerous place that API keys can get leaked. Additionally, it makes the AI more personal. I used VoiceVox to make the AI not sound so robotic and monotonous. I used pyttsx3 to convert the openai calls and etcetera to transcribe audio at first but issues with that included transcription errors and more, then I simply just had the user give in input through a command line terminal at the bottom of the page, but then I went back to using a text-to-speech using Librosa.

Challenges we ran into

VoiceVox API and software is written in Japanese. I did not pay attention in Japanese class. Hence, I had to spend two hours relearning Japanese in order to start understanding the docs. Additionally, since it is Japanese, the speech output does not go to handy with English words. Hence, I had to improvise my project a bit and make it a weeaboo desktop assistant rather than an anime desktop assistant that speaks English. I was in the middle of making an AR model using Echo3D, but after 3 am, I simply just decided to call it and move on because my art skills just weren't good enough. One of many challenges I encountered was constantly running out of credit on my OpenAI account for some reason, so I simply had to continuously increase my credit limit because I tried to be a cheapskate and save money. Additionally, the Twilio calls on my webapp continuously bugged out for some reason that I couldn't figure out at first, but after a little while trying and reading into Twilio docs, I was able to figure out that Twilio has a specific way that you're supposed to format virtual privacy phones. Hence, that's why it wasn't working. Ideally, I would've been able to make the frontend more impressive, but considering I was able to improve the backend significantly, I decided to call it there.

Accomplishments that we're proud of

I did not know how to use the GPT API before this. I did not know how to create AR/VR models before this. I didn't even know how to code in Python before this. I'm actually surprised I was able to make some progress on the issue.

What we learned

I learned how to make calls to the GPT API, which was definitely a step up from what I didn't know before. Additionally, I learned the complexity behind AR/VR development and how it typically isn't something that is done within the day or even in Python(Typically Unity C# is used for this application). Finally, I learned how to manager my time effectively because despite the all-nighters that I pull, it is definitely worth it in the end as I progress along my hacker journey and learn more about app development with AI and NLP.

What's next for REMedy

Clearly, the frontend wasn't as impressive as I had planned, so ideally, I would've made an app that integrated VR, but given that I had twenty four hours and no Meta Quest, I'm happy with the work that I did.

Additionally, while the VoiceVox API was the only way to bring the character to life, it does have a language barrier limitation, so I would've ideally used a translation API for subtitles to resolve that issue.

If the product is ever completed, I hope to donate any revenue generated by the products to charity. Initially, this project was a passion project, so I intend to keep it that way. I've always loved binge watching anime and going to the gym, so the past couple days have been really interesting as I got to do both of those activities a lot more.

Full video to come out soon for "REMedy"