The inspiration comes from a perennial need for support from your friends and family who might not always be available when you feel like you want to feel their support the most. We are meeting this need by allowing friends and family to send videos to the app users, (at any point of time) who will receive a lovely video "care package" after journaling a message about their day, how they feel, or anything at all.
What it does
CarePak is an app that utilizes NLP and speech recognition technologies to match user-inputted text to video messages' transcripts. This matching reflects automatic sentimental detection and analysis that best serves the user's needs.
How we built it
We built the back-end systems using sophisticated NLP and speech recognition technologies. User inputted text was analyzed using sentiment analysis and keyword extraction, and was in turn matched with video transcripts (which were obained using Google Cloud-based speech-to-text technology) using semantic similarity. User input (text and video streams) are passed through Flask and Restful API-based server communication between backend Python script, MySQL database and frontend interface to ensure dynamic reflection of algorithm results.
Challenges we ran into
One of the major issues we ran into on the back-end was perfecting the speech recognition in the presence of noise, as well as implementing a recommendation system that takes into account the scope of ambiguity present in human language. On the front-end, we learned Swift and SQL to implement a lightweight iOS-based app that interactively gauges user input and dynamically adjusts layout based on user need. Perhaps the biggest challenge was to figure out the puzzle of connecting multiple pieces across a variety of platforms to make everything run smoothly.
Accomplishments that we're proud of
We're really proud of the way our speech recognition and NLP algorithms turned out, and we're super excited that our UI feels sleek and user-friendly. SQL connection makes the iOS app run smoothly, as backend and frontend communication occurs.
What we learned
As many of our team members are international students, we are used to the pain of not seeing families and friends. We learned an incredible amount about how to make an awesome scalable product into reality. We learned to base our actions on users' true needs and turned the vision into reality.