Inspiration
My teammate and I are both first generation students whose parents had difficulties learning English after arriving to America. They were also always super busy trying to make ends meet to even enroll in English learning classes, which led to the deficit in reading and writing in English. Growing up we both had to be the all-in-one helper when it comes to anything medical or government related. As an elementary student we had to translate government papers in broken english, be the human translator in the doctors office or at our school's open house. All these past experienced combined led to us to create SDcohelp - a website specific to the local San Diego area government services such as Medi-cal in SD, CalFresh, LegalAid of SD, affordable housing in SD, etc.
What it does
SDcohelp is a specialized chatbot in all things government support in the San Diego area, it has a speech to text function that translate their mother language into text where the chatbot then analyzes and scans multiple websites to try and answer the user's question. Whilst there is a chance they might not know where to begin, we also included premade questions to help jog their brains. We also imbedded a "Help me get ready to apply" feature where users who need help filling out these government forms have a way to get these questions translated and physically audible for the user who lacked english reading literacy to understand in their mother language. Another feature that ties into the form filling feature is our "How to get to the application from the website" feature where users can either get an image from their files or screenshot to paste the website image into our website and the website would outline the areas in red where the user would have to press in order to get to the application form.
How we built it
We first brainstormed different ideas that we could personally relate to in terms of building a product for social good. We also wanted to make the scope of our project relatively narrow in order to maximize the impact on those who would benefit from this product the most - which would be those who lack English reading/writing skills in San Diego and predominantly speak Chinese (Mandarin and Cantonese), Spanish, and Tagalog as these 4 languages are some of the most frequently spoken foreign languages in San Diego. In the technical side, we used ClaudeCode as our coding agent and used BrightData's API to webscrape across multiple San Diego based websites related to Medi-cal, Calfresh, Affordable housing sites, and pro bono legal aid sites. Tying in with Gemini's API to analyze the data and get meaning from them; when users ask a question on our site, it is Gemini that is looking for the relevant site to answer the user's question and provide a solution prompt to the user. Using ElevenLab's API, we were able to create a speech to text, text to speech, and obtain voice IDs. The speech to text is so users can audibly talk to the site and have it automatically translated out into a question for the chatbot to answer so that the users don't have to physically type anything out. The text to speech is for accessibility to account for those who may not have the best vision or if they are someone who cant read in their mother language.
Challenges we ran into
Problem 1: When using the text to speech function and stopping it midway, the audio jumps all the way back to the beginning of the text. Solution 1: Creating a separate 'pause' and 'stop' button so users can leave off where they stopped if they want to pause the AI from speaking.
Problem 2: After asking the chatbot a question, there was a long pause right after which could makes users feel uncertain that their question did not go through or it was not working. Solution 2: Creating a physical 3 moving dots UI component to indicate that the chatbot is actively 'thinking' and that their question went through.
Problem 3: In the speech to text function, when we physically pause on our dialogue while in the middle of talking to the chatbot, the chatbot would assume we are done speaking and automatically submit the question. Solution 3: Remove the automatic submission feature and have a button that the user would have to physically stop to indicate that they are done speaking.
Accomplishments that we're proud of
We are proud that we learned how to integrate the APIs provided by this hackathon into our project, SDcohelp. Prior to this hackathon, we had little experience using such APIs. While working on our project and doing research, we learned what each one could do so that we were able to effectively integrate them into SDcohelp.
What we learned
Discovered what an API is and all the different types available and how to integrate it within our website! We also learned what webscraping is and the process of taking data from websites. One of us also learned how to use our laptop's terminal and found it to be cool. She learned that having your code in a local host would stop working if the terminal is stopped as it runs on the terminal. This hackathon also taught us the complexities of creating a website in a technical sense, as well as the challenges that come with it.
What's next for SDcohelp
We plan on broadening the website with more languages, so that more users would be able to make use of the website, regardless of the language they speak.
Built With
- apis
- brightdata
- claudecode
- elevenlab
- gemini
