Beacon.AI

Presentation Slide 1
Presentation Slide 2
Presentation Slide 3
Presentation Slide 4
Presentation Slide 5
Presentation Slide 6
Presentation Slide 7
Presentation Slide 8
Presentation Slide 9
Presentation Slide 10
Architecture Diagram

Inspiration

When put into a sudden emergency situation in public most people begin to panic, forget crucial actions, and avoid helping altogether. Beacon.AI hopes to provide individuals with the tools and support necessary to take decisive action in a crisis no matter the circumstance. From unexpected fainting/seizures at a ballpark to riptides at the beach, Beacon.AI can be used to assist in any situation. During our initial research, we noticed the potential impact of AI when utilized in urgent care and health situations. We chose to specifically gear towards average citizens and emergency response as immediate preventative action is the first step to ensuring proper care. Through an AI-centered approach we’re able to provide comprehensive next steps backed by real sources/medical advice and administered through a text to speech agent. Through this multi-faceted application, we are able to empower bystanders and save lives.

What it does

Beacon.AI utilizes a number of different workflows and agents to make informed suggestions specific to the circumstance. We utilize an initial camera feed supported by an AI agent to immediately define the environment and select key points of interest in addition to potential emergency markers. From here, you can utilize the emergency mode to activate a verbal AI agent that walks you through steps to mitigation and getting necessary help. Whether you’re looking for basic resources like immediate calling of 911 or more comprehensive searches of how to administer CPR or use an epipen, the agent will walk you through the whole circumstance. Finally, when paramedics arrive you can tell Beacon and it will generate a complete log of every event that occurred during your session including initial response detection, CPR timing, emergency services arrival tracking, etc. This hopefully will aid emergency services in better understanding the event and what mitigations were used which would support eventual medical intervention.

How we built it

Beacon.AI required various webscraping actors, voice agents, and generative AI tools to provide reliable and accessible support. We start by pulling in a visual camera feed attached to a generative AI agent to discern important areas and identify the environment. From here, the application asks for additional information from the user in the form of questions about other environmental variables such as sound, status of individuals, and other key info. This is all pulled together to provide context for the situation and create an emergency profile for next step generation and scraping. We then employ a number of different agents and web-scraping tools to pull down relevant and proven medical advice given the specific context. From here there are various support pathways including notification of emergency services, verbal co-pilot support on actionable steps, and live support of user questions. Finally, we implemented a complete tracking system that maintains all relevant events, questions, and actions throughout the entire emergency process. This then generates a complete report from start to finish in addition to uploading information to relevant sources for storage and transfer.

Challenges we ran into

Throughout our project work time there were a ton of challenges that we ran into as we tried our best to make a robust implementation that used many technologies that our team had never worked with. The first major challenge was taking the problem we defined and narrowing the scope to something that has immediate and measurable impact. Something like an urgent care support tool would be powerful, but our goal was to reach as many people as possible and provide an application that would benefit large groups of people rather than as a commercial product. As such, we decided to target situations that are encountered by the average person and ended upon Beacon. Naturally we spent a lot of time reading documentation for each of these new APIs and tools, but we had the most trouble tying everything together. We knew we wanted generative AI, Voice Agents, and web scrapers for a comprehensive experience and data, but knowing where to plug in each component and with what context was very challenging. Finally, the limited timeframe made it so that we had to quickly harden our MVP and select what features to focus on immediately and what to add on when we had the time. To do so we had to build our project incrementally and insure one piece worked within the overall context before moving to another technology or addition. Despite all of these challenges, we were able to work together to build up a strong and demo-ready product.

Accomplishments that we're proud of

We’re proud of our initial organization and planning phase which supported the work that we did throughout the entire process. During previous hackathons, many of our team members had problems with properly planning out a technical implementation and meaningful project impact before building out a product. This time around, our team took the effort to carefully consider every factor supporting and inhibiting our project’s value and implementation which made the rest of the process far smoother. In the end, we combined all of our strengths and made consistent efforts towards completing the project while still having fun at the venue.

What we learned

From a technical standpoint, our team learned how to implement and utilize tons of new AI frameworks thanks to the various sponsors and mentors at Acl Hacks. We also learned how to leverage generative AI tools to speed up coding workflows especially given the time constraints and overall scope of our project. Additionally, we learned a lot about bridging initial implementation expectations with realistic user interaction and what is actually valuable to a product instead of as unnecessary technical enhancement. We hope to maintain these skills throughout the rest of our careers and continue to leverage our newfound learnings and skills.

What's next for Beacon.AI

Our current implementation works very well in a standalone circumstance assuming nothing goes terribly wrong, but that isn’t always the case for emergencies. We hope to expand our capabilities into local models for places without service or potentially specializing care and response outside of just dialing 911 to more specific channels. Additionally, we hope to scale our project to be utilized by countless users concurrently to impact more individuals and save more lives. Finally, we hope to expand the product from an accessibility standpoint through more languages, disability accessibility, and many other channels. Overall, we hope to expand the reach and reliability of our product so that it can be used by anyone and won’t fail when you need it the most.