GestureChat

Inspiration

We were inspired by the need to create more accessible tools for individuals with mobility challenges, allowing them to interact with technology more easily and independently. Many people with mobility issues struggle to use traditional input devices like keyboards and mice, which can be a major barrier to accessing the digital world. We wanted to create a solution that would help break down these barriers, giving individuals more independence and control over their interactions with technology. The inspiration also came from observing the lack of comprehensive tools that integrate multiple accessibility features into a single platform. By combining gesture recognition, voice interaction, and location services, we aimed to create an all-in-one solution that could cater to the diverse needs of people with mobility challenges. Our goal was to provide a practical and impactful tool that would empower users, improve their quality of life, and encourage other developers to prioritize accessibility in their projects.

What it does

GestureTalk enhances accessibility by providing multiple features designed to assist individuals with mobility challenges. It offers gesture-based input, allowing users to control the application using hand movements, which is particularly useful for those who may have difficulty using traditional input devices. The chatbot feature, powered by OpenAI, provides a conversational interface that allows users to communicate easily, offering both assistance and companionship. Additionally, GestureTalk includes a vocational rehabilitation center locator, integrated with Google Maps, to help users find nearby resources and support centers. The application also features language translation, making it accessible to Spanish speakers, thus breaking down language barriers and expanding its usability. By integrating these features, GestureTalk provides a comprehensive platform that addresses various needs, from communication and navigation to accessibility, all aimed at making technology more inclusive for individuals with physical limitations.

How we built it

We built it by creating a repository on GitHub to collaborate as a group and then put our repo together with VS code for development. Then each of us divided up the tasks to work on separately. We each read documentation, watched tutorials, and debugged our code. For the chatbot, we first created a Chatbox design and components using HTML and CSS, then we used javascript to code the real time chatting and integrated OpenAI using an API key to respond back to the user. For the hand detection feature, we used MediaPipe, Google's tool for hand detection using artificial intelligence. We integrated this tool ontot he website and are working on using it to work with computer functions like mouse-clicking and typing. We built the website using HTML, CSS, and made custom features that we embedded into the website pages.

Challenges we ran into

Throughout the development of GestureTalk, we encountered several significant challenges, particularly when integrating multiple technologies to create a cohesive platform. One major challenge was implementing real-time hand gesture recognition using Mediapipe. Ensuring the accuracy of gesture detection and translating these gestures into corresponding actions required extensive trial and error. The calibration process had to be fine-tuned to account for different hand shapes, sizes, and lighting conditions, which proved to be time-consuming and complex. We also faced difficulties integrating the OpenAI-powered chatbot, as it needed to provide appropriate and context-aware responses while maintaining a natural flow of conversation. This required continuous adjustments to the model's prompts to improve its interaction quality. Another challenge was optimizing the application's performance to ensure all components—gesture recognition, chatbot, Google Maps integration, and language translation—worked seamlessly together without causing lag or usability issues. Accessibility often means supporting older or less powerful devices, which added an extra layer of complexity to our development and testing processes. Despite these challenges, we persevered, learning a great deal about cross-functional integration and the importance of extensive user testing to create a product that genuinely meets user needs.

Accomplishments that we're proud of

We are proud of several key accomplishments with GestureTalk. First and foremost, successfully integrating real-time gesture recognition using Mediapipe was a significant achievement. This feature allows users with limited mobility to interact with the application in a way that feels intuitive and empowering. We are also proud of creating a chatbot powered by OpenAI that provides meaningful and helpful interactions for users, offering both practical information and emotional support. Another accomplishment is the integration of the vocational rehabilitation center locator using Google Maps, which helps users find essential resources in their area. Additionally, we managed to create a platform that is not only functional but also user-friendly and accessible to Spanish speakers through our language translation feature. Bringing all these elements together into a cohesive and reliable platform was no small feat, and we are proud of how we successfully leveraged existing resources and integrated them seamlessly into our project. Our teamwork and perseverance were crucial to overcoming challenges and making this vision a reality. We believe that our dedication to accessibility and user experience is reflected in every aspect of GestureTalk, and we hope it will inspire other developers to prioritize inclusive design. We would also like to thank Krish, whose contributions to the project were invaluable, especially in terms of integrating gesture recognition and providing key insights into user experience improvements.

What we learned

Throughout the development of GestureTalk, we learned a great deal about the complexities and importance of accessibility in technology design. One of the key lessons was the need for iterative user testing and feedback to ensure the features we developed were genuinely useful and easy to use for individuals with mobility challenges. We learned that designing for accessibility means considering a wide range of needs and abilities, which often requires creative problem-solving and adaptability. We also learned about the challenges involved in integrating existing technologies and how detailed and precise the coding process needs to be for successful implementation. Working with multiple technologies, such as Mediapipe for gesture recognition, Google Maps for location services, and OpenAI for the chatbot, taught us valuable lessons about cross-platform integration and the importance of optimizing performance to maintain a smooth user experience. We also gained a deeper understanding of the challenges faced by individuals with mobility issues and how technology can play a role in breaking down barriers. Overall, this project reinforced the importance of empathy in design and the impact that thoughtful, inclusive technology can have on people's lives.

What's next for GestureChat

Next, we plan to expand GestureTalk's capabilities to further enhance its accessibility features and overall user experience. One of our primary goals is to improve the gesture recognition system by incorporating more gestures and refining the accuracy of existing ones, making the application even more intuitive for users with different levels of mobility. We also aim to add support for more languages, ensuring that GestureTalk can reach a broader audience and serve users from diverse linguistic backgrounds. Another area of focus is refining the chatbot's capabilities to provide more personalized and context-aware responses, making interactions more meaningful and supportive. We hope to use our Mediapipe code to enable users to control the mouse pointer with hand gestures, adding another layer of accessibility for those who struggle with traditional input devices. Additionally, we plan to collaborate with rehabilitation centers and accessibility experts to gather feedback and further tailor the platform to the needs of its users. We also hope to continue working with Krish, whose insights and expertise were instrumental in our progress, to further develop innovative accessibility solutions. Our ultimate goal is to continue improving GestureTalk, making it a leading example of how technology can be used to empower individuals with mobility challenges and promote inclusivity in digital spaces.

VIDEO DEMO LINK HERE

https://www.canva.com/design/DAGUC0tU-Do/ckh71c5yEtW3gh2pd3evEw/watch?utm_content=DAGUC0tU-Do&utm_campaign=designshare&utm_medium=link&utm_source=editor