Orbcura

Language Change - Language Change Screen for changing the language of the app
Home - Home Screen with tap-able options for Smart Camera and UPI Payments.
QR Scan - QR Scan screen for scanning the QR Code for UPI transaction.
UPI Payment Confirmation - Payment Confirmation Screen with amount input and confirmation prompt.
UPI PIN Entry - UPI PIN Entry Screen for digital payment authentication.
Smart Cam - Smart Cam interface that describes what is in front of us.

Inspiration

The world today is heavily reliant on technology for daily tasks, communication, and social interaction. However, for visually impaired individuals, navigating even the most basic smartphone functions can be challenging. Traditional screen readers like TalkBack and VoiceOver are limited in their ability to offer an inclusive experience across all apps, often leaving users feeling restricted and dependent on external help. Orbcura was born out of a desire to redefine mobile accessibility by developing a fully-integrated, intelligent app that goes beyond conventional solutions. Our goal was to empower visually impaired users with tools that make their smartphones as easy to use as anyone else's, enabling a level of independence and connectivity that everyone deserves.

What It Does

Orbcura is an AI-driven mobile app designed specifically to enhance accessibility for visually impaired users. Key features include:

4-Corner Navigation UI: Designed for intuitive navigation, this unique touch-based system allows users to access app functions through corner-based gestures, as a user can always feel and touch the edge of a mobile screen.
SmartCam: With AI-powered image recognition, SmartCam provides real-time spoken descriptions of images captured through the device’s camera. This feature enables users to better understand their surroundings or get context on objects around them.
Accessible UPI Payments: Orbcura simplifies digital transactions for visually impaired users. By offering an accessible PIN interface, it removes barriers in online payments, making it easier and safer to complete UPI transactions independently.
Extended Accessibility: Orbcura enhances accessibility in all apps, even those without built-in support. With voice navigation and screen-reader functionality, visually impaired users can now access a wider range of apps with greater ease.

Tech Used

Building Orbcura required an advanced stack of technology to deliver seamless functionality across platforms:

AI: Leveraging the Google Gemini API for the SmartCam feature, Orbcura can recognize objects and provide accurate descriptions, transforming visual data into meaningful audio feedback.
Flutter: We used Flutter for quick and easy development, which has ready-made support for various technologies like Gemini, Firebase, Google Text-to-Speech, Google Speech-to-Text
Firebase: Firebase handles our real-time data needs, including secure authentication and user data storage, essential for maintaining user preferences and transaction history.
UPI Integration: We designed an accessible UI for UPI payments, allowing users to make transactions with no visual input. Currently, we are approaching it with USSD interface.
Google Text-to-Speech (TTS): It reads out descriptions and notifications, providing audio feedback to visually impaired users as they navigate the app.
Google Speech-to-Text (STT): It captures spoken commands, allowing users to navigate and interact with the app hands-free. Also, the user can enter the amount to be paid via speech.

How We Built It

Our first step was creating the 4-Corner Navigation UI, allowing users to navigate the app through familiar touch gestures without needing visual confirmation. We have reused the same pattern across screens to make it consistent. Then, we implemented the AI-driven SmartCam, integrating Google’s image recognition tools to deliver accurate, spoken descriptions. By incorporating Firebase and Flutter, we can use the authentication provided by Flutter, and store any useful data in Firestore. The accessible UPI reused the same 4-Corner Navigation UI, but with an extra center button. These 5 buttons collectively were mapped to digits. Then, the volume up and down buttons allow changing the these digits to next 5 digits, or removing the recently entered digit.

Challenges We Ran Into

Developing Orbcura presented several unique challenges. Image recognition required a balance between speed and accuracy, especially in low-light environments. Ensuring compatibility and seamless performance across screen sizes added complexity. Crafting a UPI interface that visually impaired users could navigate with ease required multiple rounds of testing and feedback. Finally, maintaining accessibility for various gestures within the 4-Corner Navigation UI was crucial and challenging to execute without compromising ease of use.

Accomplishments We’re Proud Of

Orbcura stands as a testament to our commitment to accessibility innovation. We are particularly proud of:

The 4-Corner Navigation UI: This feature provides an accessible, user-friendly alternative to traditional app navigation methods, making it simpler for visually impaired users to perform tasks with confidence.
SmartCam’s Accuracy: The image description feature has been tested and optimized to provide accurate object identification, enabling visually impaired users to "see" through descriptive audio feedback.
Seamless Payment Experience: Our accessible UPI payment flow eliminates common barriers, allowing users to complete transactions independently without fear of errors.

What We Learned

Building Orbcura taught us the importance of user-centered design and the value of small, thoughtful details in accessibility. Every feature, from SmartCam to the 4-Corner Navigation, was developed with real user scenarios in mind. Working on Orbcura challenged our team to think beyond traditional UX and embrace the mindset of users with varying levels of visual impairment.

What’s Next for Orbcura

We plan to expand Orbcura’s AI capabilities, introducing more advanced, customizable commands for everyday tasks. Future updates will enhance the UPI interface further and expand SmartCam to recognize more context visually.
Potentially partnering with FinTech companies may also be beneficial to actually have a internet-based UPI as we generally use instead of a USSD approach.
Adding real-time descriptions as camera moves, instead of first capturing and describing individual images is also a future goal. This would be achieved via opencv and related technologies.