Voxsight

direct testing process for blind student and pooling user experience and product ergonomics
team work environment in voxsight product creation

Inspiration

What it does

How we built it

Challenges we ran into

Accomplishments that we're proud of

What we learned

What's next for Voxsight

Inspiration

People with visual impairments deserve the same rights and independence as everyone else. While technology has advanced, many are still bound by the limitations of a white cane and the inconsistency of public infrastructure. Even in places where guiding blocks (tactile paving) exist, they are often obstructed or poorly maintained.

The urgency for a better solution is highlighted by tragic, real-world incidents. Recently in Jakarta, a visually impaired woman fell into an open sewer because she wasn't properly assisted while disembarking from a Transjakarta bus. Such incidents—where lack of human empathy meets dangerous infrastructure—can be life-altering.

This is why we built Voxsight. In our early trials, Voxsight successfully identified Indonesian Rupiah (IDR) banknotes, which we then showcased to students at SMPLB-A & SMALB-A YPAB Surabaya. Their enthusiasm was overwhelming; seeing their joy as they regained a sense of autonomy was a turning point for us. These students are remarkably talented, often traveling abroad to perform Gamelan at international events. However, they face immense challenges in foreign countries where they may not be familiar with the currency or where the city layout lacks friendly guiding blocks.

By integrating Gemini’s real-time multimodal capabilities, we want to evolve Voxsight into a "Professional AI Companion." It isn't just about identifying objects; it's about providing instant, low-latency navigation and global currency recognition. Whether they are walking through a busy street in Jakarta or preparing for a performance overseas, Voxsight aims to be the "artificial eyes" that ensure they never have to feel lost, ignored, or unsafe again.

What it does

Voxsight is a wearable AI-powered smart glass designed to be a comprehensive life assistant for the visually impaired. By integrating Gemini’s real-time multimodal processing, Voxsight transforms visual data into actionable audio guidance.

Key features include:

Interactive Voice Conversation: Powered by Gemini Live, users can engage in natural, two-way dialogue with Voxsight. Instead of just receiving static alerts, users can ask complex questions like "What is the menu at this table?" or "Is there an empty seat nearby?" and receive immediate, conversational feedback.
Real-time Navigational Safety: The device constantly monitors the user’s path, identifying hazards such as open sewers, potholes, and obstacles blocking guiding blocks (tactile paving) in real-time.
Smart Shopping & Global Recognition: Users can shop independently as Voxsight identifies various global currencies, reads price tags, and recognizes product labels instantly through its "artificial eyes."
Daily Activity Support: From reading signs and documents to describing the surrounding environment in detail, Voxsight provides spatial awareness that helps users perform daily tasks with confidence.
Integrated Emergency SOS: In high-pressure or dangerous situations, a dedicated physical button allows users to trigger an emergency alert. Gemini will capture the visual context of the scene and send it along with the user’s location to emergency contacts.

How we built it

We built Voxsight using a combination of compact hardware and cutting-edge AI to ensure portability and high performance:

Core Hardware: The heart of Voxsight is the Raspberry Pi Zero 2W. We chose this Small Board Computer (SBC) for its incredibly small form factor and low power consumption, making it ideal for a wearable device.
Visual Input: A high-definition camera module is connected to the Raspberry Pi to act as the "eyes" of the system. This camera captures live frames and streams them for multimodal processing.
AI Integration: The visual data is processed through Gemini API. By leveraging Gemini’s multimodal capabilities, the system can understand complex spatial environments, recognize currency, and provide descriptive audio feedback in real-time.
Geolocation & Connectivity: We integrated a GPS module and utilized the onboard WiFi of the Raspberry Pi. This allows for real-time location tracking, which is accessible through a dedicated dashboard for guardians and family members.
Safety Logic: We developed a custom script to handle the Physical Emergency Button. When pressed, the system instantly captures the current GPS coordinates and sends an alert to the user's family, ensuring they can be located immediately in case of an accident.

Challenges we ran into

Building a real-time assistive device for the visually impaired presented several complex challenges, especially when balancing performance with portability:

Pivoting from Edge to Cloud (Latency vs. Accuracy): In our initial beta prototype, we attempted to process everything locally using an Orange Pi Zero 2W. Running concurrent models for currency classification and YOLO-based hole detection pushed the CPU to its limits, forcing us to downscale our video input to a mere 240x240 pixels. This severely hampered accuracy. We made the strategic decision to pivot to Gemini’s Cloud-based Multimodal reasoning, which allowed us to maintain high-resolution visual processing without the latency usually associated with low-power edge devices.
System Stability & Hardware Bottlenecks: Our previous dual-SBC setup (Orange Pi and Raspberry Pi) was prone to frequent system crashes due to thermal issues and lack of a dedicated GPU for local AI tasks. By consolidating our architecture into a single Raspberry Pi Zero 2W integrated with Gemini, we achieved a much more stable and reliable system. This shift ensures that the user can trust the device for critical navigation tasks without fearing a sudden system failure.
Optimizing Power Consumption & User Experience (UX): The beta version was bulky and impractical. The combination of two boards and a high-draw Quectel 4G module required a massive battery pack that was uncomfortable to wear. We optimized the hardware footprint by focusing on a single-SBC WiFi-based system for this prototype. To further enhance battery life, we are implementing a voice-triggered sleep mode, allowing users to save energy while the device is not in active use—a feature that significantly improves the overall wearable experience.
The Mobility Trade-off: Transitioning from a dedicated 4G module to WiFi was a challenge for outdoor mobility. However, this allowed us to prioritize the Live Agent interaction and speed during development. We are now exploring more power-efficient LTE-M or 5G modules for future iterations to ensure that Voxsight remains a reliable companion even in areas with limited WiFi connectivity.

Accomplishments that we're proud of

High-Level Institutional Recognition: We had the honor of presenting Voxsight to Dante Rigmalia, the Chairperson of the National Commission for Disabilities (KND). She expressed immense interest in the project and advocated for its large-scale production. Furthermore, she offered to facilitate connections with government agencies to distribute the device nationally, ensuring it reaches and empowers as many visually impaired individuals as possible.
Real-World Impact at YPAB Surabaya: We conducted early trials with students at SMPLB-A & SMALB-A YPAB Surabaya. Their genuine enthusiasm and joy as they successfully identified Indonesian Rupiah banknotes independently was a turning point for us, validating that our technology can truly bridge social gaps.
Empowering Talented Individuals: We are proud to support high-achieving individuals, such as the YPAB students who perform Gamelan internationally. Voxsight aims to be their "eyes" when they navigate unfamiliar foreign stages or currencies during their world tours.
Hardware Consolidation & Efficiency: We successfully optimized a bulky, unstable two-SBC prototype into a sleek, single-board system using the Raspberry Pi Zero 2W. Achieving this level of portability while maintaining the power of Gemini’s multimodal AI is a significant technical win.
Solving Critical Safety Gaps: We have developed a system that addresses life-threatening problems: detecting open sewers and potholes that traditional white canes often miss. Moving from limited local detection to a sophisticated, descriptive AI assistant is a leap forward for assistive technology.

What we learned

Context over Labels: We learned that for the visually impaired, knowing what an object is isn't enough; they need to know where it is and how it affects their movement. Transitioning to Gemini's spatial reasoning taught us that providing human-like context (e.g., "A car is blocking the sidewalk 3 meters ahead") is far more valuable than simple object labels like "car" or "hole."
The Power of Multimodality: Our previous struggle with local YOLO models taught us the limitations of edge computing for complex tasks. By switching to Gemini, we learned how a Multimodal Live approach can handle vision and voice simultaneously with much higher accuracy and lower cognitive load for the user.
Empathy-Driven Development: Testing with students at YPAB Surabaya was a humbling experience. We learned that accessibility is not just a feature—it’s a necessity. Their feedback helped us realize that the device must be conversational and intuitive, leading us to prioritize Gemini Live for natural, two-way interaction.
Hardware-Software Synergy: We learned that "more hardware isn't always better." Moving from a dual-SBC setup to a single, optimized Raspberry Pi Zero 2W taught us the importance of efficient system architecture and the benefits of offloading heavy AI processing to the cloud to maintain a lightweight wearable form factor.
Social & Institutional Navigation: Through our discussion with Dante Rigmalia (KND), we learned that creating a great product is only half the battle. To create real change, we must align our technical solutions with government policy and institutional support to ensure widespread accessibility for those who need it most.

What's next for Voxsight

Moving forward, Voxsight is committed to fulfilling the heartfelt request of the students at YPAB Surabaya who told us, "Kak, we want to be able to read books," by leveraging Gemini’s advanced OCR and multimodal capabilities to transform our device into a personal literacy assistant. Following the strategic recommendation from Chairperson Dante Rigmalia (KND), our next steps involve scaling production and collaborating with government institutions for national distribution to ensure this technology reaches every visually impaired individual in need. We are also refining our hardware into a more lightweight, stylish "Smart Frame" while enhancing global features—such as multi-currency support and real-time translation—to empower high-achieving groups like the YPAB Gamelan team during their international tours. Ultimately, we aim to build a hybrid safety system that ensures reliability in any environment, and you can follow our journey and see the product in action at voxsight.id.