Percevia

Cover Image
App
Glasses
Website

Pitch deck: https://canva.link/e4zrt4uxgw1um9e App Download Link: https://github.com/Yashwardhan-ed/Percevia-App-UiPath-Agent-Hackathon/releases/download/v1.0.0/app-release.apk Website: https://percevia.in

Inspiration

The most capable AI models today share two assumptions that quietly exclude the people who need them most: they assume a reliable internet connection, and they live on a server. For the 8 million completely blind individuals in India, connectivity is weakest exactly when they need help the most—an unfamiliar street or a rural bus stand. Furthermore, existing assistive wearables are often priced like premium medical devices, or force users to awkwardly hold their phones, occupying the hands they need for a cane. We built Percevia to completely dismantle these barriers: an entirely offline, highly affordable wearable ecosystem that ensures true independence is never gated behind an internet connection or an exorbitant price tag.

What it does

Percevia is an offline-first vision assistant consisting of a smartphone app and custom smart glasses. It answers the critical question, "What is in front of me?" by describing scenes, reading text, recognizing faces, and warning of obstacles via voice and dynamic haptics.

Because we built the UI specifically for muscle memory (a rigid 2x2 grid of massive buttons), users can navigate it entirely by touch. When the glasses are connected, they act as a seamless hardware extension, freeing the user's hands entirely while the phone stays in their pocket processing the environment.

Caregiver Escalation: If a user is lost or needs help, speaking a critical phrase (like "help, I'm lost" or Hindi "mujhe madad chahiye") automatically bypasses the vision pipeline and triggers an urgent cloud-based caregiver notification workflow.

How we built it

Percevia fundamentally refuses traditional cloud architecture. We built an Edge-Compute Engine where our primary vision-language model, Gemma 4, lives entirely on the user's phone, accelerated locally via Google AI Edge's LiteRT on the device's GPU.

To achieve True Spatial Awareness, we built a continuous dual-model pipeline. We divide the user's field of view into a rigid 3x3 grid. When our YOLO model detects an obstacle, we map its centroid to a sector. Concurrently, we paired this with MiDaS (a monocular depth estimation model) to generate a dense topological depth map.

The Hardware Bridge consists of custom ESP32-based smart glasses that spin up their own Wi-Fi and serve an MJPEG stream. The app pulls a clean JPEG by scanning the byte stream for frame markers, feeding the exact same Gemma 4 LiteRT inference pipeline.

UiPath Agentic Layer: We built a hybrid cloud-edge agentic model. The app authenticates via OAuth 2.0 (client-credentials) as an External Application. When voice commands are spoken, the transcript is routed to a UiPath Agent Builder agent (percevia intent router running gpt-5.4-2026-03-05) for multilingual intent classification. If the intent is help, the app starts a UiPath Maestro BPMN workflow (CaregiverEscalation) in the cloud via the Orchestrator Jobs API. The Maestro process uses a standard HTTP Connector to instantly post structured alert variables (lastAction, triggeredAt, approximateLocation) to external caregiver messaging endpoints.

Challenges we ran into

Model Concurrency Crashes: Gemma 4 is a heavy model. We found that the LiteRT-LM engine crashes hard (SIGSEGV) if a second inference touches the model while one is still running. We solved this by wrapping every request in an exclusive lock that queues sessions safely and streaming tokens straight into our TTS engine to eliminate wait times.
Taming the OS Screen Readers: Android's native TalkBack swallowed the double-tap gestures we used to stop speech. Instead of forcing users to buy external Bluetooth clickers, we engineered an invisible, zero-size screen-reader-only overlay that exposes a "Stop reading" semantic action that TalkBack can focus on perfectly.
UiPath Cloud Integration and M2M Authentication: We hit a roadblock where OAuth 2.0 token requests returned redirect codes due to org-scoped vs tenant-scoped URL requirements on UiPath Automation Cloud. We resolved this by routing token requests directly to org-scoped identity endpoints. We also overcame a silent environment variable interpolation bug where flutter_dotenv evaluated raw client secrets containing $ as shell variables by wrapping the secrets in single quotes. Lastly, when the developer sandbox lacked licenses for Action Center forms, we successfully pivoted the Maestro process to use direct service-based HTTP connectors.

Accomplishments that we're proud of

True Offline Autonomy: Achieving high-accuracy obstacle detection and scene description entirely on the edge with zero cloud dependency.
Seamless Hardware Handoff: Creating a robust MJPEG over Wi-Fi stream that allows cheap, commodity hardware to rival devices that cost 10x as much.
Deep Accessibility: Designing a UI that completely rejects scrolling, opting for a static layout that can be mastered via muscle memory in minutes.
Hybrid Cloud-Edge Integration: Successfully merging edge-compute Vision LLMs with a highly secure cloud-based automation pipeline (UiPath Agent Builder & Maestro BPMN) for safety-critical alerting.

What we learned

We learned that in assistive technology, filtering out noise is just as critical as detecting the signal. Balancing the hardware constraints of microcontrollers with the heavy demands of spatial AI taught us the vital importance of software-level optimization (like our YOLO+MiDaS fusion grid) to solve hardware-level problems.

What's next for Percevia

We are actively transitioning from lab validation to field deployment. Having validated the core architecture, our immediate next step is to deploy our first pilot batch of 50 units into the hands of real users to stress-test the hardware ergonomics and edge-inference stability in daily, unpredictable environments.