Inspiration
Every year, millions of critical situations occur where seconds determine survival—whether on the road navigating high-speed brake failures, aquaplaning, and severe collisions, or in homes managing cardiac arrests, choking, and burn trauma during the critical "Golden Hour." At the same time, standard computer operating systems and digital tools remain rigid, inaccessible to individuals with motor or cognitive impairments, and disconnected from the physical environment around them.
We were inspired by Smart India Hackathon (SIH Problem Statement SIH26204) under Assistive Technologies and Inclusive Innovation to create ARHIEA (Adaptive Road and Human Intelligence Emergency Assistant). Our vision was to craft an intelligent, multimodal Windows desktop companion that merges life-saving road and human emergency protocols with frictionless desktop automation, environmental computer vision, and inclusive accessibility.
What it does
ARHIEA functions as an autonomous, voice-first AI copilot designed for both daily productivity and high-stakes emergencies: Life-Saving Emergency & Road Intelligence: Instantly dispenses step-by-step, voice-guided critical procedures for vehicle emergencies (brake failure recovery, hydroplaning mitigation, tyre blowout stabilization) and medical trauma (CPR with 100–120 bpm rhythm guidance, Heimlich maneuver, burn care, and automated SOS dispatch). Conversational AI & Lifelong Neural Knowledge: Equipped with a hybrid AI brain supporting cloud LLMs (Google Gemini 2.5 Flash, Anthropic Claude 3.5 Sonnet, OpenAI GPT-4o, DeepSeek R1) along with an autonomous local knowledge engine and self-learning system that permanently remembers user preferences, routines, and taught concepts in an embedded SQLite brain. Environmental Vision & Spatial Memory: Uses real-time computer vision with Exponential Moving Average (EMA) coordinate smoothing to detect objects, assess their relative spatial locations (e.g., "on the desk", "in hand"), and answer queries like "Where did I leave my keys?". Autonomous Operating System Automation: Discovers and opens installed Windows applications, creates and compiles code (calculator.py, notes in .docx/.txt/.xlsx), searches live web data, controls system telemetry, and automates multi-turn flight bookings across Google Flights, MakeMyTrip, Skyscanner, and Expedia. Inclusive Accessibility Suite: Features one-click accessible adaptations for users with physical and neurodiverse needs, including OpenDyslexic typography, High-Contrast mode, Large Font scaling, Voice-First interaction, and Simplified Low-Cognitive-Load layout.
How we built it
ARHIEA was engineered using a modular client-server architecture optimized for sub-second response times and complete offline reliability: Backend & Agent Engine: Built with Python 3.11 and FastAPI, featuring a two-tier decision pipeline: A sub-5ms FastDeterministicRouter that intercepts system time, math solving, flight booking, emergency protocols, and desktop actions. An agentic reasoning loop (ArhieaAgent) integrating tool-calling, web knowledge extraction, and fallback to the internal KnowledgeBaseEngine. Multimodal Speech Pipeline: Speech-to-Text (STT): Multi-tier architecture prioritizing pure-Python SpeechRecognition with phoneme and mathematical transcription normalization (translating spoken math like "x square plus 5" into 𝑥 2 + 5 x2+5 and normalizing audio time stamps). Text-to-Speech (TTS): High-fidelity speech synthesis via Edge-TTS (7.3.1) with a non-blocking worker thread falling back to native Windows SAPI5 COM speech with explicit thread initialization to prevent event loop deadlocks. Vision & Detection: Powered by an ONNX neural detector and OpenCV video streaming pipeline with client-side WebRTC/canvas synchronization over full-duplex WebSockets. Frontend Dashboard: Crafted in React 18, TypeScript, Vite, and Tailwind CSS with a cyberpunk glassmorphism UI, interactive holographic orb (AuraCore), live audio waveform visualization, and real-time state broadcasts (LISTENING, THINKING, SPEAKING). Database & Persistence: Local SQLite orchestrated via SQLAlchemy, storing conversations, visual memories, active reminders, user profiles, and learned knowledge concepts. Challenges we ran into Windows SAPI COM Deadlock: When cloud TTS was unreachable, synchronous calls to Windows pyttsx3 inside FastAPI's async event loop locked up the COM message pump and froze the server. We resolved this by isolating SAPI in a separate daemon thread with explicit pythoncom.CoInitialize() and pythoncom.CoUninitialize() guards and timeout fallbacks. Restricted AppLocker DLL Blocks: Windows security policies blocked native C-extension DLLs like ctranslate2._ext.pyd. We re-architected our speech pipeline into a resilient multi-tier system with pure-Python fallbacks that operate reliably across any locked-down environment. STT Mathematical & Temporal Normalization: Spoken voice inputs frequently corrupted mathematical notation and clock times (e.g., recognizing "12:07 AM" as "12/7 AM"). We engineered a regex normalization parser that corrects phonetic math tokens, exponents, and time formats before the router processes them. Cloud API Latencies & Key Failures: Inconsistent network connections or invalid API tokens could introduce multi-second latency freezes. We implemented format validation (AIzaSy token prefixes) and instant zero-latency fallbacks to our local knowledge engine. Accomplishments that we're proud of 100% Test Suite Verification: Built and verified a comprehensive test suite of 59 out of 59 automated tests passing across authentication, deterministic routing, flight booking, visual memory, audio synthesis, speech normalization, and multi-AI integration. Sub-Second Voice Response: Achieved responsive, natural voice interaction with live audio waveforms, instant barge-in cancellation, and local fast-path execution. Lifelong Continuous Learning: Implemented a natural chat-based teaching system where users can teach ARHIEA facts (e.g., "Learn that X is Y"), which are immediately indexed, stored in the neural database, and recalled in subsequent conversations. Universal Accessibility: Delivered a truly inclusive design featuring dysgraphia/dyslexia-friendly fonts, voice-first commands, and high-contrast color schemes adhering to modern assistive standards. What we learned Graceful Degradation is Paramount: In mission-critical emergency and desktop tools, you cannot rely solely on external cloud APIs. Designing a multi-tiered architecture with local heuristic routers, offline databases, and pure-Python fallbacks is essential for true resilience. Async & OS Thread Safety: Interfacing asynchronous Python event loops with Windows native desktop APIs (COM objects, audio drivers, registry, Start Menu) requires strict thread isolation and proactive timeout guards. The Importance of Human-Centric AI: Building for emergency response and accessibility taught us that clarity, speed, and empathetic, low-cognitive-load design are just as important as algorithmic complexity.
What's next for ARHIEA - Adaptive Human Intelligence and Emergency System
On-Device Edge SLM Integration: Packaging quantized local Small Language Models (such as Phi-3 Mini or Llama 3.2 3B Q4 GGUF) via llama-cpp-python for 100% private, offline contextual reasoning. Automated Vehicle CAN Bus Telemetry: Connecting with OBD-II vehicle diagnostic dongles via Bluetooth to detect impact telemetry, engine temperature, and brake fluid pressure directly from vehicle hardware. Automated Emergency SOS Beacon: Integrating twilio/cellular fallback to send GPS coordinates, incident audio clips, and medical profiles directly to local emergency response dispatchers (112 / 911) during severe road collisions. Multi-Platform Desktop Packaging: Finalizing single-click installer distribution using Electron/Tauri for seamless one-click deployment across Windows 10 and 11 devices.
