Crashly

Inspiration

Car accidents happen in seconds, but emergency response often takes minutes or hours. We were inspired by the critical "golden hour" concept in emergency medicine - the first hour after a traumatic injury when medical treatment is most likely to prevent death. Our team realized that modern vehicles are equipped with cameras and sensors, but lack intelligent systems that can instantly detect crashes, assess severity, and coordinate emergency response. We envisioned a system that could act as a digital first responder, using AI to bridge the gap between accident occurrence and professional help arrival.

What it does

Crashly is an intelligent car crash detection and emergency response system that combines computer vision, AI analysis, and automated emergency calling. The system: Real-time Crash Detection: Uses YOLO object detection and OpenCV to analyze dashcam footage, detecting crashes through multiple indicators including vehicle collisions, sudden disappearances, rapid acceleration changes, camera shake, and visual artifacts like blur and brightness changes. Multi-Modal AI Analysis: Employs Cerebras AI for contextual scene understanding and Google Gemini for detailed forensic analysis, creating comprehensive crash reports with severity assessment, vehicle identification, and damage evaluation. Intelligent Emergency Response: Automatically initiates AI-powered voice calls to emergency contacts, conducting natural language conversations to assess injuries, confirm location, and determine the level of help needed. Comprehensive Documentation: Generates detailed insurance accident reports and 911 dispatch summaries, including license plate recognition, collision mechanics analysis, and pre-crash scene documentation. Web Dashboard: Provides a user-friendly interface for uploading dashcam footage, viewing analysis results, managing emergency contacts, and accessing processed videos with crash detection overlays.

How we built it

Backend Architecture: Computer Vision Pipeline: Built with Python, OpenCV, and Ultralytics YOLO for real-time object detection and tracking Multi-AI Integration: Cerebras for contextual analysis, Google Gemini for detailed forensic examination Emergency Calling System: Twilio integration with AI-powered voice conversations using speech-to-text and natural language processing Data Management: MongoDB Atlas for user profiles, crash sessions, and emergency contact storage API Framework: FastAPI for robust backend services with real-time WebSocket support Frontend Development: Modern Web Interface: Next.js with React and TypeScript for responsive user experience Real-time Updates: WebSocket integration for live crash detection status and processing updates Video Processing: Client-side video upload and server-side analysis with processed video playback AI and Machine Learning: Object Detection: YOLO v8 for vehicle identification and tracking Visual Analysis: OpenCV algorithms for detecting camera shake, blur, brightness changes, and motion patterns Context Understanding: Cerebras LLaMA models for scene interpretation and emergency assessment Forensic Analysis: Google Gemini for detailed crash reconstruction and damage assessment Integration and Deployment: Cloud Infrastructure: MongoDB Atlas for database, environment-based configuration management Telephony: Twilio for voice calling with webhook-based conversation handling Video Processing: FFmpeg integration for video analysis and overlay generation

Challenges we ran into

Computer Vision Complexity: Distinguishing between actual crashes and false positives (like cars turning, parking, or temporary occlusions) required developing sophisticated multi-factor detection algorithms combining vehicle tracking, visual artifact analysis, and temporal pattern recognition. Real-time AI Processing: Balancing detection accuracy with processing speed while handling multiple AI models (YOLO, Cerebras, Gemini) in a coordinated pipeline without overwhelming system resources. Voice Interaction Architecture: Creating natural AI conversations through Twilio required building a complex webhook system to handle speech-to-text, context management, and real-time response generation while maintaining conversation state across multiple exchanges. Multi-Modal Data Fusion: Synchronizing outputs from different AI models (visual detection, scene understanding, forensic analysis) into coherent crash reports while ensuring data consistency and avoiding conflicts between model interpretations. Emergency System Reliability: Building fail-safes for critical emergency calling functionality, including fallback responses when AI services are unavailable and ensuring calls complete even with network issues. Video Format Compatibility: Handling diverse dashcam video formats, codecs, and quality levels while maintaining consistent detection performance across different recording conditions. Accomplishments that we're proud of Comprehensive Detection System: Successfully integrated multiple detection methods (object tracking, visual artifacts, motion analysis) into a unified system that achieves high accuracy while minimizing false positives. AI-Powered Emergency Response: Created the first system that combines crash detection with intelligent voice calling, enabling natural language assessment of emergency situations and appropriate response coordination. Real-time Processing Pipeline: Built a system capable of processing video in real-time while simultaneously running multiple AI models and generating comprehensive analysis reports. Human-Centered Design: Developed an intuitive web interface that makes advanced AI technology accessible to everyday users, with clear visual feedback and easy emergency contact management. Production-Ready Architecture: Implemented a scalable system with proper error handling, logging, database integration, and real-time updates that could be deployed in real-world scenarios. Multi-AI Orchestration: Successfully coordinated three different AI systems (YOLO, Cerebras, Gemini) to work together, each contributing specialized analysis while maintaining overall system coherence.

What we learned

AI Model Specialization: Different AI models excel at different tasks - YOLO for object detection, Cerebras for contextual understanding, and Gemini for detailed analysis. The key is orchestrating them effectively rather than trying to use one model for everything. Emergency System Design: Building systems for emergency scenarios requires different design principles - prioritizing reliability over features, implementing multiple fallback mechanisms, and ensuring critical functions work even when other components fail. Real-time Video Processing: Processing video streams in real-time while maintaining quality requires careful optimization, including frame sampling strategies, efficient memory management, and parallel processing techniques. Voice Interface Complexity: Creating natural AI conversations is significantly more complex than text-based interactions, requiring careful state management, context preservation, and handling of speech recognition uncertainties. User Experience in Crisis: Designing interfaces for emergency situations taught us the importance of clear visual hierarchies, immediate feedback, and reducing cognitive load when users may be stressed or injured. Integration Challenges: Connecting multiple external services (Twilio, MongoDB, various AI APIs) requires robust error handling, rate limiting, and graceful degradation when services are unavailable.

What's next for Crashly

Advanced Sensor Integration: Expand beyond camera-based detection to incorporate accelerometer data, GPS information, and OBD-II vehicle diagnostics for more comprehensive crash detection and severity assessment. Predictive Analytics: Develop machine learning models that can predict high-risk driving situations before crashes occur, potentially preventing accidents through early warning systems. Emergency Services Integration: Build direct API connections with 911 dispatch centers, hospitals, and tow truck services to automate emergency response coordination and reduce response times. Multi-Vehicle Coordination: Create a network effect where multiple vehicles can share crash detection data to improve accuracy and provide multiple perspectives on accident scenes. Insurance Automation: Develop deeper insurance industry partnerships to automatically initiate claims processing, schedule vehicle inspections, and coordinate repair services immediately after crash detection. Mobile Application: Build companion mobile apps that can serve as backup systems when dashcam systems fail, using smartphone sensors and cameras for crash detection. International Expansion: Adapt the system for different countries' emergency services, traffic laws, and insurance requirements to enable global deployment. AI Model Improvements: Continuously train and refine detection algorithms using real-world crash data to improve accuracy and reduce false positives across diverse driving conditions and vehicle types.

Built With

• typescript