RoadSenseAI

Transforming Community Reports into Intelligent Road Decisions

Comment

Inspiration

From personal experience of getting stuck in traffic on the very day of exam.

What it does

This solution creates a feed for any traffic related issues with proofs like images, videos, text descriptions as well. Similar incidents will be merged together. The Rain forest Machine learning logic is used to determine the severity of the incident. The incidents are stored along with location to maintain clear description of incident and helping people to take decisions accordingly. For the future improvements decided to add Location-Based Smart Notifications, Emergency Vehicle Route Optimization, Predictive Traffic Congestion Forecasting, AI-Based Alternate Route Recommendations.

How we built it

RoadSenseAI was built as a full-stack AI-powered web application using React, FastAPI, TiDB Cloud, Google Gemini API and Scikit-learn. The platform allows users to report road incidents through text, images, voice, or short videos, which are processed by Gemini to extract structured information such as the incident type, location, severity, confidence score, and an AI-generated summary. A Random Forest machine learning model, trained on the Indian Road Accident Dataset (2022–2025), is used to analyze historical accident patterns and support traffic impact prediction. The processed incident data is stored in TiDB Cloud and visualized on an interactive OpenStreetMap using Leaflet, enabling users to view real-time incidents and make informed travel decisions. The application follows a modular architecture with separate frontend, backend, AI, and ML components, ensuring scalability, maintainability, and a seamless user experience.

Challenges we ran into

One of the biggest challenges was processing and integrating multiple input formats including text, images, voice, and video into a unified AI workflow. Extracting accurate and structured incident information from diverse user inputs while maintaining consistency required careful prompt engineering and backend orchestration. Another challenge was identifying a suitable dataset for training the machine learning model and mapping its features to real-world incident reports. Integrating AI-generated insights with the ML model, live map visualization, and TiDB Cloud in a modular and scalable architecture within the limited hackathon timeline also required careful design decisions. Balancing performance, usability, and rapid development while delivering a functional MVP was a key challenge throughout the project.

Accomplishments that we're proud of

We successfully built RoadSense AI, an AI-powered community traffic decision intelligence platform that transforms unstructured multimodal reports into actionable road intelligence. We integrated Google Gemini to analyze text, images, voice, and video inputs, trained a Machine Learning model using a real-world Indian road accident dataset, and visualized incidents on a live map using OpenStreetMap. We're proud of designing a scalable architecture with React, FastAPI, and TiDB Cloud, while delivering a functional MVP within a very short hackathon timeline.

What we learned

Through this project, we gained hands-on experience in integrating Generative AI with traditional Machine Learning to solve a real-world problem. We learned how to process multimodal user inputs, design effective prompts for LLMs, train and integrate ML models, work with TiDB Cloud, and build scalable full-stack applications using React and FastAPI. More importantly, we learned how to rapidly prototype, prioritize features, and make architectural decisions under tight deadlines.

What's next for RoadSenseAI

Our vision is to evolve RoadSense AI into a comprehensive AI-powered urban mobility and public safety platform. Future enhancements include real-time traffic and weather integration, duplicate report detection, AI-powered route recommendations, emergency service notifications, community trust scores, predictive accident hotspot analysis, multilingual support, and dedicated dashboards for traffic authorities and smart cities. We also plan to leverage TiDB's vector search capabilities to build a conversational assistant that allows users and city officials to query historical incident data and receive intelligent insights through natural language.

Use of Gemini API

  1. Text Understanding: Analyzes user-submitted text reports to identify the incident type, location, severity, and generate a concise summary.
  2. Image Analysis: Processes uploaded images to detect accidents, traffic congestion, road blockages, damaged vehicles, and other visual cues.
  3. Video Analysis: Extracts key information from uploaded videos by understanding the scene and identifying traffic-related incidents.
  4. Voice Processing: Converts voice reports into text (using speech-to-text) and then analyzes the content to extract structured incident details.
  5. Structured Data Extraction: Converts unstructured multimodal inputs into structured JSON containing fields such as incident type, location, severity, confidence score, and AI-generated summary.
  6. Natural Language Understanding: Understands different reporting styles and languages, allowing users to describe incidents naturally instead of filling complex forms.
  7. Decision Support: Generates human-readable incident summaries and recommendations that help commuters quickly understand the situation.

Built With

Share this project:

Updates