Inspiration

Indian cities generate thousands of civic complaints every day potholes, garbage overflow, water leaks, and broken streetlights. We noticed that reporting these issues is often frustrating: citizens must fill long forms in English, download specific apps, or wait on helplines. This creates language and accessibility barriers, especially for people who aren’t tech-savvy or English speakers.

Our inspiration came from imagining a simple experience: “What if you could just send a quick voice note or snap a photo in your own language, and the city automatically took care of routing it to the right team?”

What it does

CivicSight is a multilingual agentic AI that:

Takes a voice note, photo, or video from a citizen in Indian languages.
Understands the issue: pothole, water leak, garbage overflow, or streetlight outage.
Finds the exact location.
Looks up municipal SOPs and assigns the correct department & SLA.
Creates a structured ticket and notifies the citizen with a ticket ID.
Reflects: if GPS/landmark info is missing, it asks the citizen for clarification in their own language.

How we built it

Agent Framework: IBM Agent Development Kit

Core LLM: IBM Granite Instruct

Vision: MobileNetV3/DeiT-tiny fine-tuned for 4 classes .

Speech: Whisper-small (multilingual STT) with auto language detection.

Geo: GPS + reverse geocoding + ward-level shapefiles.

RAG: FAISS/Chroma over municipal SOP snippets.

Frontend: Streamlit demo UI for image/voice upload.

Backend: Flask mock ticket server + FastAPI gateway.

Notifications: SMS/WhatsApp.

Challenges we ran into

Multilingual complexity: Ensuring short Kannada/Hindi/Tamil voice notes transcribe cleanly.

Low-resource data: Had to augment a small vision dataset.

Geo inaccuracies: GPS was missing in some cases solved via landmark OCR/NER.

Latency: Optimizing end-to-end pipeline to run under 10 seconds on Colab/Laptop.

Integration: Making different tools work together seamlessly under an agent loop.

Built With

audio)
chroma
control
deit-tiny)-speech-?-openai-whisper-small
docker-(optional)
email)
fastapi-databases-/-storage:-demo-datasets-(images
geopandas-frontend-?-streamlit-(primary)
huggingface-transformers-vision-?-pytorch
javascript
javascript-(for-frontend-if-react-is-used)-frameworks-&-libraries:-agent-framework-?-ibm-agent-development-kit-(adk)
langgraph-(fallback)-llm-&-nlp-?-ibm-granite-instruct
llama-3.1-instruct
ngrok-(for-tunneling-demos)-communication-/-notifications:-twilio-api-(sms/whatsapp)
nominatim
python
react-(optional)-backend-&-apis-?-flask
sentence-transformers-geo-tools-?-openstreetmap-apis
smtp
timm
torchvision-(mobilenetv3
vector-db-(faiss/chroma)-cloud-&-ops:-kaggle/colab-runtime
version
vosk-rag-/-vector-db-?-faiss