CivicSense

Architecture Diagram
Chat Screen

Inspiration

Government services and benefits can be overwhelming to navigate, with complex eligibility criteria and application processes. Many citizens struggle to understand what benefits they qualify for and how to access them. CivicSense was inspired by the need to make government assistance programs more accessible and understandable to the general public, starting with Illinois Department of Human Services (IDHS) programs.

What it does

CivicSense is an intelligent chatbot that helps citizens navigate government policies and benefits, specifically focused on IDHS services. Key features include:

Provides easy-to-understand information about benefits like SNAP (Supplemental Nutrition Assistance Program)
Helps users understand eligibility criteria and application processes
Maintains contextual memory for natural follow-up conversations
Includes direct references to official sources and documentation
Offers a comprehensive analytics dashboard for monitoring system performance
Provides real-time groundedness and relevance metrics using TruLens evaluation

How we built it

Technical Architecture CivicSense is engineered with a sophisticated architecture that combines modern AI technologies with robust evaluation systems:

Frontend (StreamlitChatBot):
- Implements a responsive chat interface
- Features a comprehensive dashboard for performance analytics and monitoring
RAG System Architecture:
- Snowflake Cortex Search for efficient document retrieval
- Mistral Large LLM for natural language understanding and generation
- Custom context processing for accurate responses
Evaluation System (TruLens Integration):
- CortexEvaluator implements sophisticated feedback mechanisms:
  - Groundedness measurement with chain-of-thought reasoning
  - Context relevance scoring
  - Answer relevance metrics
  - Cost estimation feedback
Performance Analytics:
- Comprehensive metrics tracking through custom SQL queries
- Real-time monitoring of:
  - Quality Metrics
  - Cost analysis
  - Performance
  - Model Evaluation
- Version comparison capabilities for continuous improvement

Challenges we ran into

Resource Management: Credits management became a significant challenge as we scaled our testing and development. We had to strategically use multiple accounts and optimize our usage patterns to maintain development momentum while staying within resource limits.

Accomplishments that we're proud of

Comprehensive Evaluation System: Successfully implemented TruLens for sophisticated evaluation of response quality, including:
- Groundedness measurement
- Context relevance scoring
- Answer relevance metrics
- Cost tracking
Advanced Analytics Dashboard: Created a detailed monitoring system that tracks:
- Real-time performance metrics
- Cost analysis
- Quality metrics
- Version comparisons
Natural Interaction: Achieved a conversational interface that maintains context while providing accurate, sourced information.

What we learned

Building CivicSense has been an intensive learning experience in modern RAG system development. The project taught us the critical importance of evaluation metrics in AI systems - it's not just about getting answers, but ensuring they're accurate, relevant, and grounded in source documentation. Through TruLens integration, we learned to quantify and monitor these aspects systematically, leading to continuous improvement in response quality. This experience has given us deep insights into building production-grade AI systems that can handle sensitive information while maintaining high standards of accuracy and reliability.

What's next for CivicSense

Expansion of Services:
- Include additional government programs and services
- Expand to other states beyond Illinois
Enhanced Features:
- Multi-language support
- Personalized eligibility checking
- Interactive application assistance
Technical Improvements:
- Advanced document preprocessing
- Enhanced context understanding
- More sophisticated performance optimization