SixthSense

Set of eyes inside your ears. An omnipresent digital assistant that takes in video input to help users with unique audio output.

Comment

Sixthsense: Enhancing Voice Assistance with Visual Context

Inspiration

The inspiration for this project came from observing two critical limitations of current voice assistants. First, while they can process voice commands and provide responses, they lack true contextual awareness of the user's environment - something that's crucial for natural human interaction. Second, their voice output tends to be noticeably robotic and fails to capture the richness and nuance of human speech. This mechanical delivery often makes it difficult to convey complex information effectively, lacking the natural intonation, emphasis, and emotional qualities that make human communication so effective. We were particularly moved by the potential impact a more natural and contextually aware system could have on people with visual impairments, those with cognitive challenges like dementia, and anyone who needs hands-free assistance in their daily activities. By combining true environmental awareness with more natural voice interaction, we believed we could create a more intuitive and helpful assistant that better serves users' needs.

What it does

  • Our solution combines real-time video processing with voice interaction to create a more comprehensive assistive experience:
  • Processes live video feed to understand the user's environment
  • Uses advanced action recognition to interpret what's happening around the user
  • Provides real-time audio feedback through natural voice communication
  • Offers contextual awareness for various situations like navigation, safety alerts, and crowd management
  • Features memory replay capabilities to help users recall important information

How we built it

The system architecture consists of several key components:

  1. Video Processing Pipeline
    • Real-time video capture and processing
    • Action recognition algorithms
    • Scene understanding modules
  2. Voice Interface
    • Natural language processing
    • Context-aware response generation
    • Human-like voice synthesis
  3. Integration Layer
    • Combining visual and audio processing
    • Real-time synchronization
    • User preference management

Challenges we ran into

Throughout development, we encountered several significant challenges:

Real-time Performance

  • Balancing processing speed with accuracy
  • Optimizing video analysis for mobile devices
  • Reducing latency in voice responses Context Integration
  • Video parsing across dual cameras
  • Creating meaningful connections between visual and audio data
  • Determining which information is most relevant to communicate
  • Handling multiple simultaneous events

Accomplishments that we're proud of

  • Developed software for and designed/built a hardware based project in 1 day
  • Create voices with human like tonality and voices
  • Implementing a novel algorithm for real time video recognition

What we learned

We learnt unique voice can help with familarity and

What's next for SixthSense

Future development plans include:

  • Expanding the action recognition capabilities
  • Implementing more sophisticated memory systems
  • Improving personalization features
  • Adding support for multiple languages
  • Developing more specialized modules for specific use cases (dementia care, workplace safety, etc.)

Built With

Share this project:

Updates