azure - computer use

Multimodal agentic workflow for better web accessibilities

Comment

Inspiration

  • Accessibility Limitations: Existing web accessibility solutions often fall short for individuals with physical or technical challenges.
  • Prompt Engineering Overhead: Even with Generative AI, achieving optimal results requires significant prompt engineering, leading to potential hallucinations.

What it Does

  • Enhanced Accessibility: Provides improved accessibility and more interactive communication with AI agents.
  • Hallucination Mitigation: Reduces hallucinations by addressing context mismatches through intelligent agent interactions.

How We Built It

  • Backend Development: Implemented the backend using Azure AI and our inhouse agent SDK, versionhq.
  • Detailed Implementation: Please refer to the project's GitHub repositories for comprehensive details.

Challenges Faced

  • Mock Browser Configuration:

    • Initial approach: Electron (Node.js) proved insufficient for required actions.
    • Solution: Migrated to Selenium (Python) and Playwright (Python) for robust browser control.

  • Azure Ecosystem Learning Curve: Navigating and integrating with the Azure ecosystem required a steep learning curve.

  • Multi-modal Handling:

    • The initial approach to multi-modal data handling relied on backend recording, which resulted in limited flexibility.
    • Implement client-side processing using the built-in JavaScript packages. This allows for greater flexibility and responsiveness in handling multi-modal data.

Accomplishments

  • Production-Ready Quality: Successfully developed a solution with production-level quality and stability.

Lessons Learned

  • Computational Resources: Utilizing multiple AI models demands substantial computational resources.
  • Azure AI Expertise: Gained valuable experience and knowledge in leveraging Azure AI services.

Future Directions (Project Azure)

  • Security Implementation: Address crucial security considerations, particularly for browser control, which were beyond the project's initial scope.
  • Model Diversification: Explore and integrate a wider range of AI models for enhanced performance.
  • Azure Service Integration: Deepen integration with various Azure services to expand functionality and capabilities.

Built With

  • azure-ai
  • whisper
Share this project:

Updates