Harviy.ai - Smarter Hiring, Humanized by AI

We digitize pre-interviews with AI-powered 3D avatars, delivering smart behavioral and emotion analytics to empower HR teams with faster, data-driven hiring decisions.

Comment

Harviy.ai

Inspiration

As a job candidate, I experienced firsthand how recruitment processes can become slow and inefficient. HR professionals often face back-to-back interviews, leading to fatigue and reduced performance, which delays hiring decisions. This inspired me to build Harviy.ai, a solution that automates candidate pre-screening through AI-powered 3D avatars, streamlining the hiring process and reducing workload for HR teams.

What it does

Harviy.ai enables HR admins to create recruitment meetings via an admin panel, input candidate emails, and specify interview topics and questions using a large language model (LLM). The system conducts AI-driven interviews with 3D avatars that interact with candidates through voice.

Key features include:

  • Real-time video recording to analyze candidate behaviors, facial expressions, emotions, and body posture using TensorFlow.js and Human.js.
  • Voice transcription via speech-to-text services such as Google Speech API and Web Speech API.
  • Contextual conversation management using Google Gemini LLM.
  • Text-to-speech responses powered by ElevenLabs and Google TTS.
  • Automated generation of detailed behavior and emotion analysis reports.
  • Batch processing of candidate lists and centralized dashboard for HR teams to monitor interviews and results.
  • Secure storage of conversation history and analysis data in MongoDB.
  • Modular API architecture for seamless integration with major HR platforms.

How we built it

We developed the MVP by combining several cutting-edge AI technologies and web tools:

  • Large Language Models (Google Gemini) for natural and contextual conversation.
  • Speech recognition with Google Speech-to-Text and Web Speech API.
  • Text-to-Speech using ElevenLabs and Google TTS for realistic avatar voice.
  • Real-time facial, pose, and emotion analysis using TensorFlow.js and Human.js.
  • 3D avatar rendering and animation synchronized with speech using Three.js.
  • Backend and data storage handled by Node.js and MongoDB.
  • Admin panel built with React for candidate management and reporting.

The system's modular API design allows easy future integration with existing HR platforms.

Challenges we ran into

  • Fine-tuning language models to create natural, engaging AI conversations.
  • Real-time synchronization of avatar animations with speech and emotion analysis.
  • Implementing meaningful behavioral data collection while ensuring user privacy and compliance with regulations such as GDPR.
  • Optimizing performance to handle video processing and machine learning models efficiently in-browser.

Accomplishments that we're proud of

  • Successfully created a functional MVP automating candidate interviews through AI avatars.
  • Developed real-time behavior and emotion analytics without relying heavily on server-side processing.
  • Built a scalable and modular platform that can integrate with popular HR systems.
  • Delivered an intuitive admin dashboard enabling HR teams to easily manage interviews and access insightful reports.

What we learned

  • Combining natural language processing, speech synthesis, and computer vision technologies requires robust system architecture and careful coordination.
  • Automating routine HR tasks improves efficiency and reduces human workload significantly.
  • Ethical considerations, especially regarding privacy in facial and emotion data collection, must be prioritized.
  • User experience and seamless automation are critical for HR adoption and success.

What's next for Harviy.ai

  • Expanding interview capabilities to include technical assessments and skill-based testing.
  • Enhancing avatar responsiveness and emotional realism for more human-like interactions.
  • Deepening integrations with major HR platforms such as SAP, Greenhouse, and Lever and Linkedin.
  • Introducing advanced analytics to track hiring metrics, candidate sentiment trends, and improve decision-making.
  • Positioning Harviy.ai as the preferred digital assistant for recruitment processes worldwide.

Built With

Share this project:

Updates