Spokesperson AI

🚀 Inspiration

In our ever-evolving world of digitization, we've identified a need to enhance the one-on-one interview experiences with the power of analytics. This need, coupled with the potential to harness technologies such as Segment and AWS for event detection, information storage, and data analysis, triggered the conception of Spokesperson - a digital platform that transforms interviews into quantifiable data sets.

🛠️ What it does

Spokesperson is an engaging, intuitive platform designed to meticulously capture and analyze every moment of a job interview. Through amplified-interview tracking technology and event detection, it deeply tracks an interviewee's facial cues, voice modulation, and topic engagement. Using AWS Rekognition and Comprehend, it performs face analysis and NLP of transcribed conversations. Each of these are discreetly recorded as Segment events and stored in an AWS S3 Bucket. Notably, our platform's intelligence doesn't stop at data collection - it goes the extra mile to flag areas of high toxicity, neatly bundling these instances and reporting them via a swift Slack notification using Segment Destinations.

🧑‍💻 How we built it

Building Spokesperson was a journey through a technological labyrinth. First, we designed the app's functionality and interface on spokesperson.app, creating a simple and easy-to-use platform that caters to interviewers and interviewees. Each functionality trigger, from camera regulation to event reporting, is coded to record as a Segment event. AWS Rekognition was then integrated to facilitate facial analysis, transcribing every frame of the interviewee's camera and storing it as a Segment event. The app was further developed to capture audio, transcribe it and perform NLU using AWS Comprehend and Transcribe. To record and store all this data, Segment Custom Functions were coded to autonomously save them to an AWS S3 bucket.

🏞️ Challenges we ran into

Our main challenge was the integration of multiple tools like Segment, AWS S3, AWS Comprehend and Rekognition for seamless data collection, analysis, and storage. At times, capturing detailed impressions of an interviewee while maintaining a smoothly-operating interface posed a noteworthy challenge. Craftsman-like perfection was required while from integrating Segment Destinations for Slack event notifications to AWS Bedrock utility with the Amazon Titan text model for generating post-meeting insights and reports.

🎉 Accomplishments that we're proud of

We're profoundly proud of designing an integrated, intuitive platform that enhances the interviewee's experience while also providing an invaluable tool for interviewers. Be it the smooth integration of multiple AWS tools or the incorporation of Segment for recording complex event data, each of these accomplishments stand as testament to our hard work. Our crowning achievement, however, is our Report Event button, offering interviewers instant reporting of any irregularities that occur during an interview, integrating responsible reporting with the convenience of immediate Slack notifications.

🎁 What we learned

Our journey taught us the intricacies of integrating various cloud services, the power of data collection and analysis, and the nuances of creating a responsive, user-friendly interface. The project revealed the potential that machine learning and artificial intelligence can add to basic interfaces, bolstering their performance.

🔮 What’s next for Spokesperson

As we look forward to the future of Spokesperson, we're keen on introducing predictive analytics for better insight-gathering, along with improved reporting functionalities. Harnessing more advanced AI services to offer real-time insight to interviewers, and even moving beyond job interviews to encompass all sorts of 1:1 interactions, also holds a place in our roadmap. With the ability to potentially revolutionize data collection and event reporting, we are poised to navigate the future world of digital interviews. Stay tuned!

Built With

• amazon-web-services
• s3
• segment
• segment-destinations
• slack
• typescript