Pedagora

Picture which includes the place for user to insert video file.
Eye tracking technology in action, with the decimal value representing the score of eye contact.

Inspiration

As college students looking for internships, we found that the process of preparing for interviews was not only stressful, but lacked personalized resources that could tailor specific advice to individuals. This is why we decided to create Pedagora, a program that solves this problem by utilizing AI and computer vision to help people improve their conversational skills.

What it does

Pedagora takes in a video from the user answering a specific interview question of their choice as well the question that they answered in the video. We then transcribed this video into text, and then utilized ChatGPT to offer suggestions on how the user can improve their response. We also incorporated computer vision to detect facial expressions and gather data on the user's emotions. This data is then passed into ChatGPT, which then offers advice on how the user can improve on their mannerisms during an interview.

How we built it

We used Python as the main programming language, while Google Cloud was used for its Speech-to-Text API. The University of Michigan UM-GPT 4.0 API was used to analyze outputs derived from the transcribed text. We used the Deepface Python library to detect emotions. We used the eye-contact-cnn repository to analyze the user's eye contact. This is all integrated with the Streamlit library.

Challenges we ran into

We encountered difficulties with the backend implementations of the API keys that results from standard security practices. In addition, due to the amount of functions that our ambitious program attempts to provide, we encountered difficulties in implementation, whether that came from runtime issues or varying documentation from the plethora of libraries and repositories used.

Accomplishments that we're proud of

Despite our lack of previous experience at hackathons, we are proud of our efforts in making a working prototype by the end of the hacking period. We were able to identify a problem in the world that did not have many effective solutions, and in twenty-four hours, we were able to devise and implement a solution that can be directly utilized by people in not just our target demographic, but by any job-seeking adult.

What's next for Pedagora

The next steps for Pedagora would be to allow for live recordings to be inputted by the user, then using real-time facial tracking and analysis of individual frames, which would bypass the requirement of uploading a video beforehand. Another potential feature for Pedagora could be a direct audio and video editor that can directly edit and modify videos within the application, giving users finer control on the input and therefore obtaining more precise outputs.