AIVoiceRecognition

GitHub
Execute - Human
Execute - IA
Trancription

Inspiration

We were assigned a challenge presented by ALTUR related to its call agent. Although this is not currently a critical problem, there is a concern that, with the rapid development of artificial intelligence technologies, it could become a significant risk in the near future.

The main challenge is to identify who is interacting with the call agent: a real person or an artificial intelligence system, allowing us to detect synthetic voices during a call.

What it does

The system analyzes audio characteristics over different time intervals, allowing it to progressively evaluate the call and estimate the probability that the voice belongs to a real person or was generated by artificial intelligence.

The application receives input data, processes and analyzes the audio through our model, and returns a result indicating whether the analyzed voice is human or synthetic.

How we built it

The core logic of our solution was developed using LightGBM and layered logistic regression. We defined three analysis layers or computational budgets, which share the same family of features related to different audio characteristics, such as silence, background noise, tone, and voice behavior, among others.

The system analyzes these characteristics over different time intervals, allowing it to progressively evaluate the call and estimate the probability that the voice belongs to a real person or was generated by artificial intelligence.

For the implementation of our solution, we used Vultr to deploy a cloud instance that hosts our project. We also used .Tech Domain to set up our own domain and developed a web-based frontend that provides a simpler and more visual way to send requests to the system and view the results.

Additionally, we used MongoDB Atlas to store cases where the model has a high level of uncertainty, allowing us to keep track of these samples for further analysis and future improvements to the model.

We also used ElevenLabs to process the audio, separate the conversation into two channels, convert data between JSON and multimedia audio formats, and obtain the transcription of the conversation for further analysis.

Challenges we ran into

One of the main challenges we faced was selecting the right features to analyze the audio. A voice can be affected by many different factors, so we had to consider several elements that could influence the classification process.

We analyzed everything from seemingly simple characteristics, such as silence, pauses, background noise, and tone of voice, to more complex aspects related to the different ways people speak and express themselves. The main challenge was identifying which of these features provided truly relevant information to reliably distinguish between a human voice and an AI-generated voice.

Accomplishments that we're proud of

Our main achievement was developing a complete, functional, cloud-based application capable of receiving input data, processing and analyzing audio through our model, and returning a result indicating whether the analyzed voice is human or synthetic.

In addition, we implemented a structure that allows us to continue collecting information and progressively improve the algorithm's performance through new data and testing.

What we learned

During the development of this project, we strengthened our ability to work collaboratively as a team, effectively divide responsibilities, and coordinate different areas of development.

We also gained experience in the complete process of designing, developing, and implementing a technological solution for a real-world industry problem, from the initial idea and model development to system integration and cloud deployment.

What's next for AIVoiceRecognition

Our next step is to test the application in a production environment to evaluate its performance under real-world conditions. This will allow us to identify possible improvements, determine the system's performance and stress limits, evaluate the model's accuracy under different audio conditions, and make any necessary adjustments.

Finally, we plan to continue improving and refining the user interface to make the system easier to use and prepare the solution for a potential integration with ALTUR's call system.