Podchecks

Podchecks analyses and verifies claims made in podcasts, ensuring listeners get accurate, reliable information on the topics that matter most.

Comment 1

Inspiration

The inspiration for PodChecks came from watching James Smith's video "Downfall Of Modern Podcasts" link. It highlighted a trend we had started noticing ourselves - we were listening to fewer podcasts, but we weren't quite sure why. Smith's video broke down the reasons people like us were becoming dissatisfied with the quality and accuracy of podcast content:

  1. Many podcasts feature experts offering contradictory advice or viewpoints. This can confuse listeners and undermine the information's credibility, especially when shows focus more on sensationalism and controversy than factual accuracy to attract viewers.

  2. Some podcast guests or hosts misrepresent studies or facts to support their narratives, causing misinformation to spread among listeners.

  3. As demand for high-quality guests outpaces supply, podcasts may turn to controversial or less credible guests to maintain engagement. This can lead to more misinformation while detracting from the educational value.

After seeing thousands of comments echoing these points, we decided this problem was worth tackling. That's how PodChecks was born.

What it does

PodChecks is a new tool designed to combat the spread of misinformation in podcasts. It analyses the scientific claims made in podcasts and verifies that they are supported by legitimate scientific research. By doing so, PodChecks aims to improve the credibility and reliability of podcast content, allowing listeners to gain a more accurate understanding of the scientific topics discussed.

How we built it

PodChecks is an automated fact-checking tool designed to verify the accuracy of information presented in podcasts. It allows users to submit a YouTube link to a podcast through a user-friendly interface built with Bubble.io. The tool then extracts the video ID from the URL and retrieves the podcast's transcript using the YouTube Transcript API.

Next, PodChecks processes the transcript using Gemini 1.5 Pro to identify statements that appear factual. These claims are then sent to a custom API that integrates Google's Fact-Checking API and Wolfram Alpha's API to assess their veracity.

The fact-checking results are securely stored in a database and presented to the user through the Bubble.io interface, providing them with clear and reliable information about the accuracy of the podcast's content.

Challenges we ran into

-Gemini was instructed to return extracted claims as a JSON dictionary, but instead returned a JSON string. We had to convert the JSON string to a dictionary using a JSON load function.

-We also encountered issues integrating our fact-checking tool (Google Fact Check API + Wolfram Alpha API) with Vertex AI. The APIs return outputs, but they don't properly bind to the Gemini model.

  • Number of claims extracted by Gemini 1.5 pro is inconsistent.

The Gemini model was unable to call the manually defined assistant tools. This presented several challenges, including:

  • Evaluating the response of our fact-checking tool

  • Extracting necessary data from the YouTube API and populating it into our application

  • Facilitating communication between our low-code interface and AI backend

Despite these challenges, we are proud of our accomplishments:

  • Successfully leveraging Gemini 1.5 pro’s capabilities for a community-oriented purpose

  • Collaborating effectively between code and low-code environments

  • Developing the user interface with Bubble.io to ensure it is both functional and user-friendly was crucial. Building a solid UI/UX interface within the given time constraints

  • We couldn’t access Google AI API for Gemini 1.5 pro from the UK, so we had to use Vertex AI.

What we learned

  • Integrating multiple APIs (YouTube Transcript API, Google's Fact-Checking API, and Wolfram Alpha's API) taught us the importance of effectively querying APIs, handling varied response formats, and adapting to discrepancies in data formats.

  • Working with the Gemini 1.5 Pro model highlighted the limitations and variability of AI models in practice, particularly regarding reliability and output consistency.

  • Building the user interface on Bubble.io within time constraints emphasised the significance of creating an intuitive and aesthetically pleasing interface that effectively displays complex fact-checking results.

  • We discovered that YouTube comments sections contain a wealth of valuable consumer insights. These comments serve as a dashboard of what people care about and how they feel about specific topics. Additionally, viewership statistics of certain videos relative to previously uploaded videos by a given channel can provide useful insights into the popularity and engagement rate of a video, directly correlating to specific traits in consumer behaviour. For instance, "Downfall of Modern Podcasts” gained around 766k views and 2,770 comments just 9 days after its upload. For a channel with only 370k subscribers, this was exceptional growth and James Smith's best performance to date (based on growth rate). This signalled to us that people genuinely cared about the topic.

What's next for PodChecks?

-We plan to enhance the robustness of our fact-checking process, which currently relies heavily on Gemini's training data, often resulting in broken links. To address this issue, we will query claims using keyword searches via the Google SERP API. Upon locating the claim from the source, we will feed it into Gemini to structure the search results in a user-friendly format.

-Increase our rate limit to enable the analysis of longer videos.

-Make timestamps clickable, allowing users to start the video from the clicked point.

-Calculate an overall score for podcasts based on the proportion of factual information they contain. -Implement a user feedback system for our fact-checking results, which will not only improve the tool's accuracy but also foster deeper user engagement.

-In the future, we aim to develop real-time fact-checking capabilities for live podcast streams, providing listeners with instant feedback.

-Utilise collected data to provide podcast creators with detailed analytics on the factual accuracy of their content over time, along with suggestions for improvement. This service could potentially generate revenue from creators willing to pay for such insights.

Built With

Share this project:

Updates