Inspiration We were inspired by the challenge of navigating the modern information landscape, where differentiating between genuine consensus and coordinated manipulation (like astroturfing) is incredibly difficult, especially when analyzing fast-moving narratives online. Accurate information is vital for making good decisions, whether in finance, research, or daily life. We wanted to build a tool that automates the analysis of narratives, focusing not just on what is being said, but who is saying it and how trustworthy they appear. Knowing that, here's an updated version of the hackathon project description sections, reflecting the use of the advanced WCRS model involving LLM feature extraction:
What it does Truthy is a real-time narrative analysis engine. A user inputs a topic or keyword, and Truthy fetches recent posts from connected sources (initially X/Twitter and Reddit). It then performs a multi-stage analysis:
Micro-Analysis: Calculates sentiment and a detailed, multi-dimensional WCRS (Weighted Cluster Rank Score) trustworthiness score for each individual post. This score is derived by using an LLM (like GPT via OpenAI API) to quantify features across pillars like Authority, Evidence, Objectivity, Clarity, Linguistic Integrity, and User Metrics based on the post's text and metadata.
Macro-Analysis: Vectorizes & Clusters: Groups posts with similar meanings into narrative clusters using text embeddings and K-Means. Analyzes Clusters: Generates AI summaries for the main narrative and significant alternative viewpoints. It calculates the "authentic strength" of each narrative cluster, weighted by the WCRS trustworthiness scores of the posts within it. Visualizes: Creates an interactive 2D scatter plot showing the narrative clusters. Generates Consensus: Produces a final, concise takeaway message, considering the strength and WCRS trustworthiness of the different viewpoints identified. The results are streamed back to the user via a WebSocket connection.
How we built it Backend: Python with FastAPI for the web server and WebSocket endpoint. Data Sources: Integrated X (Twitter) using Tweepy and Reddit using PRAW via a modular connector system. Analysis Pipeline (whack_er_bot.py & truth.py): Sentiment: Used NLTK's VADER for fast sentiment scoring. Trustworthiness: Implemented a detailed WCRS scoring model (truth.py) utilizing the OpenAI API (e.g., GPT models) to extract and quantify numerous features (like citation density, subjectivity, account age etc.) based on predefined rules and platform-specific configurations. It also involved batch analysis to normalize scores. Vectorization: Employed Sentence Transformers (all-MiniLM-L6-v2) to create text embeddings. Clustering: Used Scikit-learn's KMeans to identify narrative groups. Summarization: Utilized Hugging Face Transformers pipeline (DistilBART/LED) for generating summaries. Concurrency: Used Python's threading and queue for background processing. Visualization Prep: Used Scikit-learn's TSNE and prepared data for Plotly. Frontend (Conceptual): Designed for a simple search bar leading to a dashboard (Streamlit/React). Environment: Used uv for package management.
Challenges we ran into Trustworthiness Model Speed & Cost: Implementing the detailed WCRS scoring using the OpenAI API for every post proved to be a significant performance bottleneck due to network latency and API call duration. This also has cost implications outside a hackathon context, but with a large scae server or cluster this is mitigated.
Concurrency: Managing WebSocket updates from background threads processing potentially slow OpenAI calls was complex.
API Rate Limits: Balancing fetches from X/Reddit with potentially numerous calls to the OpenAI API required careful consideration.
Prompt Engineering: Designing the detailed system prompt for the OpenAI API to accurately extract and quantify the WCRS features in the correct JSON format required significant iteration.
Environment Setup: Standard challenges with interpreter paths, dependencies, execution policies, and SSL errors.
Data Consistency: Ensuring data flowed correctly and maintained the expected format between connectors, the WCRS scoring module (truth.py), and the main analysis pipeline (whack_er_bot.py).
Accomplishments that we're proud of Implementing a highly sophisticated, multi-dimensional WCRS trustworthiness model based on academic principles, going beyond simple heuristics. ✨
Successfully integrating multiple complex external APIs (X, Reddit, OpenAI) and ML models (SentenceTransformers, KMeans, Summarizers).
Designing a modular system capable of fetching data, scoring it with a complex LLM-based process, clustering, and summarizing narratives.
Calculating "authentic strength" for narratives weighted by the detailed WCRS trustworthiness scores.
What we learned The significant performance trade-offs involved when using powerful LLMs like GPT for detailed, per-item feature extraction in a real-time context. 🐢
Advanced prompt engineering techniques for instructing LLMs to perform complex analysis and return structured JSON output reliably.
Methods for combining programmatic analysis with LLM-based analysis.
Deeper insights into measuring information credibility using multi-faceted approaches (Authority, Evidence, Objectivity etc.).
Techniques for managing complex data flows and potential bottlenecks in a multi-stage analysis pipeline.
What's next for truthy Optimize Trustworthiness Scoring: Explore batching OpenAI requests, using faster/cheaper models (like specialized fine-tuned models), implementing smarter caching, or developing programmatic approximations for more WCRS features to reduce reliance on the LLM bottleneck.
Refine Frontend: Build out the interactive dashboard.
Improve Concurrency: Implement a more robust method for handling background tasks and WebSocket updates (e.g., FastAPI background tasks).
Add Connectors: Integrate news APIs or web scrapers, potentially adapting the WCRS model configuration for these sources.
User Feedback & Calibration: Allow users to provide feedback on trustworthiness scores to potentially fine-tune the WCRS model or weights over time.
Built With
- ai
- api
- clustering
- fastapi
- k-means
- multithreading
- natural-language-processing
- nextjs
- optimization
- python
- rest-api
- t-sne
- transformers
- typescript
Log in or sign up for Devpost to join the conversation.