Clustr

Chart a path through X's universe with Grok. Discover sooner, Learn deeper, Understand your Universe.

Comment

Inspiration

We know the feeling of scrolling through X and seeing a lot of noise from the variety of our feed. Or sometimes, we wanted to learn more about a specific topic and follow experts in that field but didn’t know where to begin. For example, is everyone bullish on NVIDIA today, or are people freaking out about some political news? We wondered if we could use Grok to actually understand what's happening in real-time about specific topics instead of just drowning in an endless and randomized feed? So, we created Clustr.

What it does

Clustr lets you create custom "knowledge clusters" around topics and people you care about. Users type in keywords like "NVIDIA" or "election 2024" or “MrBeast and Jake Paul”, pick a category (Finance, Politics, Culture, or Anything), and Create a cluster. Through a multi-step pipeline, we pull hundreds of live posts from X and run them through Grok and a sentiment analysis algorithm to figure out what people are saying about those topics. Users see a clean personalized feed of posts that integrates into the existing X platform, plus analysis and charts that actually make sense of the chaos. There are real-time updates and analytics with Grok.

We also created a custom X/Twitter bot @heyClustr that users can use directly in the app to customize and use to create clusters easily. The bot replies in the thread on X and also creates a new cluster on our web app. Currently @grok within X does not have analysis ability or the ability to reason about live postings (just fetches details from posts), so through our platform, you can understand perception at a deeper level.

We analyze whether people are bullish or bearish for financial info such as stocks. For politics, it knows the difference between supportive and critical takes. And we go beyond just basic sentiment, extracting insights like conviction levels, price targets, quarterly earning predictions, trending narratives, etc with Grok.

How we built it

Frontend: Next.js + TypeScript. Our modern and user-friendly frontend leverages the power and flexibility of Next.js with TypeScript for end-to-end type safety and scalable UI composition. We designed the UI for modularity, scalability, and real-time responsiveness.

Backend: Python, FastAPI. At the heart of our backend lies a multi-stage, asynchronous data stream processor that pulls, transforms, and analyzes hundreds of posts per minute. Each incoming post traverses an advanced Natural Language Processing (NLP) workflow powered by Grok. We're constantly pulling from the X API, processing through Grok, and updating our database. Python powered the backend, taking data from X, analyzing it with Grok, storing the results in Supabase, and exposing everything to your Next.js frontend with API endpoints.

  • Automated Sentiment & Semantic Analysis: Every post undergoes multi-dimensional sentiment analysis using Grok, which is dynamically configured via category-specific LLM prompts. For example, in financial clusters, our system discriminates between nuanced bullish and bearish signals, while in political clusters, it discerns supportive, critical, or neutral tones. We further contextualize posts by extracting engagement metrics, performing signal aggregation, clustering, and anomaly detection to surface trending narratives and viral content.

X API and Grok The X API gives us hundreds real-time posts with all the engagement metrics, then Grok analyzes them with category-specific prompts - so it knows to look for bullish/bearish sentiment in finance posts versus supportive/critical in politics. We used streaming to consistently get and monitor the latest posts for the topic of interest.

  • We used multiple X API endpoints: Recent search, user lookup, tweet fields, and used authentication too to create a custom X/Twitter bot @heyClustr that users can customize and use to create clusters. For this, we used process webhooks to make webhook events that automatically trigger cluster updates (create a new cluster). The bot also replies directly to your Twitter post!

Database: Supabase All processed and enriched data is stored in Supabase, leveraging its scalable Postgres foundation and real-time syncing features. This ensures that the most relevant insights, charts, visualizations, and cluster updates are always instantly available to the Next.js frontend via robust RESTless endpoints.

Challenges we ran into

Streaming was a huge challenge. Coordinating between pulling live data from X, processing it through Grok, and updating the frontend without everything falling apart or having a lengthy latency took a lot of time to figure out. Since we were synthesizing hundreds of posts, X API rate limits were a restriction and we had to get creative with batching and queuing with data. Lots of error handling such as when making sure sentiment analysis actually matched back to the right posts. Understanding the developer tools for a social platform such as X was also a unqiue and challenging experience. Using X webhooks made authentication slightly confusing because there were many different tokens.

Accomplishments that we're proud of

  • Clustr actually works with real data, nothing is mocked and everything is live from X API, Grok, and existing Twitter feeds, a challenging task to accomplish.

  • Sophisticated Grok integration - we're not just asking if sentiment is positive or negative, we built domain-specific prompts that adapt based on whether you're looking at finance, politics, or culture topics and organized the info well with charts and summaries. We prioritized real user experiences and talked to fellow competitors who enjoyed learning through the platform.

  • The quality of the insights: the information we extract and organize is relevant and valuable. For example, one of us knew nothing about investing in stocks, but here we can accurate financial info about any question we have. Clustr is an application that we would all genuinely use as X users. We believe that X holds a vast amount of information that can be easily tapped into with Clustr.

What we learned

  1. Building on top of a social media platform has different challenges than working with any other API, and it was interesting and a great challenge to get to work with data that is created and seen by real people.

  2. We were surprised by how many valuable insights could be gleaned from an aggregation of social media content and user interactions that you couldn't get by just looking at individual posts or just going through your feed once.

What's next for Clustr

  • More sophisticated filtering - let users filter by verified accounts, select specific hashtags, have increased customization etc.
  • Predictive ability - using historical sentiment (caching older posts) to predict where sentiment is heading
  • Team features - let people collaborate on clusters and share insights.
  • Actionable clusters - let people build clusters that actually do something

Built With

Share this project:

Updates