CLAY

Inspiration

We all know countless people, yet it’s often tough to tap into those connections or leverage their full potential. The data we need already lives in our social feeds (Instagram, for example) but querying it effectively has been next to impossible. That’s where Clay comes in.

What it does

Clay is a chatbot that understands your Instagram DMs, photos, and voice notes. Ask it anything “When did Alex send me that receipt?” or “What should I offer to my mum for her birthday?” and it’ll not only retrieve the info but also map out your network into an interactive graph.

You can click on any vertex in the graph and get a LLM-generated profile card for your connection!

How we built it

We are parsing multiple gigabytes of Instagram exports (messages, images, audio) in a few dozens of minutes. Text and metadata go to the Together.ai embeddings API, then into MongoDB Atlas as vectors. Photos are tagged via Together.ai’s vision API; voice messages are transcribed with Whisper (voice message integration is not completed yet).The chat interface runs on Together.ai’s chat API for natural, context-aware responses. RapidHub’s Instagram API supplies profile pictures, and PixiJS renders the social graph.

Challenges we ran into

Building a performant, zoomable network graph in PixiJS turned out to be trickier than expected. We flirted with switching to D3.js for ease, but ultimately pushed PixiJS to its limits—and ended up with a much richer, more dynamic visualization.

Accomplishments that we're proud of

This is our first RAG pipeline. Designing a retrieval-augmented generation workflow from scratch. End-to-end MVP for our first hackathon, from raw Instagram data to chat-based insights and graph visuals, all in one demoable app.

What we learned

We learned Together.ai, MongoDB APIs, and how to build a scalable RAG system: preprocessing large datasets, and generating embeddings. We also learned how to use PixiJS and WebGL.

What's next for CLAY

Adding more sources of data. Instagram is one of the possible source, but we can build an adapter for Discord, Telegram, Linkedin. We hope that this can make our business network even more queryable and understandable.

We can also add more modalities and API such as adding web search to the LLM, and of course, complete the audio voice parsing part!

Built With

• instagram
• javascript
• mongodb
• parser
• pixijs
• together.ai