AI Adoption Metrics

Feeding chat history to understand how, why and when teams are interacting with LLMs. We built this generic solution so managers can understand the AI usage and accelerate AI adoption in their teams

Comment

Inspiration

We came to SF for OpenAI DevDay but… our startup’s tech stack uses MongoDB, LangChain, and Claude, so when we saw this hackathon we just HAD to participate!

We wanted to build a feature which we could later implement in our startup.

We are co-founders of Team-GPT: a collaborative chat environment, where people, organizations, and corporations use LLMs specifically for work.

Our users have sent 1.5M+ messages in 130K+ chats and really value the adoption reports inside the platform.

Improving the adoption reports was precisely our focus today.

What it does

AI Adoption Metrics is an open-source project that analyzes conversations from different chat interfaces like ChatGPT, Team-GPT, and Claude.

The end result is a 'management-ready summary' of how users have used the LLM for a given time period - how, when, why, based on the actual chats.

How we built it

We used typescript, AWS lambda, LangChain, Claude API and made it deployable through an API so other companies can also use the project. We even managed to deploy this to staging on Team-GPT (and test this with our own data).

We also bought a domain (aiadoptionmetrics.com) and created a WP website to make this a complete project.

Challenges we ran into

  • When the chats are too long (e.g. for bigger teams) we needed to chunk the data accordingly without losing context.
  • Testing lambda functions locally
  • Producing a working WP website within the day

Accomplishments that we're proud of

We are very proud that we managed to make this an open-source project which can be reused by others.

Interestingly, LLM users in interfaces without collaboration (e.g. ChatGPT), can analyze their own data (a use case we discovered after creating the project).

What we learned

We imporved our understanding of how to use chunking and processing of data for large quantities of text and we've deepened our AWS knowledge.

What's next for AI Adoption Metrics

AI Adoption Metrics is going to prod on Team-GPT for sure, so all our users can enjoy it. Making this 'feature' an open-source project opens the door for others to improve their understanding of LLMs. We really hope we won't be the only contributors to this project.

In the future, this approach could be used by any SaaS company which has UGC (user generated content) that is rich in text, e.g. Notion, Google Docs, etc.

Built With

Share this project:

Updates