PRDetective

Amplifinancials Results part 1
Amplifinancials Results part 2
whiteboarding
How it works

Inspiration

As a PM, my pain points are many. Finding all the critical insights to incorporate into my doc is one that AI can solve and save me a lot of time.

Additional Pain Points:

Writing specs takes a lot of time
Have to wait for data analysts to help prepare data
Engineers ask me if we really need to make things so I need data to back it up

What it does

PRDetective scan product specs for insights that should be covered, auto-generate key data questions, convert them into SQL queries, query results, and return them to the user to inform their spec.

How we built it

One sentence description We use two API calls to chatgpt to (1) ask critical product questions and (2) to generate SQL to answer those data questions. We use the amplifinancials dataset as a sample customer.

The insight: ChatGPT (gpt-3-turbo) is surprisingly good at doing two things:

Given a textual document, ask questions about it
Given a description of your database, come up with SQL query that solves it

*Prompt engineering."

Asking critical product questions - We say `Given the following product spec: [load spec here], what are the 3 relevant product questions we should answer?" **
Coming up with SQL - We say, `Given the following MySQL schema: [dump table data], what is the SQL query that answers the [product question from #1]. Keep it simple."

Challenges we ran into

We had three major issues: Problem #1. When asking ChatGPT for insightful questions, it would often ask very broad questions such as "Consider what is the addressable market size?". Unfortunately, this is both a very difficult question to contextually answer as well as something we didn't have the data for. Solution #1: We know that there are certain questions that can be answered, so we built a question bank of 100 questions that we could actually answer with the data (e.g. a bank of questions that data scientists have already answered). We can then use ChatGPT to search for the most relevant questions and surface them. In the prompt, we can say, ...choose the 3 most relevant questions from: [question bank]

Problem #2: Chat GPT hallucinates and makes complicated SQL query that doesn't work Solution #2: We simply append a "keep it simple" at the end of the prompt :chefs_kiss:

Problem #3: Chat GPT understands the schema and doesn't understand the data in the dataset Solution #3: We help ChatGPT by injecting the unique names of all the events in the prompt

Problem #4: Chat GPT isn't great at asking questions or coming up with SQL! Solution #4: We allow the user to modify the query and to rerun it! #Feedback

Accomplishments that we're proud of

We were able to build in some key features including:

Generating charts from queried data
Query fields are editable if user wants to improve generated SQL queries
Feedback loop (User can re-scan the PRD to refresh generated questions and queries)
Productboard insights (Imported productboard data in to quantify and summarize customer requests)

We had a fairly smooth and drama-less schedule because we laid out the biggest blockers, tackled them first, laid out timelines and decided on the next steps together as a group. #Teamwork

What we learned

How to integrate OpenAI, prompt engineering, PM pain points for some of us AI isn't good enough to replace humans but it's good enough to come up with good questions to get started.