PromptFlip

Inspiration

Everyone on our team uses AI every day, and we noticed the same pattern in ourselves and the people around us: we type something vague, get a generic answer, correct it, correct it again, and eventually get what we wanted. The AI quietly absorbs our bad habits, and nobody ever gets better at asking.

Most AI tools try to align the model to the user. We wanted to try the reverse: an assistant that answers first, like any normal chat, but occasionally (and only when it would really help) shows you how to get a better answer next time. We called the idea Reverse Alignment, and the product PromptFlip.

For teams there's a second motivation: vague prompts cost money. Every clarifying round trip is another paid request, and every follow-up re-sends the whole conversation. Managers have no visibility into how well their team uses AI, only the bill.

What it does

Answers first, coaches second. You get your answer immediately. Afterwards, a coach reviews the exchange and either stays silent (the default) or offers one short, specific tip, such as "saying who the plan is for and any deadline lets me size the steps to fit."
Prompt strength. Each prompt gets a simple weak / moderate / strong rating, shown as a small bar under the message. A weak or moderate rating always comes with a tip explaining why.
Asks before guessing. If a prompt is too thin to answer well ("React state", or "why does my code crash?" with no code), PromptFlip asks one clarifying question before sending it.
Learns your style. If you keep correcting answers ("shorter", "use bullets"), it offers to remember that preference and adapts future answers.
Manager dashboard. Average prompt quality, how often requests are clear first time, how many messages it takes before a request is clear, the cost of vague prompts, and how quality changes over time, per person and for the whole team.

How we built it

We started from the Vercel AI Chatbot template (Next.js, React, the AI SDK, Postgres via Drizzle) so we could spend our six hours on the coaching layer instead of rebuilding chat. Answers and coaching run on OpenAI's gpt-6-luna.

The coach never blocks the answer. It runs as a separate request after the answer finishes streaming. If it fails or times out, you simply see no tip.

Scoring prompts. The coach scores each prompt on four principles: a clear task, relevant context, stated success criteria, and efficiency. Each scores 2 (clear), 1 (missing, but the answer was fine anyway) or 0 (missing and it visibly hurt the answer). A 0 requires evidence from the answer, such as hedging, a guess, or padding. The model produces the scores as structured output, but plain code turns them into a tier, so the same scores always give the same rating. For the dashboard, tiers map to $s \in {0, 1, 2}$ (weak, moderate, strong), and team quality is the mean:

$$\bar{s} = \frac{1}{N}\sum_{i=1}^{N} s_i$$

Measuring cost. For every answered prompt we count tokens ourselves and store an estimated cost:

$$\text{cost} = \frac{T_{\text{in}}}{10^6}\, p_{\text{in}} + \frac{T_{\text{out}}}{10^6}\, p_{\text{out}}$$

where $T_{\text{in}}$ includes the new prompt plus all earlier messages in the chat, because the model re-reads the whole conversation every turn.

Clarity metrics. A request runs from its first message until the AI can answer it. "Clear first time" is the share of requests answered on the first message, and "prompts until clear" is the average number of messages per request.

What we learned

Silence is a feature. A wrong or generic tip is worse than no tip. Most of our design effort went into deciding when not to say anything: skipping simple and sensitive messages, cooling down after a tip, and requiring evidence before criticising a prompt.
Long chats are where the money goes. If each turn adds roughly $t$ tokens, turn $k$ re-sends about $(k-1)\,t$ tokens of history, so a chat of $n$ turns costs on the order of $$\sum_{k=1}^{n}(k-1)\,t = \frac{n(n-1)}{2}\,t$$ input tokens: quadratic in the length of the conversation. The cheapest habit to teach is simply start a new chat when the old context isn't relevant.
LLM judges need structure. Asking a model "is this a good prompt?" gives inconsistent answers. A fixed rubric, required evidence, and a deterministic tier made the ratings stable enough to show to users.

Challenges

Coaching without nagging. Getting the tone right (helpful, never judgemental, never a grade after every message) took several iterations of the coach's instructions and a small set of labelled test prompts to check against.
Three people, one branch, six hours. We split the work into clear systems with owners and a single shared type contract, and synced constantly to avoid stepping on each other.
Changing models mid-hackathon. We moved the whole app between model providers during the day, which meant reworking prompts, structured outputs and our cost pipeline under time pressure.
Honest metrics. Several of the numbers we wanted, like real token counts, live inside parts of the system we couldn't block or slow down, so we estimate them alongside the chat instead. The dashboard currently runs on a simulated 12-person team while the live pipeline fills up.