Elly — AI Product Mentor

Your AI product ally, helping PMs decide whether, when, and how to build with AI.

Comment

Inspiration

As a Product Manager, I make many decisions every day. Some are small, while others can affect weeks of work across product, engineering, and QA.

In one past project, a requirement moved directly into development without enough validation or alignment. The product team spent significant time designing and building it. When the feature was nearly ready to launch, the requesting stakeholder changed direction and said it was no longer needed. The team had completed most of the work, but the product never delivered value.

That experience stayed with me. The problem was not that the team could not execute. We executed the request successfully, but we did not pause early enough to confirm whether it was still the right problem, whether the assumptions were valid, and what evidence would justify the investment.

I originally wanted to create something that could help PMs examine everyday product decisions before development begins.

I also noticed that planning AI products introduces a new set of questions. PMs need to communicate clearly with developers and QA about when AI should be used, how much error is acceptable, how AI output should be evaluated, and when a human should step in. AI can generate ideas and outputs, but PMs still need to understand, challenge, and evaluate them.

I believe PM involvement becomes more important—not less important—as teams build more AI products.

That is why I built Elly. The name sounds like “ally,” because I wanted it to feel like a product-thinking partner that helps PMs challenge assumptions and make better decisions before the team commits to building.

What Elly does

Elly helps PMs think through a product idea before committing engineering resources.

The current application provides three product workflows:

Challenge Idea

PMs can describe a product idea and receive questions that challenge the underlying problem, target user, assumptions, risks, and MVP scope.

Product Decision

When a PM is choosing between options or deciding what to prioritize, Elly evaluates the trade-offs and suggests a smaller validation step before making a larger investment.

AI Planning

I also built an interactive AI Planning prototype for PMs who are planning AI features. It collects:

  • The product requirement
  • Product goal
  • Primary users
  • Acceptable error risk
  • Available knowledge sources

The result demonstrates how Elly can present AI Readiness, whether the problem is suitable for AI, possible use of Workflow, RAG, Agent, Memory, and Tools, major risks, and the next questions a PM should answer.

The AI Planning analysis currently uses frontend mock data. I built it this way to validate the workflow and information structure before implementing the Planning Agent and backend API.

How I built it

I built Elly with a FastAPI backend and a dependency-free HTML, CSS, and JavaScript frontend.

The conversational flow works like this:

User message
→ FastAPI chat endpoint
→ Mentor mode routing
→ Specialized product prompt
→ Relevant conversation context
→ OpenAI Responses API
→ Mentor response

The Mentor Orchestrator uses transparent routing rules to select one of three modes:

  • Challenge
  • Decision
  • General product mentoring

Each mode has a specialized prompt and response structure. Instead of asking the model to behave like a general chatbot, I defined how it should challenge assumptions, communicate trade-offs, control response length, and recommend a concrete next action.

I also implemented conversation memory with SQLite. Elly stores the conversation and retrieves a compact combination of recent and keyword-relevant messages. This allows the Mentor to retain useful product context without sending the entire conversation every time.

I used Codex throughout the build to explore the product direction, compare implementation options, create and revise code, inspect the repository, and test the user flows. I still made the scope and product decisions myself, reviewed the generated implementation, and simplified ideas that were too large for the hackathon.

I built the project with Codex using GPT-5.6. GPT-5.6 helped turn the product vision into a runnable FastAPI application, refine the specialized Mentor prompts and routing, review the SQLite memory approach, and test the product flows. I made the final product, scope, and implementation decisions. GPT-5.6 was used through Codex to build and review Elly; the running application separately uses the OpenAI Responses API with the gpt-5 model.

Challenges I ran into

Creating a Mentor that does not always agree

The first challenge was prompt behavior. A helpful AI assistant often tries to be supportive, but a useful product mentor must also challenge weak assumptions.

I had to define a clearer role and response contract: identify the most important uncertainty, explain the risk, and recommend a specific validation step instead of automatically saying that an idea is good.

Keeping responses useful on a chat screen

Early responses could become long and report-like. That made them harder to use during an actual product discussion.

I adjusted the prompts to prioritize one conclusion, a small number of high-impact challenges, and one next action. Longer analysis is available only when the user asks for it.

Retaining context without creating a complex memory system

A product mentor needs to remember previous goals and decisions, but long-term memory could easily expand the project scope.

I created a small SQLite-based Memory v0.1 that focuses on conversation context rather than personal user information. It retrieves recent messages and simple keyword-relevant history.

Controlling scope

I originally saw opportunities to build AI Planning, AI Spec, AI Evaluation, integrations, and more. Trying to implement everything would have made the core experience weaker.

I kept Challenge Idea and Product Decision as the working AI flows, then created AI Planning as an interactive frontend prototype. This allowed me to test the product direction while being transparent about what has and has not been implemented.

Accomplishments I’m proud of

I am proud that Elly is more than a single prompt connected to a chat box.

During the build, I completed:

  • A working FastAPI Mentor chat API
  • Specialized Challenge, Decision, and general Mentor modes
  • Transparent message routing
  • Structured prompt behavior for product thinking
  • SQLite conversation memory and relevant-context retrieval
  • Traditional Chinese and English response support
  • A responsive frontend for the Mentor workflows
  • An interactive AI Planning prototype
  • A product journey that can grow from idea challenge to AI planning and evaluation

Most importantly, the implementation reflects the product principle behind Elly: AI should help PMs improve their judgment, not only produce documents faster.

What I learned

I learned that designing an AI product requires more than choosing a model and writing a prompt.

The interaction structure matters. The model needs a clear role, constraints, response format, and definition of what a useful next step looks like. I also learned that the quality of an AI Mentor depends as much on the questions it asks as on the answers it gives.

Working with Codex taught me a similar lesson. Better collaboration came from giving it clear context and constraints, reviewing what it produced, and making deliberate decisions about what to keep, change, or remove.

From a product perspective, I learned to separate the long-term vision from the smallest useful implementation. Building the AI Planning prototype helped me explore the larger opportunity without pretending that every part of the vision was already complete.

What’s next for Elly

The next step is to turn AI Planning from a frontend prototype into a working product flow.

I plan to add:

  • A structured AI Planning API and Planning Agent
  • Dynamic AI Readiness and architecture recommendations
  • AI Spec generation for prompts, knowledge, tools, output schemas, and fallback behavior
  • AI Evaluation planning with test cases, datasets, rubrics, hallucination checks, and safety checks
  • Evaluation with real PM scenarios
  • Product and company context for more relevant recommendations

The longer-term goal is for Elly to support the complete AI product journey: deciding whether an idea is worth building, planning a controllable AI solution, defining its specification, and determining how to evaluate it.

Built With

Share this project:

Updates