agentic_architecure

About the Project

The Spark of Inspiration 💡

The idea for Shipyard came from a frustrating pattern I kept witnessing: brilliant engineers with amazing product ideas getting stuck at the infrastructure planning stage. Some would spend weeks researching cloud services only to feel overwhelmed by the endless options and configurations. Others would jump straight into implementation without proper planning, leading to costly architectural mistakes down the road.

I realized there was a gap between knowing what you want to build and knowing how to build it reliably at scale. Traditional solutions were either too simplistic (basic templates) or too complex (enterprise consulting). What if we could create an AI assistant that could have the kind of conversation a senior infrastructure engineer might have with a colleague—adapting to their expertise level and progressively building a comprehensive plan?

The Learning Journey 🎓

Building Shipyard taught me several crucial lessons about AI system design:

1. Context is Everything

Initially, I planned simple sequential agents, but I quickly discovered that context sharing between agents was the make-or-break feature. Users would mention they're using Railway in the first conversation, only to have the business agent ask about cloud providers later. The solution required building an enhanced context management system where each agent receives the complete conversation history from previous agents, with LLM automated summaries optimized to our use-case and the agent.

2. The Expertise Adaptation Challenge

One of the most interesting technical challenges was creating truly adaptive questioning. The system needed to:

Assess both stated expertise ("I'm intermediate") and demonstrated expertise (how they actually describe their project)
Dynamically adjust question complexity in real-time
Provide gentle explanations without being condescending

This led to a dual-model approach: o3 for nuanced conversation understanding and GPT-4o for fast operations like follow-up detection.

How I Built It 🔧

Architecture Decisions

Model Selection Strategy: I chose a hybrid approach leveraging OpenAI's latest capabilities:

Primary Operations (o3 and o3-mini): Main agent conversations, document generation, comprehensive summarization
Fast Operations (GPT-4o): Follow-up detection, skip detection, expertise assessment

The reasoning? Complex conversations benefit from o3's enhanced reasoning capabilities, while simple binary decisions (like "does this need a follow-up?") can be handled quickly and cost-effectively by GPT-4o.

State Management Architecture

The trickiest part was designing a state management system that could handle three different types of data:

$$\text{State} = {\text{Chat History}, \text{Application State}, \text{Summaries}}$$

Where:

Chat History: Natural conversation flow within each agent
Application State: Cross-agent context and incremental document building
Summaries: AI-extracted key information after each pillar completes

state = {
    "chat_history": {
        "business": [{"role": "assistant", "content": "How many users?"}]
    },
    "state": {
        "user_profile": {"expertise_level": "intermediate"},
        "current_document": {"architecture": "..."},
        "previous_pillars_completed": ["profiler", "business"]
    },
    "summaries": {
        "profiler": {"expertise": "intermediate", "project_type": "startup"}
    }
}

The Agent Flow Design

I implemented a pillar-based architecture where specialized agents handle different aspects:

graph TD
    A[Profiler Agent] --> B[Business Needs Agent]
    B --> C[App Needs Agent] 
    C --> D[Tribal Knowledge Agent]
    D --> E[Best Practices Agent]
    E --> F[Document Generator]
    F --> G[Review Loop]

Each agent builds upon the previous ones' context, preventing duplicate questions and ensuring consistency.

The Challenges Faced 🚧

1. The Follow-up Detection Problem

Challenge: How do you know when a user's answer needs clarification without over-engineering?

Solution: I built a fast operation using GPT-4o that analyzes user responses for indicators of confusion, uncertainty, or incompleteness:

async def needs_follow_up_fast_operation(user_answer, agent_question):
    prompt = f"""
    Analyze if this user response indicates confusion or need for clarification:

    Agent Question: {agent_question}
    User Response: {user_answer}

    Respond with only "YES" if follow-up needed, "NO" if clear.
    """
    # Uses GPT-4o for sub-second response

2. The Expertise Calibration Problem

Challenge: Users often mis-assess their own expertise level, leading to poorly adapted questions.

Solution: Dual assessment system:

Stated Level: What the user tells us ("I'm intermediate")
Gauged Complexity: What we observe from their responses ("higher than stated")

This allows the system to gracefully adjust question complexity without contradicting the user.

4. Document Generation Quality

Challenge: Early document outputs were generic and didn't reflect the nuanced requirements gathered during interviews.

Solution: Enhanced the Document Generator with:

Complete conversation logs (not just summaries)
Clear attribution of user vs. AI recommendations
Section-by-section building with cross-references
Industry-specific templates based on detected domain

Technical Innovations 🔬

Adaptive Prompting Algorithm

I developed a dynamic prompting system that adjusts not just content but conversation style:

$$\text{Prompt Complexity} = f(\text{Stated Expertise}, \text{Gauged Expertise}, \text{Domain Context})$$

Follow-up Limiting with Context

Instead of hard limits, I implemented contextual follow-up management:

Maximum 3 follow-ups per topic
Early termination if user shows confidence
Question complexity increases with follow-up count

Zero-Repetition Context Sharing

The biggest technical achievement was ensuring agents never repeat questions while maintaining natural conversation flow. This required:

Conversation deduplication across agents
Technology stack persistence
Semantic similarity detection for topics

What's Next 🚀

Building Shipyard taught me that the future of technical planning tools isn't about replacing human expertise—it's about amplifying it. The next iteration will focus on:

LangGraph Migration: For more sophisticated workflow management
Real-time Cost Analysis: Integration with cloud provider APIs for live pricing
IaC Generation: Direct Terraform/CloudFormation output from plans
Collaborative Planning: Multi-stakeholder input and approval workflows
Infra Validation: Using MCP to actually see what is deployed and its performance.

The core insight remains: good infrastructure starts with good conversations. Shipyard proves that AI can facilitate those conversations at scale, adapting to any expertise level while building comprehensive, actionable plans.

What started as a solution to a personal frustration became a deep exploration of conversational AI, context management, and adaptive user interfaces. Every challenge taught me something new about building AI systems that truly understand and adapt to their users.

Built With

agentic-pattern
openai
python
weave

shipyard