AWS

Inspiration

What it does

How we built it

Challenges we ran into

Accomplishments that we're proud of

ScribeBA — Context-Native Autonomous Agile BA & Incident Console

"Every claim must be traceable to its source. Every unanswered question must be marked Blocked, never invented."

💡 Inspiration

In modern software development, critical product and architectural decisions often emerge from chaotic Slack and Discord threads. Teams debate requirements, explore edge cases, and negotiate technical trade-offs. Yet once the conversation ends, these valuable decisions are easily buried in the scroll.

When teams use conventional LLMs to summarize these discussions, a dangerous pattern emerges: discussion is mistaken for consensus, and missing information is replaced with AI-generated assumptions. A casual suggestion such as "we could auto-refresh tokens" may be transformed into an official requirement, while unresolved questions about session timeouts or rate limits are silently filled with generic defaults.

We built ScribeBA to eliminate this guesswork.

ScribeBA is a context-native, autonomous Agile Business Analyst agent designed to work directly within team communication channels. Rather than inventing answers, it enforces evidence-based traceability: every requirement is linked to its source message and speaker, unresolved ambiguities are explicitly marked as Blocked, and changes are sent to task trackers or infrastructure systems only after explicit human approval.

Our goal is simple: make AI-generated work actionable without sacrificing accuracy, accountability, or human control.

⚙️ What It Does and How It Works

ScribeBA operates on a three-axis verification model:

Evidence Axis: Each claim is classified as Verified, Inferred, Assumed, or Blocked.
Decision Axis: Each decision is classified as Proposed, Approved, Superseded, or Unresolved.
Readiness Axis: Each requirement is classified as Draft, Needs Review, Blocked, or Ready.

These classifications help distinguish confirmed facts from interpretations, proposed decisions, and unresolved questions.

A requirement cannot be published unless it meets the readiness requirements, contains no blocked claims, and receives explicit human approval:

[ P_{\text{publish}} = \mathbb{I}(\text{Readiness}=\text{Ready}) \times \prod_{i=1}^{N} \mathbb{I}(\text{Claim}_i\neq\text{Blocked}) \times \mathbb{I}(\text{HumanApproval}=1) ]

Here, (\mathbb{I}) is an indicator function that returns 1 when a condition is satisfied and 0 otherwise. The publication gate therefore evaluates to 1 only when all three conditions are met.

To assess user story quality before handoff, ScribeBA calculates a composite INVEST score:

[ S_{\text{INVEST}} = \sum_{k\in{I,N,V,E,S,T}} w_k s_k, \qquad \sum_k w_k=1 ]

The score ranges from 0 to 100 and evaluates the six INVEST principles: Independent, Negotiable, Valuable, Estimable, Small, and Testable. The weights (w_k) represent the relative importance assigned to each criterion, while (s_k) represents its normalized score.

Core capabilities include:

Agile Specification Engine: Converts multi-person chat threads into structured user stories, Given-When-Then acceptance criteria, clarifying questions, and SHA-256 integrity hashes that help detect subsequent content changes.
Evidence-Based Requirement Traceability: Links extracted requirements to their source messages, helping reviewers verify who said what and distinguish proposals from approved decisions.
AIOps Incident Console: Applies the same evidence-grounded approach to infrastructure operations. ScribeBA reads Kubernetes cluster state and Prometheus metrics, including requests per second (RPS), 95th-percentile latency (P95), and HTTP 5xx error rates.
Controlled Remediation Proposals: Uses declarative allowlists to constrain proposed infrastructure actions and Kubernetes resourceVersion checks to detect stale resource state before approval and execution.

🛠️ How We Built It

ScribeBA was designed as a multi-tier, fault-tolerant agent architecture combining conversational AI, evidence retrieval, structured extraction, and human-controlled execution.

1. High-Speed AI Core — Nebius AI Studio

We integrated Nebius AI Studio through an OpenAI-compatible endpoint, using Qwen/Qwen3-30B-A3B-Instruct-2507 as the primary high-speed extraction engine. The model converts complex conversational transcripts into structured JSON specifications, supporting downstream validation and requirement generation.

2. Multi-Tier Model Fallback

We designed a fallback router across three service tiers: Low, Medium, and High. When a primary OpenRouter request encounters a rate limit (HTTP 429) or a payment or balance-related error (HTTP 402), the system can attempt an alternative provider or model.

The intended fallback sequence is:

[ \text{OpenRouter} \rightarrow \text{Google Gemini} \rightarrow \text{Nebius Qwen} \rightarrow \text{Local Deterministic Engine} ]

The Gemini tier supports the configured Flash and Pro models. The final deterministic engine provides a non-generative fallback for supported operations when external model services are unavailable.

3. Exa Neural Search for External Grounding

We integrated Exa's semantic search and highlights capabilities to retrieve relevant external references and technical guidance. These sources can help contextualize requirements involving OAuth 2.0, RFC 6749, security practices, and session lifecycle design.

External references provide supporting context; they do not automatically turn an unapproved proposal into an accepted requirement.

4. Security and Personally Identifiable Information (PII) Redaction

We implemented pre-processing filters to identify and mask sensitive information, including supported patterns for API tokens, AWS credentials, payment card numbers, IP addresses, and email addresses, before outbound text is sent to external model providers.

5. Frontend Web Studio

The frontend is built with React 18, Vite, and Tailwind CSS. It provides real-time operational telemetry, Slack workflow triggering, ClickUp task synchronization, and interactive Kubernetes remediation diff reviews.

6. Multi-Platform Ingestion

ScribeBA supports multiple ways to receive team discussions and trigger workflows:

Slack through Bolt Socket Mode and the Slack Web API.
Discord through webhooks.
Telegram through long polling.
An interactive terminal interface through interactive_cli.py.

These integrations allow the system to collect context from different communication channels and transform it into structured, reviewable outputs.

🧗 Challenges We Faced

1. Overcoming the LLM "Helpfulness Trap"

LLMs are optimized to generate useful responses, which can lead them to fill gaps with plausible but unverified information. We addressed this challenge through constrained prompts, structured extraction schemas, and explicit handling of unresolved questions.

Instead of forcing the model to provide an answer, ScribeBA can classify incomplete requirements as Blocked and surface the missing information for human review.

2. Robust JSON Extraction Across Different Models

Claude, Qwen, and Gemini may return structured data with different formatting conventions, Markdown fences, or explanatory text surrounding JSON.

We developed a resilient parsing utility, parse_model_json, that extracts candidate JSON content from model responses before parsing. This improves compatibility across providers and reduces failures caused by inconsistent response formatting.

3. Preventing Stale or Unsafe Infrastructure Changes

Applying infrastructure changes based on outdated cluster information can introduce operational risks.

To address this, proposed Kubernetes remediations capture the target resource's uid and resourceVersion. Before execution, the system can verify that the resource still matches the state reviewed by the human operator. If the resource has changed, the proposal is rejected or requires a fresh review rather than being blindly applied.

🎓 What We Learned and Key Takeaways

1. Missing Data Is Not Evidence of System Health

If a Prometheus metrics connection fails, an agent must not interpret the absence of telemetry as proof that the system is healthy. ScribeBA treats unavailable monitoring data as Blocked — telemetry unavailable, preserving the distinction between a healthy system and an unobservable one.

2. Open-Weight Models Expand the Design Space for Agentic Systems

Integrating Qwen through Nebius AI Studio demonstrated how open-weight models can support structured extraction workflows with flexible deployment options. Model quality, latency, cost, and reliability must still be evaluated against the requirements of each workload.

3. Human-in-the-Loop Design Can Accelerate Delivery

Human approval does not have to become a bottleneck. By presenting source-message citations, explicit decision states, and readable infrastructure diffs, ScribeBA helps reviewers focus on the evidence needed to approve or reject a change instead of reconstructing decisions from long conversation histories.

4. Verifiability Is a System Property, Not Just a Prompting Technique

Reliable AI workflows require more than instructing a model not to hallucinate. They need explicit state transitions, source traceability, validation rules, integrity checks, and execution gates that enforce the intended workflow.

🚀 What's Next for ScribeBA

1. Bidirectional Jira and Linear Synchronization

Extend integrations beyond ClickUp to support two-way synchronization, backlog refinement, and requirement updates across enterprise project management platforms.

2. Automated End-to-End Test Generation

Explore using OpenAI Codex or GPT models to convert verified Gherkin acceptance criteria into executable Playwright or Cypress tests, with human review before integrating generated tests into CI/CD pipelines.

3. Multi-Cluster GitOps Reconciliation

Integrate remediation proposals with Argo CD and Flux workflows, enabling operators to review infrastructure changes through pull requests and maintain an auditable GitOps-based deployment process.

4. Stronger Evaluation and Auditability

Develop systematic evaluations for requirement extraction accuracy, evidence attribution, ambiguity detection, INVEST scoring, and safe remediation behavior. This will help quantify the reliability of ScribeBA as the system expands to more teams and infrastructure environments.

Closing Statement

ScribeBA turns conversations into accountable action.

By combining context-aware AI, evidence traceability, explicit uncertainty handling, and human-controlled execution, ScribeBA aims to make autonomous Agile analysis and incident response more reliable, transparent, and practical for real-world engineering teams.