Working on business-facing AI systems, we repeatedly encountered two bottlenecks: \textbf{understanding operational context and preparing data that represents it}.

AI-assisted development has accelerated implementation. However, important requirements still live in people's experience: which differences matter, which exceptions are acceptable, and when an image does not contain enough evidence for a decision. Turning that knowledge into suitable images, labels, and acceptance criteria remains a substantial engineering task.

Agentic FDE connects business understanding with synthetic data preparation. Its broader focus is \textbf{mobility: vehicles, fleets, and mobile equipment operating within real business workflows}. Construction-equipment rental inspection is our first application.

\section*{What it does}

Users describe their business problem through a conversation and provide available reference photographs. The agent clarifies consequential questions, records assumptions, and creates a versioned specification covering required scenarios, permitted changes, protected features, business rules, and dataset conditions.

Our first application distinguishes \textbf{new damage, dirt, existing damage, and insufficient visibility}. Supported damage includes body-panel dents, shattered glass, flat tires, and bucket damage. Existing damage can be visible without being new; an obscured part remains inconclusive.

The output format includes paired images, structured observations, model-reviewed bounding boxes, provenance, assessment evidence, human-review status, and remaining coverage gaps.

\section*{Workflow}

[ \begin{gathered} \text{Understand business intent}\ \downarrow\ \text{Assess source data and identify gaps}\ \downarrow\ \text{Plan task-specific scenarios}\ \downarrow\ \underbrace{\text{Generate}\rightarrow\text{Validate}\rightarrow\text{Improve}}_{\text{Bounded, evidence-based iteration}}\ \downarrow\ \text{Review dataset coverage}\ \downarrow\ \text{Human review and export} \end{gathered} ]

The individual-case loop addresses failed or uncertain samples. A separate dataset-level loop identifies missing scenarios and comparison cases, then plans bounded additional work.

\section*{How we built it}

We implemented a \textbf{supervisor-worker architecture in Python}. The supervisor owns the shared business context; specialized roles receive the images, requirements, and schemas relevant to their task. Pydantic schemas constrain outputs, and requirements retain links to their source conversation messages.

\subsection*{Models and responsibilities}

\small\setlength{\tabcolsep}{4pt}\renewcommand{\arraystretch}{1.22} \begin{longtable}{@{}p{0.20\textwidth}p{0.30\textwidth}p{0.44\textwidth}@{}} \textbf{Stage} & \textbf{Model or technology} & \textbf{Role}\\hline\endhead Business conversation and specification & OpenAI \texttt{gpt-\allowbreak{}6-\allowbreak{}astra} & Clarify intent and compile a versioned contract with traceable requirements, policies, assumptions, and dataset conditions.\[5pt] Source inventory & NVIDIA \texttt{nvidia/\allowbreak{}Nemotron-\allowbreak{}3.5-\allowbreak{}Super-\allowbreak{}VL} through Nebius Token Factory & Inspect actual photographs for visible equipment, parts, existing marks, capture conditions, and suitability.\[5pt] Scenario planning & OpenAI \texttt{gpt-\allowbreak{}6-\allowbreak{}astra} & Map requirements to supported equipment, damage, viewpoint, background, and count settings.\[5pt] Normal-scene generation & OpenAI \texttt{gpt-\allowbreak{}image-\allowbreak{}2.5-\allowbreak{}sunburst} & Create a synthetic normal reference scene from the supplied photograph.\[5pt] Part localization and masking & \texttt{gpt-\allowbreak{}6-\allowbreak{}astra} and Meta SAM 2.1 Hiera Large & Locate a target part and segment it with a bounding-box prompt. The locator can optionally use Nemotron VL through Nebius.\[5pt] Defect editing & OpenAI \texttt{gpt-\allowbreak{}image-\allowbreak{}2.5-\allowbreak{}sunburst} & Generate the requested damage with a defect-specific prompt, editing mask, and full-image input.\[5pt] Normal-scene and contextual assessment & OpenAI \texttt{gpt-\allowbreak{}6-\allowbreak{}astra} & Compare actual images against scene requirements, preservation constraints, and business context.\[5pt] Blind observation and multiple-choice checks & NVIDIA \texttt{nvidia/\allowbreak{}Nemotron-\allowbreak{}3.5-\allowbreak{}Super-\allowbreak{}VL} through Nebius Token Factory & Observe the actual pair without its intended scenario or business policy.\[5pt] Individual and dataset improvement & OpenAI \texttt{gpt-\allowbreak{}6-\allowbreak{}astra} & Interpret recorded failures and coverage gaps, then propose supported, bounded next steps.\[5pt] Technical and dataset checks & Python, NumPy, Pillow, and deterministic rules & Check image validity, coordinates, preservation measurements, provenance, counts, repeated pairs, and source-family separation.\[5pt] \end{longtable}\normalsize

A Nebius-only reasoning option uses NVIDIA \texttt{nvidia/\allowbreak{}nemotron-\allowbreak{}3-\allowbreak{}super-\allowbreak{}120b-\allowbreak{}a12b} for text reasoning roles. The default premium configuration uses \texttt{gpt-\allowbreak{}6-\allowbreak{}astra} for those roles.

\subsection*{Image generation}

The active path is \textbf{GPT clean scene -> part localization -> SAM segmentation -> defect-specific mask -> Sunburst edit -> masked composition}.

SAM 2.1 runs on the GPU. Seeded shape functions construct different editing regions for dents, glass damage, and tires, then clip them to the segmented part. Sunburst uses high-quality, automatically sized image edits. Pillow composites the edited content through the control mask.

Each synthetic baseline retains its relationship to the original reference. Its equipment, viewpoint, background, normality, and preservation requirements are assessed before the case can qualify. No ground-truth defect segmentation is inferred from the editing mask.

\subsection*{Individual assessment}

\textbf{Blind visual assessment:} Nemotron VL produces structured observations of damage, novelty, visibility, identity, non-target geometry, and artificial patch boundaries. A separate multiple-choice call checks equipment, changes, structure, and surface transitions. The current bank is fixed and domain-specific; option order is deterministically shuffled from image hashes, and expected answers are withheld.

\textbf{Context-aware assessment:} \texttt{gpt-\allowbreak{}6-\allowbreak{}astra} sees the actual pair, versioned contract, intended scenario, applicable requirements, and processing records. It evaluates whether the requested situation is represented and protected features remain intact. Requirement applicability prevents a glass-fracture obligation from being imposed on a dirt-only case.

\textbf{Technical and consistency checks:} Python rules inspect file integrity, image dimensions, coordinates, localization, permitted-region preservation, and recorded capture transformations. Logical checks detect contradictions, such as reporting new damage while claiming no damage is visible.

Required failures reject a case; unresolved evidence produces a review state. Correctly generated insufficient-evidence cases may qualify as challenges. \textbf{Visible damage, novelty, business action, generation quality, and human approval remain separate fields.}

\subsection*{Improvement and dataset assessment}

The individual supervisor uses failure evidence to propose supported revisions. In the current path, revisions adjust supported scene inputs. Every prior attempt and its assessment remain available.

At dataset level, deterministic checks assess scenario counts, source-family separation, repeated pairs, matched comparison cases, and declared dataset requirements. A dataset supervisor plans missing cases, reuses an exact baseline for a comparison, or replaces a rejected coverage slot while retaining its history.

An optional, separately prompted Nemotron VL reference-inspector proxy demonstrates how an inspection system might respond to accepted cases. Its mistakes do not change synthetic-data quality decisions.

\subsection*{Application and execution}

The four-workspace console uses \textbf{FastAPI, Uvicorn, HTML, CSS, and JavaScript} for conversation, dataset browsing, image inspection, and specification lineage. SQLite and file-based journals preserve state, decisions, image hashes, and execution records. Generation runs in a separate worker process. Cost reservations and reusable API-response journals help prevent duplicate paid requests after interruption.

\section*{Challenges we ran into}

\textbf{Maintaining context across roles.} A single business prompt was insufficient for every stage. We introduced role-specific evidence packets backed by a shared, versioned specification.

\textbf{Separating generation intent from evidence.} Control masks constrain editing; observed defect labels require separate assessment. These are different artifacts with different meanings.

\textbf{Checking global coherence after local edits.} A plausible defect can coexist with an altered component or an artificial boundary. Assessment includes both the target region and surrounding equipment.

\textbf{Making improvement accountable.} We record the failed requirement, proposed action, and resulting evidence so that additional generation has an explicit purpose.

\section*{Accomplishments we're proud of}

We implemented an inspectable prototype connecting business conversation, structured requirements, image generation, layered assessment, dataset improvement planning, and human review. The workflow preserves uncertainty and explains exclusions and remaining gaps.

\section*{What we learned}

Synthetic-data quality depends on intended use. Visual realism, scenario correctness, and business relevance require different evidence. Individual-image assessment and dataset coverage are separate responsibilities, as are automatic assessment and human review.

Downstream model performance requires its own evaluation. Useful synthetic data comes with an explanation of what it represents and where its evidence is limited.