Aegis-OS

• 

  Aegis-OS

Inspiration

Electronics inspection often happens under pressure. A technician may be holding equipment, checking a procedure, communicating with a team, and trying to preserve evidence at the same time. A visible issue such as a disconnected connector can be missed, while the reasoning behind a corrective action is often lost after the job is completed.

I built Aegis-OS around a simple question:

What would an inspection tool look like if it could observe the same evidence as the technician, point to the exact issue, guide the correction, require approval before taking action, and then verify the result?

The goal was not to build another chatbot. The goal was to create an operational inspection workspace where voice, visual evidence, spatial localization, procedures, approvals, and reporting work as one closed loop.

What it does

Aegis-OS is a voice-first, multimodal inspection workspace designed for electronics technicians, field-service teams, hardware laboratories, maintenance operations, and technical quality-assurance teams.

Its workflow is:

Observe → locate → explain → approve → act → verify → report

A technician can connect a camera or upload an inspection image. Aegis-OS captures a deliberate evidence frame rather than continuously uploading video. The frame is evaluated through a strict inspection contract that represents:

-Scene quality and whether the evidence is usable -Visible findings -Severity and confidence -Evidence supporting each finding -Normalized spatial coordinates -Recommended corrective action -Verification instructions -Procedure status -Human-verification requirements

Findings are rendered directly over the inspection image using spatial overlays. The interface then guides the technician through the next safe action.

For consequential actions, such as creating a maintenance incident, Aegis-OS does not allow an AI response to execute an operation automatically. It generates an exact-argument confirmation request and requires explicit human authorization.

After a correction, Aegis-OS requires a newer evidence frame. The system is designed to verify the new state rather than assuming that an instruction was followed.

At the end of the workflow, the technician can generate a checksummed PDF containing the findings, verification status, incident history, and inspection timeline.

How we built it

I supplied the product direction, target users, safety boundaries, electronics use case, approval requirements, design language, and closed-loop workflow. Codex helped translate those decisions into the application architecture, implementation, validation contracts, tests, documentation, and release audit.

The application is built with:

Next.js and React TypeScript Tailwind CSS OpenAI Responses API architecture OpenAI Realtime WebRTC architecture Zod runtime validation WebRTC and browser media APIs PDF-Lib Vitest Vercel

The visual-analysis layer uses a strict Structured Outputs schema rather than accepting unrestricted model text. The returned result is independently validated with Zod before it can update the interface or inspection state.

Spatial anchors use a normalized coordinate system from 0 to 1000:

[ x_{\text{screen}} = x_{\text{offset}} + \frac{x_{\text{normalized}}}{1000} \times w_{\text{rendered}} ]

This lets the application map findings correctly across different source images, screen sizes, aspect ratios, and letterboxed viewports.

The browser owns the live camera preview and captures an image only when an inspection is requested. API credentials remain on the server. Continuous video and raw audio are not stored by the application.

The system also includes:

Signed inspection sessions Scene-quality rejection Spatial overlay transformations Procedure progression Newer-frame verification architecture Exact action-confirmation tokens Internal maintenance incidents Optional signed external webhook delivery Session timelines and history Checksummed PDF reports Responsive desktop and mobile layouts Security and contract tests Original project-owned inspection assets Design decisions

The main interface is intentionally not designed like a conventional chat application.

The inspection image remains the primary workspace. Findings, confidence, instructions, system state, procedure progress, and approval controls are organized around the visual evidence.

I used a restrained technical visual language:

High-contrast dark inspection workspace Minimal cyan accent Clear severity states Sharp borders and compact operational typography No decorative AI orb No fake telemetry No fictional performance statistics No hidden automatic actions

The interface always distinguishes between observation, analysis, finding, verification, action approval, execution, and failure.

Challenges I faced

Building a true closed loop

Detecting an issue is only the beginning. The harder problem was representing the full operational sequence: finding the fault, explaining it, approving an action, checking newer evidence, and preserving the result.

This required stateful inspection sessions rather than one-off image responses.

Reliable spatial overlays

Model coordinates are relative to the original image, while the browser may resize, crop, letterbox, or mirror the image. A small mistake in the coordinate transformation can place an overlay over the wrong component.

I created and tested a normalized spatial contract that accounts for the rendered image dimensions and offsets.

Strict model output

A free-form response is unsuitable for an operational inspection interface. I needed predictable, bounded fields that the application could validate.

The project uses strict JSON Schema output plus independent runtime validation. During production testing, I encountered Structured Outputs compatibility issues involving schema unions and discriminator definitions. These were corrected and covered with regression tests.

Human control over actions

A model should not create external incidents or perform consequential operations solely because it generated a tool call.

I implemented confirmation tokens that bind the exact session, organization, finding, severity, description, expiry, and nonce. The application presents the effect to the user before execution.

Working within API-access constraints

Build Week provided Codex access rather than general OpenAI API credits. I therefore kept API availability and product truthfulness separate. The repository contains the OpenAI Responses and Realtime architecture, while the application reports provider or quota failures honestly instead of displaying fabricated results.

This reinforced an important product principle: an inspection system must never turn an unavailable analysis into a fake success.

Accomplishments I'm proud of

I am proud that Aegis-OS goes beyond image description.

The project combines:

-Real camera and image capture -Voice-interaction architecture -Evidence-backed structured findings -Spatial localization -Procedure state -Human-controlled actions -Newer-frame verification -Audit-style reporting -Security boundaries -A focused, production-inspired user experience

The repository also documents what is implemented, what was tested, what Codex contributed, what decisions I made, and which limitations remain. I chose truthful system behavior over disguising unavailable services.

What I learned

I learned that building a useful multimodal system is less about placing a model behind an interface and more about designing the contracts around it.

The most important lessons were:

1. Evidence must be first-class. A finding should identify what is visible, which frame supports it, and where it appears.
2. Confidence is not authorization. Even a high-confidence finding may still require human verification.
3. Verification needs new evidence. A corrective instruction does not prove that a correction occurred.
4. Structured output still needs independent validation. Model compliance cannot be the final trust boundary.
5. Failure states are part of the product. Permission denial, unclear evidence, provider failures, and integration failures must be visible and recoverable.
6. Codex is most valuable when human decisions are explicit. Clear product and safety boundaries allowed Codex to contribute deeply without replacing human judgment.

What's next for Aegis-OS:

The next production phase would add:

A deterministic local inspection engine for offline checks PostgreSQL or Supabase persistence with organization-level access control Durable confirmation replay protection Stored before-and-after evidence Additional electronics inspection packs Live-model accuracy and spatial-IoU evaluations A configured external incident-management integration Team collaboration and centralized audit history Optional smart-glasses and Web Serial support

The long-term vision is for Aegis-OS to become a resilient inspection operating system that can combine local deterministic checks, optional cloud intelligence, human approval, and verifiable evidence across field-service and technical operations.

Built with GPT-5.6 and Codex

GPT-5.6 was used meaningfully through the primary Codex development session to help design, implement, test, and review Aegis-OS.

Codex contributed to:

Application architecture OpenAI integration adapters Strict inspection schemas Spatial-coordinate transformations Verification contracts Action-authorization controls Security tests Original judge assets Release documentation Production debugging

I made the core human decisions: the problem to solve, target users, electronics focus, workflow, risk boundaries, design direction, approval policy, and product priorities.

Aegis-OS turns inspection from a one-time observation into a controlled, evidence-backed loop.

Built With

• ai
• api
• audio
• calling
• codex
• computer
• css
• function
• gpt-5.6
• mediadevices
• next.js
• node.js
• openai
• pdf-lib
• react
• realtime
• responses
• structured
• tailwind
• typescript
• vercel
• vision
• vitest
• webrtc
• zod