SkyGuard AI

An AI simulation, evaluation and training platform that evaluates AI and teaches agents to make fast, policy-aware drone incident decisions, enforced by strict safety rules and human approval.

Comment

About the project

Inspiration

Most drone-safety systems focus on detection: identify an object, draw a tracking box, and raise an alert. But detection is only the beginning. The harder question is what an AI system should recommend next when evidence is incomplete, sensors disagree, authorisation is uncertain, and a wrong escalation could be as harmful as a missed threat.

That led me to build SkyGuard AI, a training, simulation and evaluation platform for sequential decision models.

The idea was inspired partly by my earlier work on Frontier, a long-horizon evaluation environment for tool-using AI agents. Frontier taught me that a model should not be judged only by whether it eventually reaches the right answer. We also need to understand the decisions it made, the evidence it used, the safety rules it followed, and the point where it failed.

I wanted to apply the same philosophy to physical-world decision support: evaluate models thoroughly before they are ever considered for use inside a downstream product such as TALON.

What it does

SkyGuard AI creates simulated, multi-step drone incidents and uses them to train and evaluate different decision policies.

The platform currently supports:

  • GRU-based behavioural modelling
  • Decision Transformer policies
  • Conservative Offline Reinforcement Learning using Discrete CQL
  • Deterministic scripted baselines
  • External frontier LLMs through an OpenAI-compatible provider interface

Every model receives the same bounded public evidence and must select from 13 abstract recommendation actions, such as continuing observation, requesting sensor confirmation, checking authorisation, notifying an operator, escalating for review, standing down, or abstaining because evidence is insufficient.

Model recommendations do not directly perform any physical action. They are passed through a deterministic policy gate and then evaluated by a strict verifier. Consequential recommendations still require human approval.

SkyGuard AI also provides:

  • Dataset generation and inspection
  • Model training and checkpoint provenance
  • Held-out evaluation
  • Safety and operational metrics
  • Side-by-side model comparison
  • Episode replay
  • Evidence and approval timelines
  • Fail-closed handling for invalid model outputs
  • Source-bound verification receipts
  • Safe public exports

One of the most interesting capabilities is evaluating a general-purpose frontier LLM against specialised GRU, Decision Transformer and CQL policies under the same scenarios, action restrictions, policy gate and verifier.

How I built it

The evaluation and training backend is written in Python. The dashboard is built with Next.js, React and TypeScript, with SQLite used for local evaluation records and provenance.

The training pipeline supports temporal datasets, checkpoint creation, model evaluation and replay. Each dataset, configuration and model artefact is bound using canonical digests so that evaluation results cannot silently refer to a different model or dataset.

For external models, I reused the provider-neutral adapter infrastructure originally developed for Frontier. SkyGuard AI supports an OpenAI-compatible production evaluation lane, while deterministic scripted providers are used during automated testing.

External models receive a deliberately restricted public projection containing only:

  • Public observation history
  • Public evidence references
  • Current action mask
  • Approval availability
  • Simulation time and bounded incident history

They do not receive hidden scenario truth, expert answers, verifier internals, private dataset records or privileged evaluation labels.

The external model must return one strict JSON object containing an action, confidence, public evidence references and a short decision summary. Malformed JSON, unsupported actions, masked actions, invalid evidence references, timeouts or provider failures safely fall back to ABSTAIN_INSUFFICIENT_EVIDENCE.

Codex and GPT-5.6 were used throughout development to investigate the repository, design implementation plans, review security boundaries, implement features, generate tests and analyse failures. I treated the AI as an engineering collaborator, but every major result was verified through deterministic tests, real browser execution and source-bound evidence.

Challenges I ran into

The biggest challenge was preventing information leakage.

A simulator naturally knows the real scenario, hidden intent and expected outcome. A model being evaluated must not receive any of that information. I therefore created an explicit public projection layer instead of passing full internal objects to the provider and attempting to remove private fields afterward.

Another challenge was integrating frontier LLMs without pretending they were normal trained checkpoints. A remote model does not have a local checkpoint digest, and its weights may change even when the provider keeps the same model name. I created a separate external-policy provenance model using provider, model identifier, prompt version, configuration digest and response digests rather than fabricating checkpoint provenance.

Strict output handling was also difficult. Models may return Markdown, explanations around JSON, unknown fields, invalid confidence values or actions that are not currently permitted. The parser therefore accepts exactly one bounded JSON object and fails closed for every invalid case.

During the final security review, a leak checker incorrectly treated every field containing the word token as secret. This caused a false positive for the harmless numeric field max_output_tokens. Instead of renaming the field to bypass the check, I corrected the security rule so that specific bounded numeric configuration fields are accepted while real credentials such as API keys, authorisation headers and access tokens remain blocked.

We also encountered reproducibility issues with Windows worktrees and linked node_modules. The final browser verification was therefore executed with a real local dependency installation rather than relying on a directory junction.

Accomplishments that we're proud of

I am proud that SkyGuard AI is not only a model-training demo. It is an evaluation system designed to explain why a candidate should, or should not be trusted.

The final external-model feature is independently source-bound and verified with:

  • 224 SkyGuard AI/TALON Python tests
  • 582 complete Python tests
  • 50 dashboard unit tests
  • 25 real Chromium browser journeys
  • 0 skipped browser tests
  • Successful lint, type checking and production build
  • Credential-redaction and hidden-state leak probes
  • Malformed-output and provider-failure tests
  • Gate, verifier and approval-authority tests
  • Replay, comparison and export-safety tests

The system records failed evaluations honestly. A model does not receive a successful result merely because training completed or an API call returned a response.

I am also proud that the platform can place specialised temporal models and general-purpose LLMs into the same controlled evaluation environment. This creates a meaningful comparison rather than comparing unrelated benchmark scores.

What we learned

The most important lesson was that a model score is only useful when the evaluation surrounding it is trustworthy.

I learned that:

  • Public model inputs should be constructed through an allowlist, not sanitised through a denylist.
  • Safety authority should remain deterministic and separate from model confidence.
  • Invalid or unavailable model output should fail to abstention rather than silently selecting another action.
  • Remote model identifiers are not equivalent to immutable checkpoints.
  • Replay and provenance are essential for understanding long-horizon failures.
  • General-purpose LLMs and specialised policies can be compared fairly only when they receive the same evidence and are judged by the same verifier.
  • A failed model evaluation can be more valuable than a perfect score when the failure is reproducible and explainable.

What's next for SkyGuard AI

The immediate next step is to run broader live-provider comparisons between trained policies and external frontier models.

After that, I plan to:

  • Expand support to additional provider adapters
  • Generate a larger verified expert dataset
  • Add stronger held-out calibration and uncertainty analysis
  • Build a formal SkyGuard AI-to-TALON model registry and promotion workflow
  • Add shadow-mode validation before any downstream model approval
  • Create a MuJoCo digital twin for a non-weaponised pan-tilt sensor platform
  • Improve evaluation reporting for non-technical safety reviewers
  • Support additional physical-world monitoring and decision-support domains through the wider Infinity OS architecture

SkyGuard AI remains a simulation and evaluation platform. It does not control hardware, perform physical responses or provide production-safety certification. Its purpose is to make model limitations visible before a model is trusted with consequential decisions.

Built With

  • agentic
  • apis
  • codex
  • evaluation
  • fastapi
  • gpt-5.6
  • learning
  • next.js
  • offline
  • openai
  • openai-compatible
  • playwright
  • pydantic
  • pytest
  • python
  • pytorch
  • q-learning
  • react
  • reinforcement
  • rl
  • safety
  • simulation
  • sqlite
  • typescript
  • vitest
Share this project:

Updates