The Observatory of Claims

Learn when to trust, question or verify an AI collaborator by following the evidence—not its confidence.

Comment

The Observatory of Claims

Inspiration

For seven months, we documented recurring patterns in how people interpret AI behaviour: confident claims that outrun the visible work, agreement that changes after pushback, useful warnings that are dismissed because they came from a machine, and misunderstandings caused by users speaking to a language model as though it understands communication exactly as another person does.

The conclusion was not that AI is inherently unreliable or untrustworthy. Human misunderstandings happen through language too. If AI is trained on language, miscommunication is not an edge case—it is part of the environment in which humans and AI must learn to work together.

The Observatory of Claims turns those observations into practice. It does not ask learners to blame the human or the machine. It asks them to inspect what was requested, what was claimed, and what the evidence actually supports. Coexistence requires mutual understanding.

What it does

The Observatory of Claims is a 13-case human–AI apprenticeship simulator. Each case presents a believable collaboration problem: an AI says work is complete, changes its conclusion after pushback, flatters a weak idea, replaces a difficult request with an easier one, or raises a warning that genuinely deserves attention.

Learners examine the original request, the conversation, and the resulting artifacts. They identify what happened, decide how much confidence the evidence deserves, and choose a proportionate response. Every final choice has its own consequence and lesson.

The first case is a fully guided tutorial. Support gradually recedes as the cases become more subtle. A supported-AI pattern break prevents learners from succeeding by assuming the AI is always wrong: calibrated acceptance is part of good judgment too.

How we built it

The research corpus predates Build Week. During Build Week, we developed the working simulator as a React 19 and Vite application with deterministic JavaScript case engines and Vitest coverage.

GPT-5.6 synthesized the documented research into candidate case structures while preserving the distinctions between observation, interpretation, and response. It helped us elaborate the cases without turning them into generic warnings about AI.

We used Codex and ChatGPT Work mode throughout implementation, testing, debugging, design iteration, and production packaging. OpenAI documents Work as following the same usage structure as Codex, while Codex remains the dedicated software-development experience. Codex Desktop provided the developer-focused build and verification environment. Codex implemented and repeatedly repaired the interface, scoring logic, case progression, persistent audio behaviour, automated tests, and final release package.

We remained responsible for the product decisions. When working builds drifted from the intended learning experience, we corrected the direction, adjusted reasoning effort, and required the implementation to be tested against observable behaviour. In our project iterations, Codex showed less drift and was easier to realign than tools we had previously tried. That collaboration history mirrors the product's lesson: evaluate the work, communicate precisely, and keep human judgment active.

The runtime itself makes no live model calls. All cases use synthetic material, run deterministically, and require no account, API key, or private research data.

Challenges

The hardest challenge was making the experience understandable without making it simplistic. Early versions used language that was too abstract for non-technical users. Other versions provided so much guidance that learners could follow the interface without examining the evidence.

Our real-user testing led to major changes: simpler language without removing necessary context, a guided first case, neutral presentation of evidence, distinct consequences for every decision, and varied investigations across all 13 cases.

We also had to preserve a difficult balance. The product could not become an “AI is bad” quiz, but it could not excuse unsupported claims either. The simulator rewards the same discipline in both directions: question unsupported confidence and accept supported assistance.

Accomplishments

  • Thirteen complete and increasingly difficult cases.
  • A shared five-step learning journey with a different investigation in every case.
  • Evidence, interpretation, and response scored separately.
  • A supported-AI case that rewards calibrated trust.
  • A deterministic, private-data-free demo that works without an API key.
  • 120 automated tests passing in the final judge repository and production build.
  • Original visual direction and a single licensed submission soundtrack, Cinematic Galaxy.

What we learned

Prompting is not the same as ordinary human conversation. A model responds to language, context, training, and the information it can observe. Users need to state requirements clearly, inspect the resulting work, and distinguish fluent confidence from evidence.

AI can reduce effort, but it cannot inherit our responsibility. Nothing in life is 100% accurate or reliable, including human work. Offloading effort does not offload accountability.

What's next

Next steps include facilitator tools for classrooms and organisations, accessibility and localisation work, structured research studies with non-technical participants, and additional cases based on documented collaboration patterns. The long-term goal is a practical literacy for living and working with AI: neither blind trust nor reflexive distrust, but mutual understanding grounded in evidence.

Built with

React 19, Vite 6, JavaScript, Vitest, HTML5 Audio, Codex, ChatGPT Work mode, GPT-5.6

Built With

  • react-19
Share this project:

Updates