SceneLedger

Know what the scene can honestly say.

Comment

About the project

I was interested in the point where research becomes a script.

A film or media production brief can contain factual questions about people, events, places or claims. Before those questions become narration or interview prompts, the production team needs to know what is supported, what is contested and what still needs primary verification.

Small teams often move quickly from research into scripting. When that happens, the source trail and uncertainty can become difficult to reconstruct. I did not want to build another system that produced a confident summary and left someone else to work out which source supported each sentence.

SceneLedger reviews original script lines against retrieved evidence, while keeping the producer's wording intact. It also supports research from a production brief before a script exists.

The script review desk connects each original line to extracted claims, model assessments, quotations and source links. If a claim remains unverified, the application withholds proposed wording rather than offering a rewrite that might conceal the evidence gap. A line omitted by the model is shown as unassessed, not silently treated as checked.

How it works

  1. Gemini 3.8 Flash on Vertex AI turns the production context and script into factual research questions.
  2. Parallel Search runs at runtime for the planned questions and returns source-linked excerpts. The application deduplicates sources and applies the selected publication-date cutoff before synthesis.
  3. Gemini is prompted to assess extracted claims using that source bundle and map them back to the original script lines.
  4. Deterministic code checks every cited source ID and requires each quotation to occur contiguously in a retrieved excerpt, allowing whitespace differences only. Unknown IDs and mismatched quotations are removed. Any rejected citation makes its affected claim unverified, even if another citation survives.

The ledger displays supported, contested and unverified assessments. Supported means the model judges the retrieved evidence to support the claim; contested is intended for disagreement between sources; unverified means the available evidence is insufficient. These remain model assessments, not independently verified facts. Passing the quotation check does not establish that a quote actually supports the model's interpretation.

One bounded follow-up

The producer can enable one follow-up round for up to two unresolved claims: at most two additional Parallel searches and one Gemini reassessment call. There is no recursive research loop. The interface reports the actual calls and before-and-after statuses, including outcomes that remain unverified. If no new excerpts are retrieved, reassessment is skipped; failures retain the initial checked ledger.

A reviewable handoff

The text evidence pack and full JSON download preserve the original input, line-level assessments, retrieved excerpts, source links, citation issues, follow-up trail and runtime metadata. This gives an editor something to inspect outside the app. No original script line is automatically replaced, and proposed wording never provides publication clearance.

How I built it

I built SceneLedger as a Node.js service running on Cloud Run, using Express and Zod. The application uses the official @google/genai SDK with Vertex AI authentication and the official parallel-web SDK. The Parallel credential is stored in Secret Manager, while Cloud Build and Artifact Registry support deployment.

I used Codex during development to help design and implement the application, write regression tests, inspect browser behaviour and prepare the deployment. Review work exposed a check that accepted fabricated quotations attached to real source IDs, and advice that could remain overconfident after a claim was downgraded. Those cases now have deterministic safeguards and regression tests. Codex is not called by the deployed application; the AI runtime uses Gemini and Parallel.

Challenges

The main challenge was keeping the model inside a clear evidence boundary. A language model can produce a convincing answer even when a source is missing or does not support the claim. I therefore had to make provenance part of the application logic rather than relying only on the model's final response.

The follow-up pass introduced another risk: a reassessment could quietly substitute a narrower claim for the original. The application now requires the returned claim to retain its identity before applying a new assessment. A failed or changed reassessment must not look like successful verification.

Deployment also required careful handling of credentials and live calls. The public demo limits repeated runs and concurrent work, and Cloud Run is configured with zero minimum instances and one maximum instance. Those controls reduce exposure but do not impose a fixed total bill.

What I learned

The main thing I learned is that a citation is not automatically evidence. A response can contain URLs and still fail to show which source supports which claim. Even a genuinely matching quotation can accompany a wrong interpretation.

The editorial consequence also matters. A research tool is more useful when it explains what the evidence means for the next decision, rather than stopping at a list of sources. Sometimes the useful result is that extra research still does not justify the sentence.

Testing and evaluation

All 36 automated tests passed on 8 September 2026. They use deterministic provider doubles and make no paid API calls. Coverage includes quotation fidelity, invented source IDs, claim downgrading, original-line preservation, missing mappings, follow-up limits and identity checks, evidence-pack contents, streaming, validation and failure handling. Passing these tests does not measure model accuracy.

The upgraded public deployment was also exercised with live Gemini and Parallel calls on 7 September. That recorded run assessed three original lines, retained 28 sources and completed two extra searches plus one reassessment. All three claims remained unverified, and a mismatched quotation was removed. Both download formats were saved and inspected. See the deployment verification record.

A separate preliminary evaluation used 24 hand-selected narration probes against four frozen NASA excerpt bundles, with provisional Codex-authored labels that have not been independently human-reviewed. With and without the citation guard, the same saved outputs produced 0/16 false acceptances and 0/8 false rejections. There was no measured guard benefit on this simple set. These are binary acceptance measures, not three-way accuracy or evidence of general reliability. The saved run also predates later advice and prompt changes. The evaluation notes and reproducible protocol preserve those limitations.

Try it

  1. Open the live app and select Load example.
  2. Keep Review script lines selected. Optionally tick Try one follow-up round for unresolved claims, then click Review narration once.
  3. Wait for the live workflow to finish. Inspect the actual citation badge, an original line and its assessment; results can vary between runs.
  4. If follow-up ran, inspect its call counts and before-and-after trail. Open a source link, then download the evidence pack or JSON.

The source repository is public under the MIT licence. For the automated checks, use Node.js 22 or newer, run npm ci, then npm test.

Limitations

SceneLedger checks source IDs and quotation presence in retrieved excerpts, not semantic entailment, source accuracy or publication clearance. The model can miss factual assertions, and an assessed line is not a guarantee that every assertion was identified.

Search results and synthesis vary. The cutoff uses available publication metadata, not historical page reconstruction; undated sources are retained and labelled. The public demo rate-limits repeated research runs. Runs are not saved between page visits, so download the evidence pack if you need to keep it, and review its original input before sharing.

SceneLedger does not replace editorial judgement, primary-document verification, consent checks or legal review. Its purpose is to make uncertainty visible before research becomes a script, not to make the final decision on behalf of the production team.

Built With

  • cloud-run
  • express.js
  • gemini-3.8-flash
  • google-cloud-agent-platform
  • node.js
  • parallel-search-api
  • vertex-ai
  • zod
Share this project:

Updates

Submission history