StoryTrace

StoryTrace cross-examines a manuscript against itself, an agentic continuity engine that catches the plot holes a human editor misses.

Comment

StoryTrace — Devpost Submission

Inspiration

I'm a web novel reader; specifically, a devoted one. Having gotten through hundreds of chapters of things like Reverend Insanity, things become apparent around chapter 200. A character's ability which was a big part of their arc just disappears from existence. A relationship which was clearly established at the beginning gets entirely ignored after some time. It's not even web novels; the age timeline of Naruto's generation is a complete disaster when you try to organize all that information. The characters are too old and too young at the same time, depending on the arc.

These are not minor flaws. These are inconsistencies in the story that destroy its immersion, but they go unnoticed because no one; neither the author, nor the editor, nor the production crew; can keep hundreds of chapters worth of information in mind at once. I wanted to create a system for doing that.

What it does

StoryTrace starts with a screenplay or novel, splits it into scenes or chapters, and pulls out structured facts about who is holding what, who is injured, who is where in each one of them. These facts are recorded in ClickHouse as an append-only event log, giving you a complete history of the world state of your story in each and every unit.

The SQL window functions analyze the event log to find any suspicious transitions — instances when something changed with no explanation between. Every such a transition becomes a candidate for the Investigation Agent which queries the database using MCP, analyzes adjacent scenes, and decides whether it is really a continuity mistake, whether it has been explained already, or the agent cannot tell due to the lack of information.

Any result always comes with evidence — the sentence that set up the state of the world and the sentence that contradicted it, and the full trace of the agent's work so that the writers understand why it was flagged. Verified conflicts are accompanied by a suggested solution: a single sentence that the writer needs to add to cover the conflict.

How we built it

Began using Ollama locally for all of testing - no API costs, rapid iteration, good enough to prove that the architecture made sense before spending any money. The architecture is purposefully divided between the deterministic and the agentic parts - parsing, extraction and candidate detection are deterministic and repeatable, while investigation is the only true agent since it is the only one that requires autonomous decision- making rather than pattern matching.

When we received the Google Cloud credits due to our participation in hackathon, we migrated LLM layer to Vertex AI. That required us to write a provider from scratch, switch from API key authentication to GCP service credentials, and to make an amusing discovery that google-adk and Gemini SDK both use the same client, which determines what backend to call based on a different environment variable - so a solution that seemed to be pointing at Vertex silently used the free Gemini API all along. Little thing, quite hard to debug.

Storage is ClickHouse everywhere - append-only events log, window function queries for candidate detection, and Investigation Agent querying it live via MCP server, not reading pre-made context. Frontend is a three-pane Next.js app with chapters on the left

What we learned

Nine dollars down the drain in worthless Vertex AI calls. Every investigation returned uncertain, 0% confidence, and we squandered far too much time chasing a model quality problem or prompt engineering problem before discovering the actual root cause.

The Investigation Agent’s prompt told it which conflict to investigate. It did not tell it which entity ID to use for querying the database. Therefore it had to guess IDs from context, returned empty for all of its queries, used up its budget of 6 calls without getting any data and returned uncertain. Again and again and again. Until we realized that nine dollars of API cost us a single missing variable.

After that was resolved, a whole series of smaller issues became obvious — because the agent was now actually returning valid conclusions that could be rejected. Status returned "Verified" when it should have been "verified". Confidence returned "high" when it should have been a float number. Arguments for the tool were supplied in escaped JSON format within a string, and anything resembling an apostrophe in an explanation broke it.

The fix was simple every time: stop trying to get the exact right textual response from the model and make some

Challenges

Putting the whole stack together to work in an end-to-end fashion was a different challenge entirely from getting the code to work. Docker Compose, ClickHouse, Ollama on the host, and Vertex AI all talking to each other – things that worked great by themselves failed when put together in certain configurations.

The CORS conflict only came out once we were actually running the frontend in the browser. Host-level firewall silently blocked traffic between containers and the host machine for Ollama. Autofill styling caused login credentials to show up in light blue in a dark theme – it looked broken, but it was actually perfectly fine.

None of this could be observed by looking at the code. We needed to actually go through the running product – and thus we ended up moving towards verification through browser automation rather than through code reviews. This same strategy uncovered the best bug ever found through the entire build process – a header button that looked fine, rendered correctly, had the right label, but was completely unclickable due to the fact that another panel was drawn on top of it in the DOM. It would have shipped as-is, nobody would have known why this

Built With

Share this project:

Updates

Submission history