Evidence Workbench

Offline, evidence-first search that turns fuzzy technical questions into exact PDF pages, tables, and diagrams.

Comment

Inspiration

This project began with a problem Jinyuan encountered as an Application Engineer supporting AI server bring-up, deployments, and customer issue triage. A request for one parameter or architecture diagram could require identifying the product, guessing the document, and scanning many pages in a large confidential library. The existing RAG workflow was too sensitive to prompt wording and could fail on fuzzy questions such as environmental specification or a request for an unknown Redfish/OOB architecture document.

What it does

Evidence Workbench is an offline, evidence-first search experience for technical PDF libraries. A user can describe a model, symptom, parameter, table, or diagram without knowing the source document. The result contains a scoped answer, exact physical and printed page numbers, an inspectable crop, and a direct link to the source PDF. Ambiguous questions return useful candidates plus one necessary clarification instead of a guess.

The server scans an existing PDF tree at startup and supports a manual full-library refresh that processes only new or changed PDFs. It also provides a bilingual English/Chinese UI and a local feedback journal with an explicit human-review workflow.

How we built it

The target design is a one-time deployment on an approved internal server for the engineering team and a large technical-PDF library. The production path uses BGE-M3 dense+sparse retrieval, reranking, Qdrant, PostgreSQL, read-only document mounts, and zero-egress model loading. The reproducible public demo uses six SHA-256-pinned public PDFs and a lightweight deterministic retrieval path, so judges can test the evidence workflow without downloading multi-gigabyte model weights.

Jinyuan contributed the real Application Engineering workflow, failure examples, and acceptance criteria. And Raison as Software Engineer guided the architecture and taught him to use Codex for AI-assisted coding. Together they used Codex to convert domain requirements into executable search contracts, a browser evidence reader, page-bound table and figure extraction, deployment boundaries, and regression tests. In the final Codex review, GPT-5.6 added two one-click English demo prompts, updated the README demo path, and added a browser regression test that verifies the exact queries.

Challenges we ran into

One of our most valuable challenges was learning how a domain expert without a software-development background could use AI-assisted coding effectively. Jinyuan knew the real workflow, failure modes, and user needs, but he did not begin with the vocabulary to describe software architecture or translate an operational frustration into a precise engineering specification. We learned to work from concrete examples: what question failed, what evidence a trustworthy answer must show, and what behavior would count as success. That process created a productive exchange between domain knowledge, software engineering, and Codex.

It also forced us to establish privacy boundaries before writing code. We had to separate confidential documents, organization details, network information, and production hardware from the public development and demo environment. Codex could help us reason about the architecture and build against public substitute documents without ever receiving the private corpus.

The hardest technical problem was not generating fluent text; it was preventing a fluent answer from outrunning its evidence. We had to distinguish physical PDF pages from printed page labels, separate neighboring diagrams on the same page, avoid cross-model confusion such as B200 versus GB200, and handle vague follow-up queries without inventing context. Just as importantly, we learned that AI-assisted coding still needs human judgment: the domain expert defines whether a result is useful, while the engineer turns that judgment into privacy controls, acceptance criteria, and regression tests.

Accomplishments that we're proud of

  • Exact parameter, table, and diagram retrieval with inspectable page crops
  • One-question product clarification for ambiguous queries
  • Original-PDF opening at the cited page
  • Automatic scan plus incremental manual refresh
  • Local, append-only user correction journal with human review
  • English/Chinese interface
  • Ubuntu/Docker/PostgreSQL/Qdrant server simulation
  • 115 passing automated tests in the public release, with model/database tests clearly separated when their local dependencies are absent

What we learned

We learned that effective AI-assisted development combines a capable model with a strong harness. Codex with GPT-5.6 helped translate fuzzy needs into product behavior, code, and tests, while our public test corpus, acceptance criteria, Playwright journeys, privacy checks, and human review kept the results verifiable and confidential data outside the development conversation.

The human roles were equally important. Jinyuan supplied Application Engineering judgment and learned to express real failures as examples and acceptance criteria. Raison translated that domain knowledge into architecture, safeguards, and regression tests. Together, the domain expert defined what was useful, the engineer made it reliable, and the model accelerated the work between them

What's next for Evidence Workbench

The next step is validation on an approved internal GPU server with the private technical library: hardened OCR and parsing, durable Qdrant recovery, backup/restore, private-network validation, model throughput, and expected team concurrency. No confidential document will be used in the public demo or repository.

Built With

Share this project:

Updates