Research Desk

Inspiration

A financial assistant can do the arithmetic correctly and still give the wrong answer. It may select a quarterly figure for an annual question, confuse sales with total revenue, or reverse the inputs to a formula. An analyst must then find the source and explain the mistake, often repeatedly.

We built Research desk to make that review work carry forward. A checked correction becomes a candidate procedure that must demonstrate improvement before it can influence future answers.

What it does

Research desk supports operating-margin and reported revenue-growth analysis for Target and Costco. Each answer exposes its source figures, fiscal periods, units, formula and reporting notes. Users can import PDF, HTML or text reports, check the extracted evidence, run analyses and export reviewable memos.

Its central workflow is analyze, correct, validate, approve and reuse. A correction produces an inactive procedure with explicit conditions and exclusions. Paired validation compares answers with and without it. Activation requires an applicable improvement, all applicable cases passing, boundary coverage, zero observed regressions and human approval.

Accepted rules preserve a selection method rather than a numerical answer. For example, an annual operating-margin rule identifies operating income as the numerator and total revenue as the denominator, while excluding quarterly columns and unrelated revenue bases.

We built a custom HTML, CSS and JavaScript dashboard over a Python backend using FastAPI and Pydantic. LangGraph coordinates planning, evidence retrieval, source selection, auditing and human review. SQLite preserves checkpoints, analyses and versioned procedures.

Qwen3 8B runs locally through Ollama, performing separate planner, analyst, auditor and teacher calls. Its weights stay fixed. Learning happens through persistent procedural memory and a bounded Python executor for typed source-label rules. Python also executes restricted financial formulas.

Hybrid retrieval combines BM25 keyword search with Chroma vector search and local Nomic embeddings through LlamaIndex. pdfplumber extracts PDF evidence for human curation. MLflow records local traces and evaluation metrics.

Challenges

Our Costco upload exposed a real failure: the assistant reversed the annual operating-margin inputs and returned 2740.47%. Its auditor accepted the choice. Correct arithmetic and model agreement had not established that the source selection was correct.

A text-only lesson initially produced no validation gains, so the acceptance gate blocked it. We added a typed source-label execution path and tested the combined saved-rule system. We also had to preserve annual versus quarterly distinctions, handle both 52- and 53-week fiscal years, and invalidate approvals when their validation configuration became stale.

Results and what we learned

Recorded development validation improved from 9/12 to 11/12, with two gains and zero regressions. Both gains concern annual periods within one separate Costco release. One sales-basis boundary case still fails in both conditions.

The approved rule returned 3.65% on the original FY2024 Costco upload, with its execution visible in the trace. A Target growth question used zero Costco procedures. The project also has 68 offline software tests using mocked inference and isolated state.

We learned that saving feedback is only the beginning. A useful lesson needs evidence of improvement, a clear scope and an inspectable execution history. These results measure the combined procedural-memory and executor system on development validation. An equivalent nonlearning typed-rule baseline and an independent final evaluation remain future work.

What's next

Test more independent reports, compare against an equivalent fixed-rule system, and measure repeated corrections and analyst review time. Shared team procedures are a future direction, subject to access controls and further validation.