Inspiration
Staying current in tech is a manual job. You open ten tabs, skim the same story three times, forget what mattered last week, and when something does matter you research it by hand: reading blog posts, cross-checking claims, and still not knowing which source said what.
My first version of CortexResearch was a basic multi-agent pipeline that answered one question at a time. That wasn't the real problem. The real problem was that I didn't know what to research, and I couldn't trust the output when I did.
There was a second problem inside the first. Most "AI features" ask a chat model a yes/no question, get a paragraph back, and regex the answer out. Nothing tells you how sure the model was. I wanted every fast decision in the system (is this relevant? is this claim supported?) to come back with a probability and a confidence, so the system knows when it is unsure. That led me to Jev's System One.
So I rebuilt the project into a market-pulse radar plus a deep-research engine with Jev as its fast decision layer. It reads the internet for me, tells me what matters to me, and when I click something, it researches it and verifies every claim against its citations.
What it does
Pulse (the radar). Six free sources feed one stream: Hacker News, arXiv, RSS/blogs, Reddit, GitHub trending, and Product Hunt. Items are deduplicated across sources, so the same story from HN, Reddit and two blogs collapses into one card with four source links. Each item is scored against a YAML interest profile and carries a written "why this matters to you" line.
Deep brief (the research engine). Click any story, or ask a question, and a LangGraph pipeline runs planner → researcher → writer → verifier → reviser. The output is a cited report with a TL;DR, primer, timeline, risks, FAQ, and a verification panel showing which claims were supported, flagged, or removed.
Library and Digest. Hybrid search runs over every item and every report you've generated. A daily digest (file, email, or Slack) covers what's new and what changed. Digests never call an LLM.
It's one SQLite file, a FastAPI + SSE backend, and a Next.js 16 UI that streams real pipeline events as the run happens.
How we use Jev
Jev's System One is the fast decision layer. Instead of asking a chat model to write a paragraph, the client sends one named choice question to Jev's /v1/systemone endpoint and gets back one answer. The chosen option's probability is the score, and Jev's derived confidence decides whether the decision escalates.
Three kinds of fast decisions are routed this way:
| Decision | Where | What Jev returns |
|---|---|---|
| Query classification and decomposition | Planner | Query class and sub-questions, with confidence |
| Claim support | Verifier | Supported / unsupported per claim, with confidence |
| Item relevance | Pulse ranking | A relevance label with a probability |
The routing logic:
$$\text{route}(x) = \begin{cases} \text{System 1 (Jev)} & \text{if } c(x) \ge \tau \ \text{System 2 (fast chat tier)} & \text{otherwise} \end{cases}$$
where $c(x)$ is the confidence attached to the decision. "Unsure" is a first-class outcome: it escalates instead of silently guessing.
- Batching: within the verifier, a low-confidence group of claims escalates to the fast chat tier in one call, not one call per claim.
- Resilience: a circuit breaker opens after 3 consecutive Jev failures. If Jev is down, or no
JEV_API_KEYis configured, a local logistic-regression fallback fitted on your own score history takes over, so the app keeps working. - Transparency: every routed decision is recorded in the run timeline and the report trace, so you can see why something was judged relevant or flagged unsupported. The
/healthendpoint reports which System 1 backend is live. - Tracing: Jev routes show up in Langfuse traces alongside generations, guardrail catches and the verification score.
How it's built
- Ingestion: one adapter per source, each with its own timeout, retry, and failure isolation, so one broken feed never takes down a cycle. Dedup uses URL plus a content hash, then clusters the same story across sources.
- Ranking: a cheap prefilter (recency + embedding similarity) shrinks the pool, then an LLM scores only the survivors with structured output. Scores are cached by
(content_hash, profile_version). Re-running the same ingest makes zero LLM calls, and a test asserts exactly that. - Retrieval: SQLite FTS5 (BM25) and dense MiniLM vectors, merged with reciprocal-rank fusion and reranked by a cross-encoder:
$$\text{RRF}(d) = \sum_{r \in R} \frac{1}{k + \text{rank}_r(d)}, \qquad k = 60$$
RRF needs no score calibration between retrievers. I used exact cosine in numpy instead of FAISS, since a personal corpus doesn't need an ANN index.
- Agents: a LangGraph state machine with a bounded revision pass. The verifier checks citations deterministically first (a missing or unknown source id means unsupported, with no LLM call). The remaining claims go through Jev, and the reviser is authoritative for the final verification block.
- Guardrails: every scraped page and snippet passes an injection scan and a PII/secret scrub, then is framed as data, never as instructions.
- Tooling: Langfuse tracing, per-run cost metering with tiered models, an MCP server exposing 7 tools to clients like Claude Desktop, and an eval harness.
- Quality: 246 offline tests (no network, no live API, no model downloads), ruff, GitHub Actions CI on every push, and Docker.
Challenges
1. Decisions you can't calibrate. Parsing yes/no answers out of prose gives you no confidence signal, so there's nothing to escalate on. Moving fast decisions to Jev's choice-question contract fixed that. It also forced me to design the failure path: the circuit breaker, the local logistic fallback, and the rule that low confidence escalates to System 2.
2. The internet is an injection vector. A poisoned blog post can tell the model to ignore its instructions, and my agents read the web all day. I built a scanner with six weighted heuristic categories and a PII/secret scrubber (emails, phones, SSNs, Luhn-valid cards, API keys, JWTs). A scanner that flags everything is useless, so the attack corpus includes near-misses, such as articles about prompt injection and legitimate PII. Quality is measured by precision and recall:
$$P = \frac{TP}{TP + FP}, \qquad R = \frac{TP}{TP + FN}$$
The eval is threshold-gated in CI, so the build fails if detection regresses.
3. "The report sounds right" isn't good enough. LLM reports are fluent whether or not they're grounded. I moved to claim-level verification with citations checked per claim. Reports are also scored by an LLM-as-judge suite with a four-dimension rubric (groundedness weighted 2×), and deterministic facts are computed in code instead of being asked of a model.
4. Malformed model output. Schema-v2 parsing remaps field aliases and salvages entries one by one, so a single malformed row can't destroy a report. If the whole pipeline fails, it still returns something schema-valid.
5. Stale state. The old version relied on a prebuilt faiss_index/ that went stale whenever the data changed. The new index is content-addressed and rebuilds itself when the store changes.
6. Cost and graceful degradation. I used a tiered model setup (fast tier for scoring and extraction, frontier tier for synthesis), cached scores, and a cheap prefilter before any LLM call. Optional pieces (GraphRAG, Supermemory, Langfuse) are feature flags the app never hard-depends on. Retrieval still answers if the embedding model fails to load, and a report still ships if the reviser crashes.
What I learned
- Calibrated decisions beat parsed prose. A probability plus a confidence lets a system know when it is unsure and escalate, which a paragraph of text can't do.
- Design for failure first. Per-adapter isolation, the Jev circuit breaker, and schema-valid fallbacks mattered more than any single clever feature.
- Measure what you claim. Making guardrail quality a CI gate turned "it seems safe" into a number that can regress.
- Simple storage goes far. SQLite with WAL and FTS5, plus numpy, covered hybrid retrieval with no services to operate.
- Offline tests change how you build. With fixtures and no network, I could refactor across six sources and five agent stages without fear.
- Write the spec before the code for the hard parts (guardrails, routing, evals, memory, GraphRAG, MCP). The design specs are in the repo.
What's next
A trends/momentum view, a command palette, audio briefings, and multi-user accounts.
Built with
Jev (System One) · Python · FastAPI · LangGraph · LangChain · SQLite (WAL + FTS5) · sentence-transformers (MiniLM + cross-encoder) · numpy · Anthropic Claude · Next.js 16 · Tailwind v4 · APScheduler · Langfuse · LightRAG · Supermemory · MCP · Docker · GitHub Actions · pytest · ruff
Log in or sign up for Devpost to join the conversation.