Research Society

AI agents debate a paper's methods, novelty, and gaps in real time, plus natural-language search, so you never need to find them yourself

Comment

Inspiration

The honest truth is that keeping up with new research is exhausting. Each week, dozens of papers drop in fields I'm interested in, and what I really needed was a way to quickly assess whether a paper was worth my time before committing to a deep read. I didn't just want a single "this paper is interesting" summary — I wanted to see why it might be good or bad.

That's where the idea for Research Society came from. What if I could offload that initial triage to a panel of AI reviewers who would actually argue about the paper, highlight its strengths and weaknesses from different angles, and preserve disagreements — so I could get a full picture of a paper's quality in minutes instead of hours? Not a black-box "score" I'd have to trust blindly, but a transparent debate I could watch and learn from, making my own judgment along the way.

What it does

Research Society turns paper review into something you can watch happen, instead of a black-box summary.

Give it a paper — by uploading a PDF, or an arXiv ID, DOI, or URL — and a panel of five specialist Qwen-powered agents (an Executive Moderator plus a Structure Analyst, Contribution Scout, Methodology Critic, and Literature Reviewer) goes to work. They read the paper, form independent opinions, then debate each other live over a WebSocket feed: raising hands, rebutting each other's claims, and converging on consensus — with disagreements preserved in a dissent ledger rather than smoothed over.

Alongside the debate, the system:

  • Fact-checks specific claims in the paper against cross-references
  • Generates an interactive concept map of the paper's ideas
  • Benchmarks its own output against a single-agent baseline, so you can see concretely what a debate-based review catches that a single "summarize this paper" prompt misses

And if you don't have a specific paper in hand, you don't need an arXiv ID or DOI — you can just describe what you're looking for in plain English, and the system searches across arXiv, OpenAlex, PubMed, and Crossref to find it for you.

How I built it

  • Backend: Python/FastAPI, exposing a REST API (/api/paper/*, /api/debate/*, /api/map/generate, /api/fact-check/verify, /api/benchmark/compare) plus a /ws/chat WebSocket that streams debate events to the frontend in real time
  • Agent orchestration: a DebateManager coordinates the five agents through structured rounds — hand-raising, rebuttal, consensus voting — with each agent's reasoning routed through a shared LLM client
  • LLM: every agent, the moderator, and the benchmark comparison run on Qwen models via Alibaba Cloud's DashScope API (Model Studio), called through an OpenAI-compatible client
  • Paper discovery: a natural-language and structured-ID ingestion pipeline pulls metadata and full text from arXiv, OpenAlex, PubMed, Crossref, and Unpaywall — no paid keys required for paper discovery itself
  • Frontend: a React (Vite) single-page app with dedicated views for the debate stage, verdict cards, the benchmark comparison, and the concept map, all driven by the live WebSocket feed
  • Deployment: a multi-stage Docker build (Node builds the frontend, Python serves it and the API from one process) running on an Alibaba Cloud ECS instance, so the whole stack — frontend, backend, and the live debate feed — is one container talking out to Qwen Cloud

Challenges I ran into

  • Getting agents to actually disagree. The naive version of "five LLM calls on the same paper" just produces five similar summaries. Getting agents to hold distinct roles, push back on each other, and preserve genuine disagreement — rather than collapsing into agreement on the first round — took real tuning of prompts and the consensus threshold.
  • Qwen3's hybrid thinking mode. Qwen3.x models default to an internal "thinking" mode that isn't compatible with the structured, low-latency responses a live debate needs. I had to explicitly disable it (enable_thinking=false) to keep the debate fast and parseable.
  • Streaming a multi-agent debate live, not just returning a final report — coordinating WebSocket broadcast timing so the frontend shows agents "speaking" in the right order without the UI getting ahead of or behind the actual agent calls.

What I learned

The most surprising thing was that agents genuinely disagree when given distinct personas and enough space to argue. I expected them to converge to agreement quickly, but in practice, the Methodologist and Novelty Reviewer often clash, and the Literature Reviewer sometimes contradicts both. The dissent ledger ended up being one of the most valuable parts of the system — it shows real intellectual tension, not manufactured disagreement.

Deploying on Alibaba Cloud was smoother than expected once I understood the security group model. The ECS instance with Docker Compose worked flawlessly after opening the right ports. The Qwen Cloud DashScope API was responsive from the US Virginia region — latency was minimal, and the OpenAI-compatible client made integration straightforward.

What's next for Research Society

  • Support for more paper sources and full-text retrieval coverage — expanding beyond arXiv to more subscription-based journals via institutional access
  • Persistent storage of past debates/reviews — currently fully in-memory/stateless; I'd like to build a database so users can revisit past reviews, compare papers, and track how their own assessments evolve
  • Letting users join the debate — ask the panel a follow-up question mid-review, or challenge an agent's claim and see how the society responds
  • Expanding the agent roster — a Reproducibility Checker (to flag papers with insufficient code/data sharing), an Ethics Reviewer (for datasets and potential harms), and a Statistics Auditor (to catch common statistical errors)
  • Better visualizations — turning the concept map into an interactive exploration tool where users can click on nodes to see related debates or fact-check results
  • Domain-specific agent personas — starting with drug discovery literature, where I'd introduce a Pharmacologist, Medicinal Chemist, and Clinical Trial Auditor to handle molecular efficacy, safety profiles, and clinical trial methodology based strictly on published literature

Built With

  • alibaba
  • cloud
  • docker
  • fastapi
  • qwen
  • qwencloud
  • tailwind
  • vite
Share this project:

Updates