Arxiv GigaResearch

We read arxiv papers, but miss out on hundreds every week. Now we can read everything and understand which are truly novel. Our Gigaresearch agent was simple to build using subconscious.dev.

Comment

Is your research truly novel? We want to know.

Inspiration

We built Arxiv GigaResearch using Subconscious.dev as the core agent. Subconscious.dev is our own platform (we came out of stealth 2 weeks ago), and we wanted to use this hackathon to showcase its power. We learned a lot, experimented with some new patterns, and if we had more time we planned to run the agent 100x in parallel to discover the "most novel" research of the day.

Our process first understands the research, then looks up relevant survey papers in the domain to compare it to, then looks at related and recent papers, and finally determines a novelty score and most applicable industries. All of this workflow is simply encoded in our prompt and the agent generates the workflow to get the job done.

What it does

Arxiv GigaResearch lets you explore, analyze, and connect massive amounts of research data well beyond the normal context limits of LLMs. It uses subconscious.dev to handle large multi-step reasoning chains, integrating custom-built search and processing tools for rapid, high-volume research synthesis.

How we built it

We integrated subconscious.dev as the reasoning engine, added custom Google Search and JavaScript tools for data retrieval, and built the frontend using convex.dev, Vercel, Next.js, Shazian, and Tailwind. The workflow moves from fetching large-scale research data to structuring and summarizing it, all in a single agent pipeline. In addition to using Subconscious, we built tools for the agent using Google SERP API and simple JavaScript, then vibe coded the frontend/db with convex.dev, Vercel, Next.js, Shadcn, and Tailwind.

Challenges we ran into

We pushed token throughput to extremes and hit new performance and scaling limits in Subconscious.dev. We also discovered that convex.dev, while great for quick development, struggles with large datasets. It was a pain to debug. That took us more hours than we thought it would.

Accomplishments that we're proud of

We built the entire system starting around 6pm Wednesday despite time constraints from customer meetings and a pitch event. We fixed multiple bugs in subconscious.dev along the way, and proved that the system can support high-volume, high-context research workflows.

What we learned

We uncovered scaling bottlenecks in our own system and fixed many of them. We also learned more about the real-world limits our customers might hit and how to address them. Plus, we explored a ton of interesting research papers along the way.

What's next for Arxiv GigaResearch

We plan to open source the repository, make improvements for better performance with large datasets, and share it so anyone can build their own GigaResearch agent.

Built With

  • convex
  • nextjs
  • shadcn
  • subconscious.dev
  • vercel
Share this project:

Updates