immerXiv

immerXiv is a repository that provides interactive insights and trends on arXiv papers through real-time sentiment analysis and keyword visualization.

Comment

Inspiration

As students involved in research, we're familiar with the painstaking effort required to parse through databases and draft literature reviews. It would be a timesaver and lifesaver to have a tool that can easily highlight the most popular papers in a given field and indicate whether their topics are gaining traction or fading.

What it does

immerXiv delivers real-time, interactive insights into arXiv papers by combining sentiment analysis and keyword visualization, enabling researchers to quickly identify emerging trends and popular research topics. By analyzing abstracts, Reddit discussions, and user feedback through a "Hot or Not" game, immerXiv simplifies the discovery of valuable research directions.

How we built it

We built immerXiv using the Next.js framework bootstrapped with create-next-app, featuring a front end coded in JavaScript and a back end in Python utilizing Flask API for seamless integration. To aggregate and process data, we integrated multiple APIs including arXiv, Kaggle, Reddit, and Semantic Scholar. Sentiment analysis on abstracts and Reddit discussions was performed using a Large Language Model (LLM). To showcase how papers fit into specific research fields, we implemented advanced filtering algorithms that generate dynamic lists of relevant papers, providing users with a focused selection highlighting key trends and emerging topics.

Challenges we ran into

  • Real-time analysis of X (Twitter). We were only able to use the X API v2 Free plan, which did not allow for real-time analysis of posts on X.
  • Working with the Semantic Scholar API. Semantic Scholar by default gives very general and unspecific information in .json files, which we had to preprocess and modify to have data that could generate our graphs.
  • Originally, we wanted to create knowledge graphs that provided a visual representation of the relationships between papers. We were able to create the graphs but had a difficult time integrating them through the API. Thus, we had to pivot to creating dynamic lists.
    • We wanted to have an analogous sentiment analysis function for datasets with Kaggle API but couldn't get it done with the data provided by the API.

Accomplishments that we're proud of

  • Hot-or-Not provided some interactivity that we think makes our tool more fun than traditional repositories.
  • Sentiment analysis of abstracts and alternative data led us to create dynamic visualizations of real-time trends in research.
  • Flexible code structure enables us to swap out state-of-the-art LLMs for sentiment analysis.

What we learned

  • Learned about how full-stack teams work together with clear divisions of labor in front-end and back-end development.
  • Learned about how popular APIs can be integrated into web frameworks.

What's next for immerXiv

  • Temporal sentiment analysis will help uncover more insights into the research topics
  • Translating the capabilities of immerXiv to different industries/applications that involve databases; we can position ourselves as an end-to-end data processing solution for visualizing relationships between data.

Built With

Share this project:

Updates