NewsPulse

digest article summaries expanded
coverage pulse and calendar reminder
homepage digest
article statistics

Inspiration

Often times, the modern news feels overwhelming. There's so much noise, so many distractions in our everyday lives that it's hard to make sense of it. With the power of modern AI, however, we can now filter this arbitrary noise and condense the information down to a format that's actually useful on a regular basis.

What it does

NewsPulse solves the aforementioned problem by grouping and summarizing articles into useful feeds with customizable filters that anybody can query. It summarizes articles at both the individual level and topical level, granting users control over the granularity of the summaries they read. This both saves time and helps increase their awareness of the topics they are interested in.

How we built it

The majority of our app consists of a 3-step data pipeline:

Extracting, cleaning and filtering data from GDELT (Kelly)
Summarizing, attaching topics/themes, and grouping into buckets via LLMs on NVIDIA Nemotron and LocalLLM (Tingxu)
Ingesting the processed data into a realtime application database that was performant enough to serve to live users (Tristan)

Once we had a database up and running, we hooked it up to a Rust/Axum server (Tristan) which provided both regular iCalendar streams for easy interoperability and an API which our frontend used fetch data (Deep).

Challenges we ran into

Nemotron ratelimit - Due to our huge data processing workload and the amount of other people also using Nemotron at the same time, we ran into ratelimiting issues while processing our data. We fixed this by offloading some of the work to a non-ratelimited LocalLLM instance and aggregating the final data back together (though even then, we barely finished processing on time).
Configuring deployments - Due to the compile time of our Rust backend (despite Docker container optimization), iteration and debugging on live deployments was a slow and tedious process of trial and error.
Choosing our stack - With our diverse skillsets, it was difficult to choose a stack that equally accommodated all of our skills while also coming together to form a functional app. In the end, we decided on developing in a bunch of independently maintainable, testable units that were eventually tied together with real data as we neared the completion of the project.

Accomplishments that we're proud of

Processing the sheer amount of data we did on such a short time constraint.
Simply delivering a product on time (this was the first hackathon for most of us, so we didn't know what to expect going into it)
Creating a novel, broadly applicable tool with the potential to enhance the everyday lives of many people.

Experience Gained

Collaborating on a team and weighing diverse opinions
Prototyping rapidly in order to deliver on a tight deadline
Utilizing AI to convert redundant natural language into a more streamlined summary.

What's next for NewsPulse

We hope to scale up our database and automate our pipeline into a running service via cronjobs. We're also considering adding an email newsletter for yet another way to integrate the app into daily life. We might also improve the theme selection algorithm by making a custom MCP server to provide more context and get more accurate and less redundant results.

Built With

axum
gdelt
javascript
nemotron
nextjs
python
rust
surrealdb

Submitted to

SteelHacks XIII

Created by

I was mainly responsible for the AI processor and overall architecture design of the project. This module reads news data from the database and processes it through a three-layer AI / machine learning structure. The first layer uses the NVIDIA Nemotron API and local AI models to summarize articles and extract semantic information. The second layer uses Transformer Embeddings to convert news articles into vectors and maintain their distances and similarities inside a Semantic Vector Space. The third layer uses K-means, similarity search, and a Bucket system to automatically group articles, discover topics, and perform Topic Synthesis. After processing, the structured data is sent through a unified interface for the backend to integrate and output. The entire AI module was designed as a highly decoupled “black box,” so the outside system only needs to care about the input and output, while the internal API, local models, and machine learning algorithms can be replaced or adjusted independently. I was responsible for much of the core code, data structures, API / Server interfaces, and the complete processing pipeline. I also worked closely with my teammates to communicate requirements, data formats, and interfaces, and to connect the AI, database, and backend parts of the project.

RaineyMisty
I was in charge of the backend server + database stack for this project. I created the entire SurrealQL schema and Axum server, as well as the scripts to ingest processed data into the app's database. I also played a significant role in determining the overall project idea and feature list. As a first-time hackathon goer, this was definitely a new and exciting experience to me, and I'm glad I got to work with such an awesome team for this project!

Tristan Murphy
I worked on extracting and preprocessing the data. It was my first time working with an API and a huge dataset (queries can take up several TB). I learned about setting up BigQuery API, working with variables in different formats (comma separated, XML), using trafilatura for webscraping, and using the python package for filtering articles with broken links and certain word counts. I'm glad to experience a project coming to life working with other people of different backgrounds and skills! I contributed to the project idea and market research.

Kelly H.
deep kotecha