DuckDuckGo Aggregator Agent

Helps you find the best results from Internet and curated with OpenSource LLM's

Comment

🧠 Inspiration

We were inspired by the inefficiencies and overload in traditional search engine experiences. With so much information scattered across various sources, we wanted to create an intelligent agent that not only fetches search results but scrapes, reads, and summarizes the most relevant information into one coherent response. By combining the open accessibility of DuckDuckGo with the reasoning power of Llama 3 via Nebius, we built a conversational research companion that feels more human and context-aware.

🔍 What it does

The DuckDuckGo Aggregator Agent searches the web using DuckDuckGo, extracts content from the top 5 websites, scrapes up to 3000 characters from each page, and sends the aggregated content to an LLM (Llama 3) hosted on Nebius for summarization. It then delivers a detailed, concise, and contextual summary to the user, saving hours of reading and filtering.

🛠 How we built it

  • Python with the uAgents framework to create autonomous, message-passing agents.
  • Used BeautifulSoup to scrape meaningful content from search result pages.
  • Integrated the DuckDuckGo HTML endpoint to avoid API restrictions.
  • Summarization handled by Nebius API using Meta Llama 3 70B.
  • Built multiple agents: one to send queries, another to search and summarize (aggregator).
  • Carefully handled request retries, timeouts, and scraping logic to ensure robustness.

🧗 Challenges we ran into

  • Parsing content from multiple websites consistently (each site has a different HTML structure).
  • DuckDuckGo’s search links often use redirects and encoded URLs which had to be cleaned.
  • Coordinating asynchronous message flows between agents while maintaining state.
  • API rate limits and token handling while working with large chunks of scraped content.
  • Ensuring summarization didn't lose key points across the scraped content.

🏆 Accomplishments that we're proud of

  • Successfully built an autonomous agent that mimics human-level research behavior.
  • Achieved meaningful multi-source summarization using real-time data.
  • Created a clean, decentralized agent system using uAgents.
  • Maintained performance and coherence even with long, noisy web content.

📚 What we learned

  • How to build and orchestrate agents using the uAgents framework.
  • Deeper understanding of web scraping and cleaning real-world data.
  • Practical integration of LLMs via API for summarization tasks.
  • The importance of prompt engineering in getting useful summaries from large models.

🚀 What's next for DuckDuckGo Aggregator Agent

  • Add citation and source tracking to summaries.
  • Build a voice-based search interface for hands-free queries.
  • Enable multi-lingual summarization.
  • Incorporate ranking mechanisms for more trustworthy sources.
  • Expand into a research assistant bot that can dig deep across multiple pages on request.

Built With

Share this project:

Updates