Scrapeit

Be super-ready before a hackathon! Your Hackathon Buddy!

Comment

Inspiration

Every time I scroll through Luma events (especially hackathons), my brain does this fun little thing where it opens 20 tabs and then immediately forgets why it opened 20 tabs.

I kept running into the same problem: “This looks cool… but should I attend? Can I realistically build something strong for this? Do I even have time that day?” I wanted a way to turn a messy list of events into a clear recommendation based on:

  • My skillset + past projects
  • Sponsor relevance (so I can align ideas with sponsor tracks)
  • My availability (so I don’t overcommit and regret it later)

That became the core idea behind Scrapeit.

What it does

Scrapeit is an agent that:

  1. Scrapes Luma events (example: https://lu.ma/sf) and extracts key details:
  • event title, date/time, location, registration link
  • sponsors
  • tags/keywords (AI, hackathon, founders, security, etc.)
  1. Scrapes sponsors’ GitHub pages to understand what they actually build / maintain:
  • repos, topics, tech stack hints, SDKs, integrations, APIs
  • what kind of projects they tend to support
  1. Uses Claude (Sonnet 3.5 + Haiku) to:
  • summarize events
  • map sponsor tracks to realistic project ideas based on my experience
  • recommend whether I should attend or skip
  1. Checks my calendar availability (or time windows I provide) and tells me:
  • ✅ Attend + best time plan
  • ⚠️ Attend but only if…
  • ❌ Skip because conflict / low relevance / too risky for my skill alignment

So instead of “random FOMO scrolling”, it gives me a ranked digest of what’s worth my time and why.

How we built it

Core stack

  • Retool Agents (A2A) for orchestration and tools

  • Claude models for reasoning + summarization:

    • Claude Sonnet 3.5 for deep analysis and recommendations
    • Claude Haiku for fast extraction + lightweight summaries

  • Auth0 for authentication and user access control

  • Sponsor integrations (3): Retool, Auth0, Claude

The flow (end-to-end)

  1. Frontend (React chat UI) A simple chat interface: “Show me hackathons this weekend” or “Summarize AI events today”.

  2. Backend (Node/Express gateway) Keeps secrets safe (Retool API key), calls Retool A2A endpoints, and polls until results are ready.

  3. Retool Agent

  • Tool 1: Luma scraper (event list + event details)
  • Tool 2: Sponsor extractor (sponsor names + links)
  • Tool 3: GitHub scraper for sponsors (repos/topics)
  • Tool 4: Claude-powered ranking and recommendation engine
  • Tool 5 (optional): calendar/availability check logic
  1. Claude reasoning layer
  • Haiku: quick “digest + extraction”
  • Sonnet 3.5: “should I attend + what can I build + risk assessment”

This setup lets Scrapeit behave like a decision-making assistant instead of a raw scraper.

Challenges we ran into

  • A2A request schema mismatch The Retool A2A endpoint required a specific message structure (message.kind = "message"), and small payload mismatches returned confusing 400s.

  • Long-running tasks Some scraping + analysis runs took 90+ seconds, which caused frontend aborts/timeouts. Fix was to:

    • remove strict timeouts
    • use 15-second polling
    • stream status updates via repeated /status calls

  • Keeping debugging human-readable When something fails, you don’t want “state=working forever” with no clue why. So we added:

    • request IDs (reqId)
    • full raw task JSON visibility
    • step-by-step logs: agent-card → message:send → task polling

  • Sponsor understanding Sponsor pages can be vague. GitHub helps, but it still takes interpretation to map “what they build” into “what I can build quickly at a hackathon.”

Accomplishments that we're proud of

  • Built an end-to-end agent pipeline that goes: Luma → sponsors → GitHub → Claude reasoning → attendance recommendation ✅

  • Made it realistic, not motivational-poster AI: Scrapeit factors in time availability and skill alignment, not just “this sounds exciting”.

  • Integrated 3 sponsor ecosystems cleanly:

    • Retool (agent orchestration + A2A)
    • Claude (Sonnet 3.5 + Haiku reasoning)
    • Auth0 (authentication)

  • Debugging that actually helps Instead of hiding complexity, it exposes where things are stuck and why.

What we learned

  • Agents aren’t hard — reliability is hard. The difference between a demo and a usable product is observability: logs, polling, retries, and clear states.

  • Model choice matters

    • Haiku is perfect for fast summarization and extraction
    • Sonnet 3.5 shines when you need: “should I attend?” + “what should I build?” with tradeoffs and reasoning

  • Scraping is messy Real web data is inconsistent. You need guardrails and fallbacks.

  • Decision support > information A list of events isn’t helpful. A ranked list with reasons is.

What's next for Scrapeit

  • Calendar integration (real) Sync directly with Google Calendar to compute conflicts automatically.

  • Personal skill profile Maintain a lightweight “builder profile” of my tech stack + past projects, so recommendations get sharper over time.

  • Sponsor track templates For each sponsor, generate “starter project templates” (repo scaffolds, suggested APIs, architecture hints).

  • Notifications Push alerts when:

    • a high-relevance hackathon appears
    • deadlines are approaching
    • an event matches my priorities + availability

  • Event follow-through After attending, Scrapeit can help generate:

    • project idea shortlist
    • MVP scope plan
    • submission write-up draft (this exact template, but auto-filled)
Share this project:

Updates