CrowdShip

CrowdShip lets users suggest and vote on new features, while AI coding agents turn the winning ideas into working software, making app development a collaborative, community-driven experience.

Comment

Inspiration

Most users never get a say in what software gets built. Feedback goes into a form, joins a backlog, and months later something ships, maybe. We wanted to close that loop live, in one room: the people using an app decide what changes next, and they watch it get built while they're still there.

Coding agents made that possible. If an agent can turn a clear spec into a pull request in minutes, the slow part of product development stops being the coding. It becomes deciding what to build and checking that it actually works. CrowdShip is about those two parts.

What it does

An organizer connects a GitHub repo and opens a session. People scan a QR code and join from their phone browser: no account, no app install, just a nickname.

  • Give feedback in two taps. "Something is broken", "This is confusing", "I have an idea" or "I like this". Inside the app being tested, a floating widget lets people point at a specific button or element and comment on it.
  • AI turns noise into proposals. Similar suggestions ("dark mode", "the white background is too bright", "I use this at night") are grouped into one proposal with requirements, acceptance criteria and a size estimate. When the AI isn't sure two suggestions match, it flags them for the organizer instead of merging them.
  • Everyone votes live. Counts and the countdown update on every phone and on the projected dashboard in real time.
  • A human approves; an agent builds. The organizer approves the winner. CrowdShip creates a GitHub issue and starts a Devin session with a structured coding spec. Audience text is fenced off as untrusted input, under fixed rules: approved repo only, no merging, no deploying, no touching CI.
  • "Draft PR opened" is not "done". CrowdShip checks that the PR is in the approved repo, follows its latest commit, and only marks the change ready for review when type-checking, unit tests and browser tests all pass on that exact commit. A new push resets it.
  • The crowd tests it. After verification, the organizer releases a preview. People try version A (current) and version B (AI-built) on their phones and answer "Did this change improve the app?"

The projected dashboard plays like a live broadcast: participants online, ideas received, the vote, the agent's progress, the PR, each check turning green, and a running activity feed.

How we built it

  • Web app: Next.js (App Router) and TypeScript for the phone app, the projected dashboard and the organizer console. Tailwind and Motion for the interface.
  • Database and live updates: Supabase Postgres holds all state; every rule is enforced in the database. Supabase Realtime pushes live updates on private, per-room channels, with access tokens issued by our server. Only the server can broadcast.
  • Background worker: a long-running Node process, with a Postgres-backed job queue (retries with backoff, duplicate protection, failed-job inspection, pause/resume). It starts and monitors the coding agent, checks CI and sends live updates.
  • Coding agents: one interface with three implementations. Devin (API v3), Claude Code running in GitHub Actions, and a clearly labeled mock provider for offline demos. The mock is never used in place of a real agent that's misconfigured.
  • AI triage: Claude with structured output validated by Zod, and a non-AI fallback that clusters by keyword and synonym, so the product works without any AI keys.
  • GitHub: Octokit for issues, PR discovery and check results; signed webhooks.
  • Demo app: a small Vite + React task board with stable test IDs, a CI workflow that uses no secrets, and one deliberate bug for the bug-report demo.
  • Testing: Vitest unit and integration tests run against an isolated database; Playwright end-to-end tests run on a phone-sized screen.

Challenges we ran into

  • Live updates broke at scale. Our first version sent a live update to every phone for every single vote. In a load test with 150 participants, the realtime service started rejecting messages and zero updates got through. We switched to batching: changes are queued, and the worker sends at most one update per room every 300 ms, with fresh vote totals. After that, all 450 expected updates arrived, the slowest in about 400 ms.
  • Venue Wi-Fi. We had limited joins to 30 per minute per IP address. At an event, almost every phone shares the venue's IP, so the 31st person would have been turned away. Abuse limits now apply per participant instead.
  • One person, one vote, without accounts. The database guarantees one ballot per participant per round, and 20 simultaneous duplicate votes still produce exactly one. Since there are no logins, though, guest sessions only stop casual repeat voting, not someone determined to vote twice. We say so openly instead of overclaiming.
  • Agents launched twice by accident. A timed-out "start session" request might or might not have started a session. Instead of blindly retrying, CrowdShip tags every session and looks it up before ever creating another. In our tests, eight simultaneous approvals produced exactly one session.
  • GitHub Actions quirk. PRs opened by the Actions bot don't trigger the usual CI workflow, so CrowdShip starts CI on the PR branch itself when no checks appear.
  • Agent repo permissions. On our first live Devin run, Devin fixed the bug and passed all the tests on its own machine, but GitHub refused its push because its app could read our demo repo and not write to it. CrowdShip correctly reported "waiting for input" rather than success.

Accomplishments that we're proud of

  • The whole loop runs end to end (feedback → AI grouping → vote → agent → PR → required checks → preview → rating) and is covered by an automated browser test.
  • Every status on screen comes from real data in the database. A failed agent never shows as completed, and a PR with failing tests can never be released as a preview.
  • Measured, not guessed: 150 simultaneous participants with zero errors, exact vote counts after 225 simultaneous vote requests, and live updates delivered within about 400 ms.
  • Humans stay in control: an organizer approves the proposals on the ballot, the winner before any agent runs, and the preview before anyone sees it.

What we learned

  • With coding agents, the hard engineering is everything around the agent: approval gates, avoiding duplicate launches, checking the result independently, and treating audience input as untrusted.
  • A real-time app should treat live messages as hints that something changed, not as the data itself. Each phone fetches the current state after every hint and after every reconnect, so a dropped message never leaves anyone showing the wrong thing.
  • Load testing early pays off. Both of our biggest scaling problems only showed up under realistic concurrency.

What's next for CrowdShip

  • Several agents building the same winning idea in parallel, with the crowd choosing the best result.
  • Proper A/B experiments after release, measuring whether people actually use the feature, not just how they rate it.
  • Voice feedback: hold one button, speak, review the transcript and send.
  • An AI product manager that explains the trade-offs between competing proposals before people vote.
  • The feedback widget published as an SDK, so any team can add CrowdShip to their own app.

Built With

Share this project:

Updates

Submission history