CrowdSurge: Real-Time Edge Polling for Streamers

A real-time live polling tool for creators. Combines Vercel Edge functions with sharded AWS DynamoDB counters to handle massive voter surges with zero lag, synced instantly to an OBS overlay.

Comment

Inspiration

Interactive live streaming overlays often lag or crash when thousands of viewers vote at the exact same second. Standard relational databases immediately buckle under the write contention of a sudden "surge" of votes on a single poll option.

I wanted to build a live polling system that could handle a massive wave of votes from a global audience—like a major Twitch or YouTube stream—maintaining sub-10ms server response times and providing a perfectly synced, zero-lag OBS overlay for the creator.

How I Built It

I built a hybrid serverless architecture designed for extreme write-throughput and cost-efficiency:

  1. The Edge & Pinned Compute: I deployed Next.js on Vercel. To eliminate cross-ocean database round-trips (RTT), the vote-casting endpoint is pinned to the database region iad1 in Virginia.
  2. DynamoDB Write Sharding: To prevent partition throttling on a single database key, I sharded the vote counters. Instead of incrementing a single option key, I distribute writes randomly across 5 sub-shards (from option_0 to option_4). The total votes for an option are aggregated on read by summing the shards: Total Votes = shard_0 + shard_1 + shard_2 + shard_3 + shard_4
  3. Real-time WebSockets: I integrated Pusher to broadcast vote events. The OBS overlay client ingests these WebSocket events directly to update the UI instantly, completely bypassing database read-storms.
  4. Event-Driven Archiving: When a poll is closed, a DynamoDB Stream captures the write event and triggers an AWS Lambda function. The Lambda unmarshalls the record, aggregates the sharded counters, generates a temporary RDS IAM token, and archives the poll into AWS Aurora PostgreSQL.

Challenges I Faced

  • The "Hot Partition" Bottleneck: High-concurrency voting on a single option creates a write bottleneck in DynamoDB. I solved this by implementing the 5-way write sharding algorithm shown above, distributing the load across multiple physical partitions.
  • Cross-Ocean Latency: During testing from India, client-to-server RTT was ~240ms. I resolved this by shifting my compute from global Edge functions (which executed near the client but had to ping the Virginia database across the ocean) to a standard Node.js serverless runtime pinned directly to iad1 (Virginia), dropping database write latency to 23ms.
  • Secure Serverless Authentication: Managing database passwords securely in serverless environments is a hassle. I configured AWS OIDC and the RDS IAM Signer to generate rotating, short-lived tokens, eliminating hardcoded database passwords entirely.

What I Learned

I learned how to design databases for high-throughput write workloads and how to optimize regional routing for serverless functions. I also learned how to build real-time visualizers that track data flowing through DynamoDB Streams and AWS Lambda in real time.

Built With

Share this project:

Updates

Submission history