Inspiration
LARPStudio was born from the intersection of two distinct passions: cloud engineering and dramatic performance.
- From Classroom to Production Cloud: One half of our team wanted to take the AWS concepts we have been studying in class (CloudFormation infrastructure-as-code, IAM least-privilege roles, DynamoDB Global Secondary Indexes, SQS dead-letter queues, SSM Parameter Store, and CloudWatch alarms) and push them out of textbook diagrams into a resilient, live production deployment.
- Pursuing the Dream of Voice Acting: The other half of our team dreams of becoming an actor and spends free time practicing monologues and character impressions to break into dub voice acting. Practicing timing and delivery alone is tough because nobody gives you real-time feedback.
We realized we could build a voice-over party game and acting coach that solves both: "code scores your timing; an AI judge scores your taste."
What it does
LARPStudio is an iPad voice-over party game where players watch a ~15-second public-domain or Creative Commons movie clip (Tears of Steel, Night of the Living Dead, and His Girl Friday), improvise the dialogue live, and receive a hybrid scorecard in about 15 seconds.
- Live iPad ADR Booth: Players grant one-line consent, enter a nickname, preview the original scene, and record their dub using the iPad's built-in microphone with a live reactive soundwave and instant replay.
- Hybrid Scoring System:
- Deterministic Timing (0–100): The iPad runs an RMS-energy voice-activity detector on the mic input, and the server compares the player's speech intervals against hand-checked Silero VAD reference windows (with a 0.2-second grace buffer).
- Multimodal AI Judging (
gemini-4-hydrogen/gemini-3.5-flash): Instead of using speech-to-text—which strips away tone, inflection, and comedic delivery—Google Gemini listens directly to the raw audio alongside a beat-by-beat timeline of the scene to score Emotion and Comedy (0–10) and deliver a one-line talent-show-style Roast focused strictly on the performance.
- Visual Timing Bar, QR Share Pages & Live Leaderboard: Players see an animated scorecard reveal with a visual timing bar comparing their speech directly beneath the character's lines, scan a QR code (
/d/{take_id}) to watch and share (or delete) their dub on their phone, and compete on a live second-screen leaderboard (/board).
How we built it
We engineered LARPStudio end-to-end across SwiftUI (iOS/iPadOS), a Go (arm64) API server, and a single AWS CloudFormation stack:
- SwiftUI iPad Client (
ios/): Built withAVAudioEnginefor low-latency microphone capture, real-time RMS voice-activity detection, audio-video replay sync, and direct-to-S3 video uploads. - Go Backend & Concurrent Gemini Judging (
server/):- Runs as an always-on service on EC2 Graviton (
t4g.small) behind Caddy (automatic Let's Encrypt TLS) so live takes never suffer cold starts. - Fires two parallel Gemini API calls with strict JSON schemas: an analytical scorer (
temperature: 0) that evaluates emotion and comedy from the raw audio, and a high-temperature roaster that critiques the take alongside the player's timing score. - Includes a no-audio guard (skipping Gemini if peak volume is below −45 dBFS so silence isn't hallucinated) and automatic cross-model retry (
gemini-4-hydrogentogemini-3.5-flash).
- Runs as an always-on service on EC2 Graviton (
- Full-Stack AWS Infrastructure (
infra/template.yaml):- Zero-Server-Media Pattern: The iPad uploads the dubbed
.mp4directly to Amazon S3 via a presignedPUTURL, and/d/{take_id}signs a fresh presignedGETURL on every visit so QR links never break when temporary credentials rotate. - DynamoDB & GSIs: Stores
takes,clips, andplayerson-demand, powering/boardand per-clip leaderboards with descending GSI queries (by_clip,global_board) and zero table scans. - Security & Privacy by Design: Secrets (including the Gemini API key) live in SSM Parameter Store encrypted with KMS, accessed via a scoped IAM instance role with IMDSv2 enforced. An S3 lifecycle rule and DynamoDB TTL automatically delete all player media and records after 7 days.
- Observability & Operations: Streams Embedded Metric Format (EMF) telemetry to CloudWatch dashboards and alarms (alerting via SNS and AWS Budgets), with zero-SSH deployments handled via SSM Session Manager and Run Command.
- Zero-Server-Media Pattern: The iPad uploads the dubbed
Challenges we ran into
- Surviving Live Gemini Outages with SQS: During the hackathon, Gemini went down twice (model timeouts and prepaid credit exhaustion). We built an asynchronous recovery pipeline using Amazon SQS and a dead-letter queue: players immediately receive their real timing score with placeholder AI scores, while a background worker long-polls SQS, fetches the raw audio from S3, re-scores once Gemini recovers, and updates DynamoDB conditionally so the leaderboard re-sorts automatically without overwriting deleted takes.
- iOS
AVAudioEngine& Bluetooth Mic Traps: Enabling Apple's voice processing silently reconfigured the audio hardware and haltedAVAudioEnginewithout throwing an error, producing empty takes. Meanwhile, AirPods forced iOS into a low-quality hands-free mic profile. We fixed both by listening forAVAudioEngineConfigurationChange, adding an input-tap watchdog, locking recording to the iPad's built-in mic while keeping AirPods on high-quality A2DP output, and disabling voice processing when headphones are connected. - SwiftUI State Freeze: We hit a UI freeze on the "That's a wrap!" transition caused by SwiftUI rendering stale values during
@Published'swillSet. Using a custom simulator autopilot flag (-autoRecord) to reproduce the bug 3 out of 4 times allowed us to isolate the state update and verify a 5-for-5 fix. - Timing Window Ceiling & LLM Silence Hallucinations: Naive overlap against widened speech windows capped a perfectly timed line at 88%, so we separated the 0.2-second grace buffer from the raw reference union so a perfect take scores 100. We also discovered Gemini would invent confident scores for silent takes based on the clip's beat sheet, leading us to add a −45 dBFS audio-floor guard.
Accomplishments that we're proud of
- Classroom AWS Concepts Running in Production: Deploying a complete, automated CloudFormation stack—combining EC2 Graviton, Caddy TLS, S3 presigned URLs, DynamoDB GSIs, SQS + DLQ fault tolerance, KMS-encrypted SSM parameters, and CloudWatch alarms—under 36-hour hackathon constraints.
- Measured Live Performance: Across real player takes during the event, end-of-take to scorecard latency measured p50 6.2 s and p95 18.8 s, seamlessly covered by our "Judges deliberating…" animation.
- True Multimodal Acting Feedback: Passing raw audio directly to Gemini alongside deterministic VAD timing metrics so aspiring voice actors get genuine feedback on how they delivered a line, not just what words they said.
What we learned
- Cloud Resilience: Designing for external API failure from day one—using SQS visibility timeouts as free backoff, conditional DynamoDB writes, and cross-model fallbacks—kept our live demo running smoothly even during real API outages.
- Audio Engineering on iOS: Hardware audio routes, Bluetooth profiles, and signal-level guards matter just as much as backend architecture when building live voice applications.
- Quantitative + Creative ADR Craft: Separating objective speech-interval timing from subjective vocal emotion and comedic delivery creates a fair, transparent, and genuinely helpful practice loop for voice acting.
What's next for LARPStudio
- Language-Learning Shadowing: Letting language learners dub native-speaker clips using the exact same VAD timing scorer to practice pacing and cadence.
- Auto-Scaling Architecture: Moving the SQS re-score worker to AWS Lambda and placing the Go API behind an Application Load Balancer with an Auto Scaling group across multiple Availability Zones, plus Amazon CloudFront in front of S3 dubs for faster share-page playback.
- Spoken Roasts & Judge Consistency: Using Gemini text-to-speech so the AI judge reads its roast aloud, and benchmarking multi-run score variance to measure judge consistency.
- Community Tab: Helping Casting Directors and Actors alike review dub work and improve skills.
Built With
- amazon-cloudwatch
- amazon-dynamodb
- amazon-ec2
- amazon-sns
- amazon-web-services
- cloudformation
- cursor
- gemini
- go
- iam
- kms
- sqs
- ssm
- sts