Inspiration

Adobe Express offers powerful AI-driven video generation, but the generated videos lack audio, limiting their impact. Inspired by this gap, we created SoundSnap to enhance Adobe Express videos with AI-generated audio, delivering a complete, immersive editing experience within the platform.

What it does

SoundSnap is an Adobe Express add-on that allows users to upload videos (MP4, MOV, WebM), generate custom audio using the fal-ai/thinksound API, and add the result to the Adobe Express canvas. Users can provide a text prompt or let the AI create ambient sound that matches the video’s content.

How we built it

Frontend Tech Stack: Built with React, TypeScript, and Adobe Spectrum Web Components, featuring a clean UI with a centered "Upload Video" card and large "SoundSnap" heading. Backend Tech Stack: Developed using Node.js, Express.js, and Multer for handling video uploads. The fal-ai/thinksound API generates audio, with video storage managed via Fal AI’s storage API. API Usage: Leverages fal-ai/thinksound for audio generation, with parameters like num_inference_steps, cfg_scale, and seed for consistent results. Integration: Utilizes Adobe Express Add-On SDK to add generated videos to the canvas, with fallback logic to extract a video frame as an image if needed. Deployment Platform: Hosted on Render for scalable cloud deployment, configured with environment variables for API keys and port settings. CORS: Configured to support Adobe Express domains for seamless integration.

Challenges we ran into

Deployment Issues: Inconsistent fal-ai/thinksound API responses on Render required retries and extended timeouts. Video Compatibility: Ensured video formats and sizes (max 10MB) were compatible with fal-ai/thinksound through strict validation. Response Handling: Adjusted backend to handle variable API response structures for reliable audio generation. SDK Integration: Configured CORS and SDK to work seamlessly with Adobe Express domains.

Accomplishments that we're proud of

Seamless Adobe Express integration for adding AI-enhanced videos. Intuitive UI aligned with Adobe’s design principles. Robust backend with retries and logging to handle deployment issues. AI-driven audio generation enhancing video content.

What we learned

Handling AI API variability with retries and validation. Managing Render’s deployment constraints (e.g., network latency). Using Adobe Express SDK for smooth integration. Balancing simplicity and functionality in UI design.

What's next for SoundSnap

Add real-time progress updates via Server-Sent Events. Support more audio customization options. Implement client-side video validation. Expand video format support and analytics for user feedback.

Built With

• express.js
• fal.ai
• react
• render
• swc
• thinkcloud
• typescript