Clipper

AI agent to automatically identify viral moments in long-form videos and generate engaging, edited clips for social media.

Comment

Inspiration

The inspiration for Clipper stems from the significant time and effort content creators invest in producing long-form content. Recognizing the dominance of short-form video on modern social media platforms like TikTok, Instagram Reels, and YouTube Shorts, we saw a clear need for a tool that automates the tedious and time-consuming process of manually identifying compelling moments and editing them into shareable clips.

What it does

Clipper automates the entire workflow of repurposing long-form video content into engaging, bite-sized clips. It takes a long-form video (like a podcast, interview, or live stream), transcribes the audio, and uses an AI agent to pinpoint the most engaging and relevant moments. It then leverages an FFMPEG microservice to automatically edit these moments into captivating short videos, complete with audio-synced subtitles optimized for mobile viewing.

How we built it

Clipper was built with a modern, scalable architecture separating the user-facing frontend from the intensive processing backend.

The frontend is a Next.js application deployed on Vercel, allowing users to upload their long-form videos directly to an AWS S3 bucket.

Once a video is uploaded, an event is sent to Inngest, our queueing system, to signal that processing is needed.

The backend, a serverless Python service, is deployed on Modal and triggered by an Inngest function. This backend, running on powerful GPUs, handles the heavy lifting:

  • It downloads the video from S3.
  • It transcribes the audio to text using an AI speech-to-text model (like OpenAI).
  • It identifies key moments by analyzing the transcript with an AI model.
  • It generates clips using FFMPEG, adding animated, audio-synced subtitles.

The final clips are then uploaded back to S3, and the video's status and clip information are updated in a PostgreSQL database managed via Prisma. For monetization, Clipper uses a credit-based system, with users purchasing credits via Stripe.

The project repository is organized into /frontend (Next.js, React, Tailwind CSS, shadcn/ui) and /backend (Python, Modal, ffmpeg-python, OpenAI).

Challenges we ran into

Developing Clipper in just 24 hours for the Hack AI Toronto hackathon presented several challenges:

  • Integrating diverse technologies: We had to seamlessly connect various services like Next.js, Vercel, AWS S3, Inngest, Python, Modal, FFMPEG, OpenAI, PostgreSQL, Prisma, and Stripe within a very tight timeframe. Ensuring smooth communication and data flow between all these components was a significant hurdle.
  • Real-time video processing: Handling video transcription, AI-driven moment identification, and FFMPEG-based editing efficiently, especially within a serverless GPU environment, required careful optimization to ensure performance.
  • Audio-synced subtitles: Generating accurate and aesthetically pleasing animated, audio-synced subtitles proved to be more complex than anticipated, requiring precise timing and FFMPEG scripting.
  • Debugging distributed systems: With components spread across different platforms (Vercel, AWS, Inngest, Modal), debugging issues that arose during the workflow was a complex task, often involving tracing events and logs across multiple services.
  • Time constraints: The 24-hour hackathon limit meant rapid development, quick decision-making, and prioritizing core functionalities over extensive features or thorough testing.

Accomplishments that we're proud of

Despite the tight deadline and challenges, we are incredibly proud of:

  • Delivering a functional MVP: We successfully built a working prototype that automates the entire video repurposing workflow, from upload to clip generation with subtitles.
  • Leveraging a modern, scalable architecture: The chosen system architecture, utilizing serverless functions (Modal), a robust queueing system (Inngest), and cloud storage (AWS S3), provides a strong foundation for future scalability.
  • Integrating AI for intelligent content creation: The use of AI to pinpoint engaging moments is a core differentiator and a significant technical achievement.
  • Seamless user experience: The Next.js frontend provides an intuitive interface for users to upload and manage their videos.
  • Completing the project within 24 hours: Developing a complex, multi-service application like Clipper from scratch in such a short period is a testament to the team's dedication and skill.

What we learned

The Hack AI Toronto experience and building Clipper taught us valuable lessons, including:

  • Rapid prototyping and iteration: The importance of quickly getting a working version out and iterating based on immediate needs and challenges.
  • The power of serverless and managed services: Tools like Modal and Inngest significantly reduce infrastructure overhead, allowing developers to focus on core logic.
  • Deepening understanding of video processing with FFMPEG: Gaining hands-on experience with FFMPEG for tasks like cutting, merging, and adding subtitles was crucial.
  • Best practices for building distributed systems: Understanding how to design for resilience, monitor events, and debug across multiple services.
  • Effective teamwork under pressure: Collaborating efficiently and leveraging individual strengths to achieve a common goal within a strict deadline.
  • Practical application of AI in content creation: Seeing how AI can be directly applied to solve real-world content creation problems.

What's next for Clipper

Clipper has a lot of potential for future development:

  • Enhanced AI capabilities:
    • Implementing more sophisticated AI models for sentiment analysis and topic extraction to identify even more compelling and nuanced moments.
    • Allowing users to provide specific keywords or themes for the AI to focus on when generating clips.
    • AI-driven dynamic background music and sound effects.
  • More customization options:
    • Providing users with more control over subtitle styles (fonts, colors, animations).
    • Enabling users to select specific segments for clipping manually, in addition to AI suggestions.
    • Adding options for intro/outro branding and watermarks.
  • Support for more platforms and formats:
    • Direct integration with social media platforms for one-click publishing.
    • Support for different aspect ratios and video resolutions optimized for various platforms.
  • Monetization enhancements:
    • Introducing subscription plans in addition to the credit-based system.
    • Offering premium features for higher-tier plans.
  • Community features:
    • Allowing users to share their generated clips directly within the Clipper platform.
    • Providing analytics on clip performance.
  • Improved user interface and experience:
    • A more robust dashboard for managing uploaded videos and generated clips.
    • Real-time progress tracking for video processing.
  • Advanced FFMPEG features: Exploring more complex video effects, transitions, and audio manipulations.

Built With

Share this project:

Updates