Inspiration
Creating YouTube Shorts involves much more than generating a video. Creators have to research ideas, write scripts, create narration, find visuals, edit everything into a vertical format, add captions, and prepare the content for publishing. We wanted to explore whether an AI agent could coordinate this entire workflow instead of simply generating one piece of content.
That idea led to ShortsPilot AI — an autonomous AI production team for YouTube Shorts.
What it does
ShortsPilot AI takes a creator's idea and turns it into a production plan and finished Short. The workflow can handle:
Topic and audience understanding Hook and script generation Scene and visual planning AI narration Stock footage selection Vertical video rendering with FFmpeg On-screen captions Video production tracking Preparation for YouTube publishing
The goal is to reduce the amount of repetitive manual work required to produce consistent short-form content.
How we built it
The project combines a web interface with a Python-based agent backend. The frontend provides the creator experience, while the backend handles the privileged production workflow.
We used Google ADK to build the ShortsPilot agent and Gemini as the reasoning model. The video pipeline uses tools such as FFmpeg, stock footage sources, and text-to-speech to transform the agent's plan into an actual vertical video.
The application is deployed with a Vercel frontend and a separate backend service, allowing the browser to remain lightweight while the production work happens server-side.
What we learned
One of the biggest lessons was that building an agentic application is not just about generating good AI responses. The difficult part is connecting multiple steps into a reliable workflow.
We learned how to:
Build an agent around a real production workflow Connect AI planning with deterministic video processing Handle authentication and server-side operations Work with FFmpeg for programmatic video rendering Optimize video generation for limited computing resources Deploy a full-stack application across multiple services Debug real production issues involving APIs, deployment, and environment variables
What's next for ShortsPilot AI
ShortsPilot AI is still a foundation release, but the long-term goal is to make it a genuinely autonomous content-production assistant.
Future improvements include faster generation, better caption synchronization, stronger content research, richer video generation, creator-specific style learning, analytics-driven topic selection, and more reliable automated YouTube publishing.
Ultimately, we want ShortsPilot AI to feel less like a collection of AI tools and more like a supervised autonomous production team that creators can delegate work to.
Log in or sign up for Devpost to join the conversation.