RayFlow AI — The Intelligent Creative Studio

Inspiration

I'm a short filmmaker and an engineering student at IIT Patna. Through my filmmaking experiences and participation in competitions and hackathons, I repeatedly encountered a problem: bringing a creative vision to life often meant switching between multiple disconnected platforms.

Whenever I had a story in mind, I needed different tools for brainstorming, developing scripts, planning scenes, exploring visual references, deciding cinematic styles, finding music, and finally editing the footage. This fragmented workflow consumed valuable time and frequently interrupted the creative process.

I wanted to solve a problem I had personally experienced. Instead of building just another video editor, I envisioned an intelligent creative studio that could understand a creator's preferences and help transform their ideas into visual stories.

That idea became RayFlow AI — The Intelligent Creative Studio.

The biggest inspiration behind its intelligence came from one question:

What if an AI video editor could understand not just what you ask it to do, but also what you love?

That's where Qloo came in.

We wanted to move beyond generic AI editing instructions and build an experience where cultural preferences, creative identity, and personal taste could influence how videos are edited.

What it does

RayFlow AI is an agentic, Qloo-powered video-editing studio that uses cultural intelligence to create personalized editing experiences.

Its core innovation is the integration of Qloo Taste AI, which helps uncover the cultural connections behind a creator's preferences and translates them into personalized creative direction.

Here's how it works:

Discover Your Taste: Creators enter their favourite movies, directors, artists, genres, and moods to establish their creative identity.
Qloo Cultural Intelligence: Qloo's Search, Tags, and Insights APIs help resolve preferences into entities and tags, uncovering cultural associations and taste signals.
AI-Powered Creative Direction: Google Gemini interprets these insights and translates them into editing decisions such as pacing, color mood, transitions, motion, and music direction.
Personalized Auto Reel: RayFlow creates an initial reel based on the creator's taste profile.
AI Director: Creators can give natural-language instructions such as "Make this more cinematic" and review proposed editing changes before applying them.
Manual Video Editing: A fully editable timeline supports trimming, reordering, transitions, titles, and color presets.
Music and Captions: Audio beat analysis enables beat-synchronized cuts, while optional Faster-Whisper integration supports speech-to-text captions.
MP4 Export: Creators can render and export their final videos.

Unlike text-to-video tools, RayFlow works with actual uploaded footage. Its AI-generated editing decisions become a real, editable timeline, keeping the creator in control throughout the process.

The central workflow is:

You Said → Qloo Inferred → AI Decided → RayFlow Executed.

How we built it

We built RayFlow around three major intelligence layers that connect cultural understanding, AI reasoning, and actual video execution.

1. Cultural Intelligence — Qloo Taste AI

We integrated Qloo's Search, Tags, and Insights APIs to understand the relationships between a creator's stated preferences and broader cultural interests.

For example, if someone loves Christopher Nolan's films and Hans Zimmer's music, Qloo helps uncover associated cultural signals rather than treating those preferences as isolated keywords.

This allows RayFlow to build a richer understanding of a creator's taste and use it as the foundation for personalized editing recommendations.

2. Agentic Reasoning — Google Gemini

We use Google Gemini 2.5 Flash to reason over the cultural insights returned by Qloo and generate a structured editing strategy.

Instead of simply generating descriptive suggestions, the AI translates creative intent into actionable editing decisions.

We implemented Pydantic validation to ensure the AI only produces editing instructions that our video editor actually supports. This makes the agent's decisions more reliable and executable.

3. Video Execution — RayFlow Editing Engine

The editing engine converts the validated strategy into an editable timeline.

FFmpeg and FFprobe handle video processing, rendering, and export. Librosa performs audio beat analysis, while Faster-Whisper provides optional transcription capabilities.

This connects AI-generated creative direction with actual editing operations, allowing creators to inspect, modify, and control the final output.

Technology Stack

Frontend: React 19, TypeScript, Vite, Tailwind CSS
Backend: Python, FastAPI, Pydantic
AI: Qloo Taste AI, Google Gemini 2.5 Flash
Video Processing: FFmpeg, FFprobe, Librosa, Faster-Whisper
Cloud Infrastructure: Google Cloud Run, Cloud Storage, Cloud Run Jobs, Secret Manager, Artifact Registry

We deployed the application on Google Cloud Platform using separate frontend and backend services, with a dedicated Cloud Run Job for rendering videos.

How we used OpenAI Codex

OpenAI Codex played an important role in our development workflow.

We used Codex as an engineering assistant to work across the existing codebase, understand application architecture, develop and refine implementation logic, troubleshoot technical issues, and improve the overall development process.

Rather than treating Codex as a tool for generating isolated code snippets, we used it to support the development of an interconnected application involving frontend components, backend services, AI integrations, video processing, and cloud infrastructure.

Our focus was on using Codex to accelerate engineering work while maintaining control over architectural decisions, implementation, and validation.

The specific contributions and development examples should be documented using the actual Codex development history.

Challenges we ran into

Building an intelligent creative studio came with several challenges:

Understanding subjective creativity: A simple prompt cannot fully capture someone's creative taste. Integrating Qloo helped us explore cultural associations and translate them into more personalized editing decisions.
Connecting cultural intelligence with AI reasoning: We had to design a reliable pipeline that converts Qloo's structured insights into meaningful instructions for Gemini.
Keeping AI grounded: We didn't want the AI to suggest editing operations that our editor couldn't execute. We addressed this through structured schemas and Pydantic validation.
Building a real editing engine: Generating an editable timeline instead of simply producing a rendered video required careful integration between the AI agent, editor runtime, and FFmpeg.
Handling cloud rendering: Video rendering is resource-intensive. We separated rendering into Cloud Run Jobs and used Cloud Storage to coordinate media files and render tasks.
Privacy and reliability: We ensured user-uploaded videos, images, and audio are never sent to Qloo. Only cultural preference information and derived identifiers are used for taste analysis. We also implemented fallback strategies so manual editing and exporting remain available without AI credentials.

Accomplishments that we're proud of

Built an AI-powered video-editing studio that combines cultural intelligence, agentic reasoning, and actual video processing.
Successfully integrated Qloo Taste AI to go beyond basic keyword-based personalization.
Created a three-layer intelligence pipeline: You Said → Qloo Inferred → AI Decided.
Implemented a taste-aware AI Director and personalized Auto Reel workflow.
Ensured AI-generated editing strategies are validated and converted into editable timelines.
Built manual editing and MP4 export capabilities using our own editing pipeline.
Deployed the application on Google Cloud Run with dedicated rendering infrastructure.
Developed an architecture with fallback mechanisms and privacy-conscious handling of user media.

Most importantly, we built something inspired by a real problem in our own creative journey.

What we learned

This project taught us that building an AI-powered product involves much more than integrating an LLM.

We learned how cultural knowledge graphs can complement generative AI, how to design agentic workflows, how to validate AI-generated decisions, and how to connect intelligent reasoning with real-world software execution.

Working with Qloo particularly changed how we approached personalization. We realized that understanding a user's preferences and cultural context can make AI interactions much more meaningful than relying on generic prompts alone.

We also gained practical experience in video-processing pipelines, asynchronous cloud rendering, API integration, system architecture, and deploying production applications on Google Cloud.

Working with OpenAI Codex also gave us an opportunity to explore how AI-assisted software engineering can support the development of a complex, multi-component application.

Above all, we learned that AI should not replace human creativity. It should amplify it.

What's next for RayFlow AI — The Intelligent Creative Studio

Our current implementation focuses on personalized, agentic video editing. But our larger vision goes beyond editing.

We want RayFlow to become a complete creative filmmaking workspace where creators can move from an initial idea to a finished visual story without constantly switching platforms.

Our roadmap includes:

AI Story Development: Turn initial ideas into structured storylines and scripts.
Intelligent Storyboarding: Plan scenes, shots, camera angles, and visual sequences.
Visual Research: Discover references, cinematic moods, and creative directions.
AI Pre-production Assistant: Organize scenes, shot lists, and production plans.
Deeper Qloo Integration: Use cultural intelligence to make creative recommendations more context-aware and personalized.
Advanced AI Editing: Expand the AI Director's capabilities while preserving complete creative control.

We started RayFlow to solve a problem we personally faced as creators. We want to make professional creative workflows more accessible, connected, and intelligent.

RayFlow AI — Your creative vision. One intelligent studio.