AI Visual Zine Editor

Watch, debate, publish: A live AI co-editor that analyzes a public YouTube video, collaborates with the user to shape the editorial angle, and publishes the result as a multimedia issue.

Comment

Inspiration

AI Visual Zine Editor originated in a simple frustration: most AI tools can summarize media, but very few can help turn it into something worth publishing.

I imagined a workflow that could begin with a public YouTube video—a music video, runway film, or visual essay—and end not in a disposable chatbot reply, but in a finished editorial issue. Instead of treating Gemini's extraordinary video processing capabilities as a mere novelty, I decided to position it as the opening act in a larger creative process: watch and comprehend the audiovisual source, debate its implications, shape an editorial angle, bring in visual references, synthesize it all, and finally publish the result.

This multimodal editorial workflow was designed to move from sensory interpretation to publication. It first analyzes a given public video. Then, it opens an editorial conversation with the user. When reference images are attached during the discussion, they are treated as real layout assets. Once the user settles on an editorial angle and chooses to publish, the system turns the entire dialogue into a cohesive multimedia zine.

What it does

AI Visual Zine Editor accepts a public YouTube URL, initiates an editorial conversation around it, and turns that dialogue into a polished publication. If the user opts to include background music via a simple checkbox, the zine is generated with a soundtrack tailored to its thematic core.

When provided with a public YouTube URL, the system deciphers the source material, identifies compelling editorial frames, and produces an initial critical reading of the video’s language, mood, and themes, focusing primarily on its visual elements. From there, the user can freely challenge the AI’s interpretation—pushing for sharper criticism, deeper symbolism, more detailed multi-sensory observation, or even an entirely different thesis statement.

Users can upload their own reference images during the conversation as well. These are not treated as disposable chat attachments. They are preserved as editorial assets and carried into the compositional layout alongside source frames, giving the user’s visual direction tangible layout influence in the final issue.

When the user is ready, the app generates a multimedia zine that includes:

  • A magazine-style title and deck
  • A long-form editorial grounded in the source material
  • Curated source frames and uploaded reference images with contextual captions
  • A generated visual backdrop aligned to the tone of the piece
  • An optional Lyria-generated instrumental soundtrack for the browser-based issue
  • Exportable HTML, PDF, and Markdown outputs

As the eventual issue derives from both the original audiovisual material and the live conversation around it, the work feels argued, curated, and publication-ready rather than randomly generated. In other words, the depth of the user's conversation directly determines the zine's direction and final editorial quality.

How I built it

This co-editor is built in Python and Streamlit, with Google Cloud Vertex AI as the multimodal model backend. Streamlit provides the interface, while the local Python app coordinates the multimodal publishing pipeline.

For source ingestion, the app uses yt-dlp to process public YouTube videos. OpenCV handles frame extraction and scoring so the system can avoid weak transitional frames and surface stronger editorial images. User-uploaded reference images are normalized with Pillow and carried through the pipeline as stable assets for layout and publishing.

The core reasoning layer runs on Gemini 3.1 Pro Preview through Vertex AI. It manages source-video understanding, editorial conversation, editorial frame selection, and final issue generation. The Gemini Flash Image family generates a tone-matched backdrop for the zine. Lyria 2 generates an optional soundtrack for the browser-based edition.

As the project evolved, I ended up introducing more structured asset handling for uploaded reference images. Instead of relying entirely on unconstrained multimodal generation throughout publishing, the pipeline now separates creative writing from asset assembly: Gemini generates the editorial content and layout intent, while Python preserves stable mappings for uploaded visual assets and cleans the final draft before rendering.

For the resulting artifact, the app assembles a custom HTML layout and exports the issue as HTML, PDF via WeasyPrint, and Markdown for portability and documentation.

The full stack includes:

  • Streamlit for the interface
  • Python for pipeline coordination
  • yt-dlp for video ingestion
  • OpenCV for frame extraction and scoring
  • Pillow for uploaded image processing
  • Gemini 3.1 Pro Preview for video analysis, editorial conversation, frame selection, and publication
  • Gemini Flash Image family for backdrop generation
  • Lyria 2 for soundtrack generation
  • HTML + WeasyPrint for export
  • Google Cloud / Vertex AI as the AI backend

Challenges I ran into

The hardest challenge was not generation itself, but editorial coherence. It is relatively easy to build a system that just summarizes a video. It is much harder to formulate one ranging from raw media processing to an audiovisual publication that makes an intentional, grounded, and cohesive impression.

Frame extraction was an early bottleneck. Simple timestamp sampling often produced blurry transitions or awkward compositions, which immediately weakened the spread. To fix that, I built a custom extraction and scoring workflow with OpenCV, which allows the app to surface more telling, editorially useful frames.

Another problem emerged in multimodal asset handling. When the model was asked to write a long-form editorial while keeping track of multiple uploaded reference images, layout mappings could drift. Solving that demanded reworking the publishing pipeline so that uploaded images were treated as stable assets rather than loosely remembered chat context.

An unexpected hurdle in the later phases was maintaining quality in the conversation itself. Grounding the publication too aggressively through extra structural scaffolding could actually worsen the quality of live editorial discussion. Ironically, ensuring reliable publication output and preserving sharp, high-quality criticism in conversation turned out to be two distinct optimization problems. The conversational quality improved once I addressed these steps separately: keep the conversation flexible and source-aware at the same time, then apply stricter asset handling during publication.

Integrating Lyria 2 brought its own constraints. Early prompts could be too brittle or too literal, requiring refinement into safer, more abstract instructions that still matched the mood.

I took care throughout to ensure this project never felt like a pile of arbitrary AI tricks. Needless to say, the obvious challenge was orchestration: building and refining this sequential process of source-video analysis, frame extraction, editorial dialogue alongside user-uploaded references, image generation, audio generation, and multi-format export into one seamless workflow.

Accomplishments that I'm proud of

This AI video critique tool doesn't stop at mere summarization or interpretation. It closes the loop from perception to publication. Instead of a fragmented chatbot response on a video, AI Visual Zine Editor crafts a full multisensory edition: a title and deck, long-form writing, curated source frames, uploaded references, a generated backdrop, an optional soundtrack, and multi-format documents across HTML, PDF, and Markdown.

I can proudly state that its conversation function is not cosmetic at all. The user’s interaction with Gemini shapes the editorial identity. The publication resembles an evolving thesis rather than a static first-pass summary.

Another accomplishment was turning debugging into product improvement. As I tested the system more proactively, I encountered some places where multimodal context could drift—especially around uploaded image handling and visible internal placeholders. Reworking those parts of the pipeline after having already completed the demo was not pleasant, but definitely made deliverables cleaner, more readable, and more publishable.

Integrating multiple modalities to serve a single, cohesive editorial vision is what defines this project best. Video comprehension, image-aware reasoning, text generation, music generation, and document rendering all blend together harmoniously into a complete publication. My intention was for the interactive process itself to be the main appeal, with the final multimedia zine serving as a rewarding bonus.

What I learned

The biggest lesson from building this is that the most compelling aspect of a creative AI experience is not full automation, but collaborative refinement. The project progressed much better once I stopped assuming conversation as a secondary feature and started appreciating it as the editorial engine. The best results emerged when Gemini first analyzed the video, then discussed it with the user thoroughly, and only after that committed to publication, since the final output centers around a clear editorial policy established with Gemini in advance.

It surprised me a lot that "more grounding" does not guarantee "better conversation." Some of my later iterations aimed to be more structurally robust, but as a result, they performed worse at recall and criticism, since the model was carrying too much auxiliary scaffolding during dialogue. I eventually realized this trade-off, which led me to separate conversational quality from publishing determinism and tune them differently.

Another major takeaway was the value of stable asset handling in multimodal systems. User-uploaded images sometimes played a key role in conversation and in publication. The issue became more specific, controllable, and intentional, once I began treating them as persistent editorial assets rather than elements of transient chat.

On the technical side, the project reinforced that multimodal pipelines are significantly more reliable when each stage has a clear functional role. Video analysis, frame extraction, conversation, asset handling, publication, image generation, music generation, and export all worked better when they were ordered with distinct but connected steps.

Mediating between user input and the generation of text, images, and music for an artistic critique-zine underscored the subtle complexity of attentive prompt design, structured output, and iterative testing. It took time to notice how true advancements are made: not by adding complexity everywhere, but by knowing when to enforce strictness and when to allow flexibility.

What's next for AI Visual Zine Editor

The next step is to push AI Visual Zine Editor from a functional prototype toward a widely usable editorial tool.

On the design side, I want to deepen the layout system so the outputs feel closer to commercial-grade editorial design across print and web. For PDF, I plan to implement richer grid systems, more stylish typography, and more deliberate multi-page spreads. For web, current static pages could transform into more immersive digital storytelling media through scrollytelling, motion, and interactive components.

I am also thinking of enhancing source grounding across longer conversations without sacrificing conversational sharpness. The technique depends on how the system keeps access to or, in some cases, reverts to the original video context while preserving the editorial back-and-forth.

Another priority is finer asset control. The references uploaded by a user already strongly shape the zine, but I hope the AI could guarantee greater precision and versatility when generating the final issue.

Ultimately, these improvements will help the project evolve from a prototype into an innovative piece of editorial software, supported by a hardened cloud deployment path.

Built With

  • gemini-3.1-flash-image
  • gemini-3.1-pro-preview
  • google-cloud
  • google-genai
  • html/css
  • lyria-2
  • opencv
  • pillow
  • python
  • streamlit
  • vertex-ai
  • weasyprint
  • yt-dlp
Share this project:

Updates

Submission history