Brio

An Infinite canvas Multimodal and Miltilingual AI Workspace for Content Creators

Comment

Inspiration

Content creators today are drowning in context fragmentation. A typical creator workflow looks like this: open 15 browser tabs for research, switch to a chat app to brainstorm, jump to a doc for strategy, open a design tool for visuals, then another tool for voice notes—and somewhere along the way, the thread connecting all of it gets lost.

We’ve watched creators lose hours not to the creative work itself, but to the friction of moving between tools. Ideas live in one place, references in another, and generated assets in yet another. The “messy middle” of content creation—the space between inspiration and publish-ready output—is where most of the pain lives.

Our inspiration was simple: What if we could collapse that entire workflow into one spatial workspace where every source, every insight, and every generated asset stays connected—and where Amazon Nova powers the intelligence layer from reasoning to visuals to voice to video?

We built Brio to be that workspace: a creator-first operating system where research, strategy, content generation, and asset creation happen in one place, with Amazon Nova models on Amazon Bedrock as the core intelligence driving every step.

What It Does

Brio is a voice-native, multimodal creator workspace that turns scattered content into production-ready campaigns. It combines an infinite canvas, context-aware AI chat, and a full media pipeline—all powered by Amazon Nova.

Core Experience

  1. Ingest & Organize
    Users pull in source material from YouTube, Instagram, websites, PDFs, and images. Everything lives on an infinite canvas where nodes can be grouped, connected, and arranged spatially.

  2. Context-Aware AI Chat (Brio Assistant)
    A singleton AI chat node per workspace that understands connected nodes and groups. Users ask questions, request analysis, or trigger workflows—and Brio responds with context from the entire workspace.

  3. Creator Studio Flow
    A single prompt can kick off an end-to-end campaign pipeline:

    • Nova Pro generates a structured campaign plan (audience, pillars, hooks, script, CTA).
    • Nova 2 Omni refines image prompts for better visual output.
    • Nova Canvas generates campaign visuals (thumbnails, social cards, covers).
    • Nova Reel renders short-form video from a text prompt (when configured).

  4. Voice-First Interaction
    Nova 2 Sonic powers:

    • Speech-to-text: Voice notes are transcribed for use in chat and research.
    • Text-to-speech: AI responses can be spoken back for hands-free ideation.

  5. Deep Research
    Nova 2 Lite synthesizes multi-source research (web, Reddit, YouTube) into creator-grade reports with citations and action plans.

  6. Mind Maps
    Nova Pro turns connected content into structured mind maps with nodes, edges, and layouts.

  7. Image Studio
    Nova Canvas generates images from prompts, with optional reference images for style consistency (image variation mode).

Amazon Nova Models Used

Model Use Case
Nova 2 Lite Fast reasoning, orchestration, deep research synthesis, Creator Studio fallback
Nova Pro Campaign planning, mind map generation, agent runs, strategic artifacts
Nova 2 Omni Multimodal understanding, image prompt refinement, visual context in chat
Nova Canvas Text-to-image and image-variation generation for campaign assets
Nova Reel Async text-to-video generation for short-form content
Nova 2 Sonic Voice transcription (speech-to-text) and speech synthesis (text-to-speech)

Every major AI capability in Brio is powered by Amazon Nova on Amazon Bedrock. We use the Converse API for text, InvokeModel for Canvas/Reel, and InvokeModelWithBidirectionalStream for Nova Sonic’s multimodal voice pipeline.

How We Built It

Architecture Overview

  • Frontend: Next.js with React, infinite canvas (React Flow), and real-time Convex subscriptions.
  • Backend: Convex (serverless functions, real-time DB, file storage).
  • AI Layer: Amazon Bedrock Runtime with Nova models.
  • Auth: Clerk.

Amazon Nova Integration

We built a dedicated bedrock.ts module that wraps all Nova interactions:

  1. Converse API (Nova Lite, Pro, Omni)

    • converseWithNova, streamConverseWithNova for sync and streaming text.
    • converseWithNovaLite, converseWithNovaPro, converseWithNovaOmni for model-specific calls.
    • Fallback logic for inference profile–related errors (e.g., us. / global. prefixes).

  2. Nova Sonic (Voice)

    • generateWithNovaSonic uses InvokeModelWithBidirectionalStreamCommand for:
      • User audio input (speech-to-text).
      • Assistant text + audio output (text-to-speech).
    • Chunked audio handling for long recordings.
    • Retries for transient errors.

  3. Nova Canvas (Images)

    • generateImagesWithNovaCanvas uses InvokeModelCommand with:
      • TEXT_IMAGE for text-to-image.
      • IMAGE_VARIATION for reference-image–driven generation.
    • Support for aspect ratios (1:1, 16:9, 9:16), negative prompts, and quality settings.

  4. Nova Reel (Video)

    • startNovaReelVideoGeneration uses StartAsyncInvokeCommand for async text-to-video.
    • Output written to S3; we poll status via GetAsyncInvokeCommand and surface it in a Video node.

Intelligent Model Routing

We implemented routeNovaModel() to pick the right Nova model per request:

  • Fast mode → Nova 2 Lite.
  • Thinking mode → Nova Pro.
  • Auto mode:
    • Images in context or visual keywords → Nova 2 Omni.
    • Deep reasoning keywords or large context → Nova Pro.
    • Default → Nova 2 Lite.

This keeps latency low for simple queries and uses Pro/Omni where they add the most value.

Creator Studio Pipeline

The Creator Studio flow chains multiple Nova models:

  1. Evidence collection (Tavily, Perplexity) for web context.
  2. Nova Pro → campaign plan (JSON: title, audience, pillars, hooks, script, CTA, image concepts, video prompt).
  3. Nova 2 Omni → refines image prompts for Nova Canvas.
  4. Nova Canvas → generates images per concept.
  5. Nova Reel (optional) → starts async video generation when the user requests video output.

All outputs are created as nodes on the canvas and linked to the AI chat.

Voice Pipeline

  • Transcription: Voice nodes use Nova Sonic first; Deepgram Nova-2 as fallback.
  • Synthesis: In chat, when voice response is enabled, we use Nova Sonic to speak the assistant’s reply. Fallback to Deepgram if Sonic is unavailable.

Deep Research

  • Nova 2 Lite synthesizes evidence from web, Reddit, and YouTube into structured reports.
  • Optional Nova Act integration for browser-automation–eligible tasks (when configured).

Error Handling & Resilience

  • Inference profile fallbacks for Converse (e.g., us. / global. model IDs).
  • Retries for Nova Sonic (up to 3 attempts with backoff).
  • Creator Studio fallbacks: Nova Pro → Nova Lite for plan generation; graceful degradation when Canvas or Reel fail.

Challenges We Ran Into

1. Nova Sonic Bidirectional Streaming

Nova Sonic uses a bidirectional stream with a custom event envelope format. Parsing contentStart, textOutput, audioOutput, and handling speculative vs. final transcripts required careful state management. We also had to split long audio into chunks to stay within payload limits.

Solution: We built a robust event decoder, tracked content metadata by ID, and implemented chunked audio upload with proper ordering.

2. Inference Profile Requirements

Some Nova models require inference profiles in certain regions. We hit “on-demand throughput isn’t supported”–style errors when using default model IDs.

Solution: We added fallback logic to try us. and global. prefixed model IDs when inference profile errors occur, so the app works across different Bedrock configurations.

3. Creator Studio Orchestration

Coordinating Nova Pro (plan) → Nova Omni (prompt refinement) → Nova Canvas (images) → Nova Reel (video) in a single flow, with partial failures and user-configurable outputs, was complex.

Solution: We made each step independently fallible, collected notes for partial failures, and allowed users to toggle strategy, images, and video outputs.

4. Context Window and Token Limits

Campaign plans, research reports, and mind maps can produce large outputs. We had to balance maxTokens with quality and avoid truncation.

Solution: We tuned maxTokens per use case (e.g., 7000 for deep research, 3500 for mind maps) and truncated inputs (e.g., content slices) where needed.

5. Nova Reel Async Workflow

Nova Reel is async: we get an invocationArn and must poll for completion, then fetch the video from S3. Wiring this into our real-time Convex + React flow required careful state handling.

Solution: We created a Video node type that stores invocationArn and outputS3Uri, and added a Convex action/scheduler to poll status and update the node when the video is ready.

6. Multimodal Context for Nova Omni

Passing images from the canvas into the AI chat context for Nova Omni required resolving storage URLs, fetching images, and formatting them for the Converse API.

Solution: We built resolveReferenceImagesForNovaCanvas and integrated image content blocks into the Converse request when the context includes images.

Accomplishments We're Proud Of

  1. Full Nova Stack Integration
    We use six Nova model families (Lite, Pro, Omni, Canvas, Reel, Sonic) in a single product. Each is used for a distinct, high-value creator workflow.

  2. Intelligent Model Routing
    Our routeNovaModel() logic automatically selects Nova Lite, Pro, or Omni based on message content and context, improving both cost and latency.

  3. End-to-End Creator Pipeline
    One prompt can produce a campaign blueprint, multiple images, and an optional video—all as connected nodes on the canvas. This is a true “prompt to campaign” flow.

  4. Voice-Native Experience
    Nova Sonic powers both transcription and synthesis. Creators can speak their ideas and hear responses, enabling hands-free ideation.

  5. Robust Error Handling
    Inference profile fallbacks, retries for Sonic, and graceful degradation in Creator Studio make the app resilient across different Bedrock setups.

  6. Reference-Image Support for Nova Canvas
    We support Nova Canvas’s IMAGE_VARIATION mode with inspiration images from the canvas, so creators can steer visual style.

  7. Deep Research with Nova Lite
    Multi-source (web, Reddit, YouTube) research synthesized by Nova Lite into actionable reports shows Nova’s strength in orchestration and summarization.

  8. Real-Time UX
    Streaming responses, live status updates for Reel jobs, and Convex’s real-time subscriptions keep the experience responsive.

What We Learned

  1. Nova’s Strengths by Model

    • Nova 2 Lite: Excellent for fast orchestration and synthesis; we use it for research and routing.
    • Nova Pro: Strong for structured, strategic output (campaigns, mind maps).
    • Nova 2 Omni: Best when images are in context or when refining visual prompts.
    • Nova Canvas: Reliable for both text-to-image and image variation.
    • Nova Reel: Powerful for short-form video; async design requires clear UX for status.
    • Nova 2 Sonic: Enables true voice-first interaction; bidirectional streaming is powerful but needs careful handling.

  2. Bedrock Converse API
    The Converse API’s consistent interface across Lite, Pro, and Omni simplified our integration. System prompts, streaming, and token usage are straightforward.

  3. Orchestration Over Single-Model Calls
    Chaining Nova models (e.g., Pro → Omni → Canvas → Reel) produces better results than asking one model to do everything. Each model excels in its domain.

  4. Context Design Matters
    How we format workspace context (nodes, groups, images) for Nova significantly affects output quality. Structured prompts and clear boundaries (e.g., CONTEXT_STATUS, formattedContext) helped.

  5. Fallbacks Are Essential
    Inference profiles, regional availability, and rate limits vary. Building fallbacks (model IDs, providers like Deepgram) made the app usable in more environments.

What's Next for Brio

  1. Nova Act Integration
    We have scaffolding for Nova Act–eligible tasks in Deep Research. We plan to fully integrate browser automation for tasks like “summarize this live page” or “extract data from this dashboard.”

  2. Enhanced Nova Reel Workflows

    • Image-to-video and video-to-video when supported.
    • Better progress UX (e.g., estimated time, thumbnails).
    • Batch generation for multiple concepts.

  3. Nova-Powered Collaboration
    Multi-user workspaces with Nova summarizing changes, suggesting connections, and generating shared briefs.

  4. Smarter Caching with Nova
    Use Nova to decide when cached responses are still valid (e.g., based on context drift) to improve cache hit rates and cost.

  5. Expanded Multimodal Inputs
    Deeper Nova Omni integration: analyze video thumbnails, compare image styles, and generate alt-text and captions from canvas assets.

  6. Nova for Localization
    Use Nova to adapt campaign copy, hooks, and CTAs for different languages and locales, leveraging its multilingual capabilities.

  7. Agentic Loops
    Let Nova proactively suggest next steps (e.g., “You have 3 images; want a Reel?” or “This research suggests a follow-up query”) to reduce friction in creator workflows.

  8. Analytics & Optimization
    Use Nova to analyze performance data (e.g., from connected analytics) and recommend content improvements.

Technical Summary

Component Technology
Frontend Next.js, React, React Flow, Tailwind
Backend Convex (queries, mutations, actions)
Auth Clerk
AI Amazon Nova on Amazon Bedrock (Lite, Pro, Omni, Canvas, Reel, Sonic)
Storage Convex Storage, AWS S3 (Nova Reel output)
Search Tavily, Perplexity
Voice Fallback Deepgram

Demo Video & Links

  • Demo Video: [Link to your 3-minute demo]
  • Live App: [Link to your deployed app]
  • GitHub: [Link to your repository]

Hashtags

#AmazonNova #AmazonBedrock #Hackathon #AI #ContentCreation #Multimodal #VoiceAI

Brio — One workspace for creators who think in campaigns, shots, hooks, and systems. Powered by Amazon Nova.

Condensed Versions (for strict character limits)

Inspiration (≤500 chars)

Content creators lose hours to context fragmentation—research in one tab, strategy in another, assets somewhere else. We built Brio to collapse the entire workflow into one spatial workspace, with Amazon Nova powering reasoning, visuals, voice, and video from start to finish.

What It Does (≤500 chars)

Brio is a voice-native, multimodal creator workspace. Ingest content from YouTube, Instagram, websites, PDFs; organize on an infinite canvas; and generate campaigns with Nova Pro, Omni, Canvas, and Reel. Nova 2 Sonic enables voice input and spoken responses. Nova 2 Lite powers deep research synthesis. One prompt → strategy + images + optional video.

How We Built It (≤500 chars)

Next.js + Convex backend, Amazon Bedrock for all Nova models. We built a bedrock.ts module wrapping Converse (Lite/Pro/Omni), InvokeModel (Canvas), StartAsyncInvoke (Reel), and InvokeModelWithBidirectionalStream (Sonic). Intelligent routing picks Lite/Pro/Omni per request. Creator Studio chains Pro→Omni→Canvas→Reel in one flow.

Challenges (≤500 chars)

Nova Sonic's bidirectional stream required custom event parsing. Inference profile errors led us to add us./global. fallbacks. Orchestrating Pro→Omni→Canvas→Reel with partial failures needed careful error handling. Nova Reel's async workflow required polling + S3 integration for real-time UX.

Accomplishments (≤500 chars)

Integrated 6 Nova model families in one product. Built intelligent model routing. End-to-end Creator Studio: one prompt → blueprint + images + video. Voice-native with Nova Sonic for transcription and synthesis. Reference-image support for Nova Canvas. Deep research synthesis with Nova Lite. Robust fallbacks for production resilience.

What We Learned (≤500 chars)

Nova Lite excels at orchestration; Pro at strategy; Omni at multimodal; Canvas at images; Reel at video; Sonic at voice. Chaining models beats single-model calls. Converse API's consistency simplified integration. Fallbacks (inference profiles, Deepgram) are essential for real-world deployment.

What's Next (≤500 chars)

Full Nova Act integration for browser automation. Enhanced Reel workflows (image-to-video, batch). Nova-powered collaboration and proactive suggestions. Smarter caching with Nova. Expanded Omni for video/thumbnail analysis. Nova for localization and performance optimization.

Nova Model Reference (for judges)

Nova Model Bedrock ID Brio Use Case
Nova 2 Lite amazon.nova-2-lite-v1:0 Chat (fast), Deep Research synthesis, Creator Studio fallback
Nova Pro amazon.nova-pro-v1:0 Campaign planning, mind maps, agent runs
Nova 2 Omni amazon.nova-2-omni-v1:0 Multimodal chat, image prompt refinement
Nova Canvas amazon.nova-canvas-v1:0 Text-to-image, image variation
Nova Reel amazon.nova-reel-v1:1 Async text-to-video
Nova 2 Sonic amazon.nova-2-sonic-v1:0 Speech-to-text, text-to-speech

Built With

Share this project:

Updates