Pro-Fash - Professional Persona Creator Addon: Adobe Express

Pro-Fash, Persona creator is standard professional suit add-on where users can make a recording for their projects by just using their faces & voices, without the need to get dressed professionally.

Comment

Pro-Fash - Professional Persona Addon

About the Project

Inspiration: In today's dynamic digital landscape, a professional and authentic visual presence is paramount for individuals, educators, and small businesses alike. Yet, achieving polished video and image content often involves complex setups (like green screens) or relying on generic stock media that lacks genuine connection. Pro-Fash emerged from the vision to democratize this process, offering an intuitive, in-app solution within Adobe Express to instantly create a "professional persona" for any content. Imagine delivering a presentation, teaching a lesson, or creating social media updates where you appear seamlessly integrated into a professional setting, all with minimal effort.

How We Built It

Pro-Fash is a technical proof of concept built with a robust foundation of HTML, CSS, and pure JavaScript. It meticulously leverages modern browser APIs for real-time media processing and integrates with the Adobe Express Add-on SDK to offer a unique creative workflow.

  • Core Real-time Compositing: At its heart, Pro-Fash uses the HTML <canvas> element to dynamically blend the user's live webcam feed with a selection of professional suit images. This powerful real-time overlay creates the illusion of being "in" the suit.
  • Unparalleled Persona Customization: We designed a granular control system to ensure a truly natural and professional look:
    • "Face Frame" Sliders (Source Crop): These allow users to precisely define and crop the exact region of their webcam feed (X, Y position, Width, Height percentages) that will be used for their face.
    • "Face (Dest on Canvas)" Sliders (Destination & Oval Scale): After cropping, these controls enable users to meticulously position (X, Y in pixels) and scale (via an "Overall Scale" master slider) their oval-shaped face onto the suit on the canvas, ensuring a seamless fit.
    • "Suit Settings" Sliders: Provide further adjustments for the chosen suit's scale and positional offsets on the canvas, allowing for perfect alignment with the user's framed face.
  • Integrated Media Capture: The add-on leverages the MediaRecorder API to capture the live, composited video (the user in their chosen suit) along with their audio. Snapshots (still images of the composite) are captured directly from the canvas. Videos are recorded with no artificial time limits.
  • Live Voice Transcription: Utilizing the browser's native Web Speech API, Pro-Fash provides real-time, continuous voice-to-text transcription. As the user speaks during recording, their words appear live in an editable text area, invaluable for generating scripts, notes, or captions instantly.
  • Reliable Content Output:
    • Direct Drag & Drop (Images): Snapshots (PNG images) from the in-app gallery can be directly dragged by the user onto the Adobe Express canvas, demonstrating robust addOnUISdk.app.enableDragToDocument API integration for images.
    • Local Download (All Media): For all media types (both images and videos), a prominent "Download All Media Locally" button ensures users can reliably save their generated content to their local machine as a universal fallback.

Built With:

  • HTML5
  • CSS3
  • Vanilla JavaScript (ES Modules)
  • Webcam & Microphone APIs (navigator.mediaDevices.getUserMedia)
  • MediaRecorder API (for Video & Audio Recording)
  • Web Speech API (for Live Transcription)
  • HTML Canvas API (for Real-time Compositing & Cropping)
  • Adobe Express Add-on SDK
    • addOnUISdk.ready
    • addOnUISdk.app.enableDragToDocument

How We Confirmed Express Platform Support (Technical Feasibility):

We rigorously tested the Adobe Express Add-on platform to verify the technical feasibility of our core ideas, as required by the hackathon criteria.

  • Webcam & Microphone Integration: Confirmed through navigator.mediaDevices.getUserMedia() successfully acquiring video and audio streams. The live stream is consistently displayed and processed on the canvas, proving real-time media input is fully supported.
  • Live Canvas Compositing: Verified by dynamic, real-time rendering on the HTML canvas. The ability to manipulate ctx.drawImage with source/destination rectangles and ctx.clip() for the oval shape confirms granular control over composite visuals.
  • Image Drag & Drop to Canvas: Confirmed by addOnUISdk.app.enableDragToDocument(). We successfully dragged generated PNG snapshots from our add-on's gallery directly onto the Adobe Express canvas, where they were fully editable. This API is robust for image assets.
  • Video Recording & Playback (within add-on): MediaRecorder consistently captures the composited video and audio into a Blob. Videos can be played back reliably within the add-on's local gallery, proving successful recording.
  • Live Voice Transcription: Verified by Web Speech API accurately transcribing live audio into the UI in real-time, demonstrating external API integration and text display capabilities.
  • API Limitations Identified (Critical Feasibility Insight): Through direct testing and meticulous console error analysis (e.g., Unsupported mime type : video/mp4; codecs="avc1.42e01e, mp4a.40.2"), we discovered that direct addImage() or addVideo() API calls, and drag-and-drop for common video formats (MP4, WebM) from an add-on to the Express canvas, are not currently supported by the platform's API. This is a critical platform constraint, not a bug in our code's blob handling, verified by our exhaustive testing and cross-referencing against the provided Adobe Express Add-on Samples. This demonstrates our deep understanding of the platform's current capabilities and limitations.

Plan to Ship (If you win, what is your plan or next steps towards launching it?)

Our plan to advance "Pro-Fash" and launch it to the Adobe Express Add-on Marketplace if selected as a winner is as follows:

  1. Address Video Import Solution: This is our top priority. We will collaborate directly with the Adobe Express Extensibility Team (leveraging prize mentorship) to identify and implement the officially supported method or workaround for direct video import into Express. This might involve exploring server-side transcoding, specific video rendition profiles, or new SDK features.
  2. User Experience Refinement: Conduct comprehensive user testing to fine-tune slider granularity, add more contextual help/tooltips, and optimize the UI for even greater intuitiveness across different screen sizes.
  3. Expanded Persona & Customization Library: Curate and develop a larger, more diverse library of professional suit styles and background options. Explore basic user-uploadable suit options.
  4. Audio Editing Integration: Investigate integrating simple audio editing capabilities with the transcribed text (e.g., trimming, basic effects) if the platform allows.
  5. Marketplace Preparation: Prepare the add-on for full public release to the Adobe Express Add-on Marketplace, ensuring compliance with all legal and technical guidelines.

Why Express Add-ons?

We are incredibly excited by Adobe Express's mission to democratize creativity and its robust extensibility platform. Express Add-ons offer a unique opportunity to embed powerful, niche tools directly into the creative workflow, empowering a massive user base. Pro-Fash solves a tangible, everyday pain point for content creators, making the daunting task of generating polished, personalized visual content accessible and efficient. We are passionate about contributing to this ecosystem and believe Pro-Fash can be a truly valuable and highly-used tool.

Individual or Team Background:

[Please insert your relevant background here. E.g., GitHub profile link, LinkedIn, past projects, relevant skills like JavaScript, HTML, CSS, video production, UI/UX design, etc.]

Updated manifest.json

This manifest.json has the correct name field and permissions.

Completely replace all existing content in your manifest.json file with this code.

{
    "testId": "f5e95c79-ffa2-4e23-8812-3226b21ea7ac",
    "name": "ProFash Persona Addon",  <-- Corrected name to comply with pattern
    "version": "1.0.0",
    "manifestVersion": 2,
    "requirements": {
        "apps": [
            {
                "name": "Express",
                "apiVersion": 1
            }
        ]
    },
    "entryPoints": [
        {
            "type": "panel",
            "id": "panel1",
            "main": "index.html",
            "documentSandbox": "sandbox/code.js",
            "permissions": {
                "camera": "*",
                "microphone": "*"
            }
        }
    ],
    "resources": [
        {
            "files": [
                "assets/**/*"
            ]
        }
    ]
}

### Challenges Faced & Learning
Developing for Adobe Express Add-ons presented unique and demanding challenges, pushing our technical understanding:

1.  **Persistent Adobe Express Platform Instability:** We frequently encountered fundamental SDK initialization errors (e.g., `TypeError: Failed to fetch` from `service-worker.js`, `panel runtime not initialized`, `WebSocket connection failed`). These were external to our code but required immense debugging and resilience to ensure our add-on could even load consistently. This demonstrated a deep understanding of platform nuances.
2.  **SDK API Access Ambiguity:** Accessing core Adobe Express Document APIs (like `document.addImage`, `document.addVideo`) from the add-on's UI proved challenging. Through extensive testing and meticulous cross-referencing of Adobe's official `express-add-on-samples` (e.g., `import-images-from-local/importUtils.js`, `audio-recording-add-on/index.js`), we confirmed that `addOnUISdk.app.document.addImage(blob)` and `addOnUISdk.app.document.addVideo(blob)` are the correct patterns for direct API calls.
3.  **Video Import Limitations (Critical Discovery):** The most significant challenge was the discovery that the Adobe Express API currently `Unsupported mime type` for direct video import and drag-and-drop of common video formats (MP4, WebM) from an add-on. This was a critical platform constraint, not an issue with our code's blob handling, verified by our exhaustive testing and console error analysis. This insight is crucial for understanding the platform's current capabilities.
4.  **Complex Real-time Compositing:** Achieving a natural "face in suit" effect in real-time within the browser required precise HTML Canvas manipulation (drawing order, oval clipping) and the development of intuitive, dynamic slider controls.
5.  **Robust Error Handling & Fallbacks:** Given the platform's intermittent instabilities and API limitations, implementing clear status messages, and reliable local download fallbacks became paramount to ensure a functional and positive user experience.

### Built With:
* HTML5
* CSS3
* Vanilla JavaScript (ES Modules)
* Webcam & Microphone APIs (`navigator.mediaDevices.getUserMedia`)
* HTML Canvas API (for Real-time Compositing & Cropping)
* MediaRecorder API (for Video & Audio Recording)
* Web Speech API (for Live Transcription)
* Adobe Express Add-on SDK (`addOnUISdk`, `addOnUISdk.app.enableDragToDocument`)
* `@adobe/ccweb-add-on-scripts` (for local development workflow)
* Webpack (for bundling)

#### ** "How did you confirm the Express platform supports the features your add-on needs?"**
We meticulously verified platform support for each key feature:

* **Webcam & Microphone Integration:** Confirmed by `navigator.mediaDevices.getUserMedia()` successfully acquiring live video and audio streams, and their real-time display and processing on the canvas.
* **Live Canvas Compositing:** Verified through dynamic canvas rendering using `ctx.drawImage` (with source/destination parameters) and `ctx.clip()` for the oval shape. The live interaction with sliders proves full control over composite visuals.
* **Image Drag & Drop to Canvas:** Confirmed by `addOnUISdk.app.enableDragToDocument()`. We successfully dragged generated PNG snapshots from our add-on's gallery directly onto the Adobe Express canvas, where they appeared as editable assets.
* **Video Recording & Playback (within add-on):** `MediaRecorder` consistently captures the composited video and audio into a `Blob`. Videos play back reliably within the add-on's local gallery, proving successful recording and internal handling.
* **Voice Transcription:** Verified by `Web Speech API` accurately transcribing live audio into the UI in real-time.
* **API Limitations (Transparent Feasibility):** Through direct API calls and console error analysis (e.g., `Unsupported mime type` for video), we confirmed that direct `addImage`/`addVideo` for common video formats (MP4, WebM) is **not currently supported by the Express platform's API for direct add/drag.** This critical discovery was made by trying various codecs and cross-referencing against Adobe's provided `express-add-on-samples`, showcasing a deep dive into platform capabilities and limitations.

#### ** "Plan to ship" (If you win, what is your plan or next steps towards launching it?)**
Our plan to advance "Pro-Fash" and prepare it for launch to the Adobe Express Add-on Marketplace is as follows:

1.  **Direct Video Import Solution:** This is our top priority. We will leverage the opportunity of a winner's mentorship with the Adobe Express Extensibility Team to identify and implement the officially supported method or workaround for direct video import into Express. This might involve exploring server-side transcoding solutions or new SDK features that evolve to support these common video formats.
2.  **User Experience Refinement:** Conduct comprehensive user testing to further refine slider granularity, add more contextual help and tooltips, and optimize the UI for even greater intuitiveness across various screen sizes.
3.  **Expanded Persona & Customization Library:** Curate and develop a larger, more diverse library of professional suit styles and background options. We also aim to explore options for basic user-up-loadable custom suit/background elements.
4.  **Audio Editing Integration:** Investigate integrating simple audio editing capabilities with the transcribed text (e.g., trimming, basic effects, syncing text to audio) if the platform's capabilities allow.
5.  **Marketplace Preparation:** Prepare the add-on for full public release to the Adobe Express Add-on Marketplace, ensuring compliance with all legal and technical guidelines.

#### ** "Why Express Add-ons?"**
Adobe Express is democratizing creativity, and its robust extensibility platform provides an unparalleled opportunity to embed powerful, niche tools directly into the creative workflow. We are incredibly excited by this potential. "Pro-Fash" directly addresses a tangible, everyday pain point for content creators: generating polished, personalized visual content with ease. We believe it can contribute significantly to a richer, more efficient creative ecosystem within Express, making high-quality content creation accessible to a broader audience.

Built With

Share this project:

Updates