built-in-ai-and-janus-pro

Inspiration

The core inspiration stemmed from the desire to create a fully self-contained, offline creative loop within the browser. We observed the emergence of two powerful client-side AI capabilities: the highly efficient, browser-native multimodal description in the Chrome Built-in AI Prompt API, and the growing ability to run large generative models like Janus Pro locally using technologies such as WebGPU and Transformers.js. Our goal was to prove the feasibility of linking these two distinct, massive AI tasks—perception (description) and creation (generation)—into a seamless, zero-cloud pipeline that offers maximum speed and privacy.

What it does

"Built-in-AI-and-Janus-Pro" demonstrates a powerful, two-step image-to-image pipeline that operates entirely offline once the models are cached:

1. Offline Image Description: Utilizes the Chrome Built-in AI Prompt API to analyze a user's image and generate a detailed, exact text description of the content. This leverages the browser's on-device multimodal model for fast, local perception.
2. Offline Image Generation: Takes the exact text prompt from the Built-in AI and passes it directly to the Hugging Face Janus Pro Image Generation model (specifically, the DeepSeek-R1-webgpu variant).
3. Client-Side Creation: The Janus Pro model, running locally in the browser via WebGPU, generates a completely new image based on the descriptive text, making the entire image-to-image workflow private, fast, and independent of external APIs after the initial model download.

How I built it

The project was built using standard web technologies with a heavy reliance on modern client-side capabilities:

• HTML, simple JavaScript, inline CSS: For the basic UI and core logic, adhering to a preference for simple code for teaching purposes.
• Chrome Built-in AI Prompt API: Integrated for the initial, local, multimodal image description.
• Transformers.js and WebGPU: Leveraged to run the massive Janus Pro 1B ONNX model (approx. 2.4 GB) directly in the browser's memory, enabling the crucial offline image generation capability.
• Local Caching Model: The architecture relies on the browser's cache to store both required models, allowing the application to function fully offline after the necessary initial download with internet access.
• Asynchronous JavaScript: Used the async and await promise format for managing the sequential, non-blocking steps of description and generation.

Challenges I ran into

• Massive Initial Download: The primary hurdle is the mandatory model downloads for both the Chrome Built in AI and the one-time download of the Janus Pro model (~2.4 GB), which is necessary to enable the crucial offline capability.
• WebGPU/Client Performance: Running a large image generation model like Janus Pro simultaneously with a Chrome Built-in AI model is highly resource-intensive, requiring careful optimization.
• Browser Dependency: The entire workflow is dependent on a modern browser (Chrome) that supports both the experimental Built-in AI Prompt API and the high-performance WebGPU technology.

Accomplishments that we're proud of

• Achieving True Edge AI: Successfully demonstrating a functional, high-fidelity AI pipeline where both the input analysis and the output generation occur entirely on the client device and offline.
• Seamless Model Chaining: Creating a clean interface for chaining two complex AI models (multimodal description --> text-to-image generation) locally.
• High Performance via WebGPU: Showcasing the power of modern browser APIs like WebGPU to handle computational loads previously reserved for dedicated servers.

What I learned

• The Power of Caching: The initial large download is a worthwhile trade-off for the performance, privacy, and reliability of an offline application.
• WebGPU's Potential: Gained an appreciation for the vast potential of technologies like WebGPU and Transformers.js to push sophisticated AI models to the edge device.
• Resource Trade-offs: Reinforced the understanding that client-side AI demands a significant upfront resource commitment (the 2.4 GB model download).

What's next for built-in-AI-and-janus-pro

• Code Refactoring: A priority is to refactor the entire JavaScript codebase to use descriptive my prefixed, camelCase variables and function names (e.g., myImageInput, myGenerateImage()), consistent with preferred simple teaching style.
• Clearer Status Indicators: Improving the user interface to provide clearer, real-time feedback on the model download status, caching success, and the current WebGPU generation progress.
• Model Selection: Exploring the possibility of letting users choose between different, smaller cached models to reduce the initial download hurdle.

Built With

• chrome-built-in-ai
• huggingface
• javascript
• transformersjs