SoulCanvas

Image Editing Features at a Glance
Text-to-Image Prototype Creation
Image-to-Image Prototype Creation 1
Image-to-Image Prototype Creation 2
Identity Studio - Change Companion's Pose
Comic Story Generation
Video Generation - Wan2.2
Cute Chibi Generation
Background Removal
Virtual Try-on
Multi-Subject Scene Composition
Style Converter - Anime to/from Realistic
Image Fusion
Chat and Voice
CogVideoX Improvement

Inspiration

We are living through a global loneliness epidemic. The WHO has declared social isolation a public health threat comparable to smoking 15 cigarettes a day. Over 300 million people worldwide suffer from depression, yet fewer than 25% have access to adequate mental health support — not because therapists don't exist, but because of cost, stigma, language barriers, and sheer unavailability at 3 AM when the walls close in.

Expressive Arts Therapy — the evidence-based practice of using creative expression (visual art, storytelling, role-play) as therapeutic intervention — is one of the most effective modalities for processing complex emotions. But it requires a trained facilitator and sustained engagement. What if that facilitator could live in your pocket, available 24/7, in any language, wearing the face of a companion you designed yourself?

This matters to me on a visceral level. I was diagnosed with bipolar disorder in middle school. The illness forced me out of school for an entire year — and when I returned, I was the oldest in every classroom, carrying a silence no one around me could understand. During my darkest periods, it was my school counsellor who became my lifeline. No matter how many times I burst into her office unannounced — mid-panic, mid-spiral, sometimes unable to explain why I was there — she always welcomed me with the same gentle smile. She never made me feel like a burden for showing up. That consistency, that unconditional presence, transformed me more than any medication or textbook ever could. I recovered. And now I build so that the next kid — the one spiralling at 2 AM with no office to run to — doesn't have to face that void alone.

What it does

Therapeutic Core: Bilingual (CN/EN) counselling LLM; voice conversations; customizable companion appearance as Expressive Arts exercise

Companion Creation: Text-to-image; image-to-image style transfer (e.g., animefy a real photo); consistent pose generation; chibi version; background removal; video generation

Advanced AIGC: CogVideoX temporal consistency improvement; Flux Kontext reference similarity and style similarity enhancement; manga/comic generation; smart inpainting (virtual try-on, object removal, DIY image modification and more); multi-subject scene composition; image fusion/style blending; colorization; consistent identity generation

Professional Tools: Model switching (any SDXL-compatible model); training convergence visualization; style conversion between any two visual domains

How we built it

SoulCanvas unifies three AI pillars into a single therapeutic experience:

A Specialized Counselling LLM — A fine-tuned large language model trained specifically for therapeutic conversations, supporting both Chinese and English. It doesn't just respond — it actively listens, validates, gently challenges cognitive distortions, and guides users through Expressive Arts Therapy exercises.

A Visual Identity Engine — Powered by a novel dual-phased diffusion approach (accepted at CVPR 2026 with sole first authorship - SyntheticManga: Training-Free Manga Generation with Phased Diffusion; below the Predictive Drift Controller(PDC) is exactly the Adaptive Drift Modulator(ADM) in SyntheticManga). Users create their own companion's appearance — anime, photorealistic, 3D cartoon, or any style they desire. This isn't just customization; it's therapeutic in itself. In Expressive Arts Therapy, the act of creating your own supportive figure is a recognized exercise in self-empowerment.

A Voice Interaction Layer — Using pretrained voice synthesis (Qwen3-TTS), the companion speaks. Conversations become real dialogues, not cold text exchanges.

The Technical Innovation (CVPR 2026):

The core contribution coming from our CVPR 2026 paper is a dual-phased diffusion guidance strategy that separates the denoising process into two distinct stages:

Phase 1 — During the unstable mid-noise regime, a PID-inspired controller actively corrects spatial feature drift, locking in character identity and global composition. This solves the notorious "identity collapse" problem in stylized domain generation.

Phase 2 — In the low-noise regime, fine-grained details are refined by modeling feature anchoring as a Dirichlet boundary value problem. This ensures seamless integration of local features — hair strands, eye highlights, fabric textures — without artifacts.

This training-free approach works atop any SDXL-compatible base model, enabling unlimited visual styles without retraining.

The founding insight: denoising is not one process

Standard SDXL inference runs a U-Net for 20–50 denoising steps, applying classifier-free guidance uniformly at every step. So does ControlNet. So does IP-Adapter. The implicit assumption is that every step requires the same type of intervention.

This assumption is wrong, and exploiting that wrongness is the entire basis of our framework.

We observe — supported by prior drift-analysis work on DDPM trajectories — that denoising proceeds in three functionally distinct regimes:

High-noise regime (late timesteps, ~850–1000): pure noise; too chaotic for any stable guidance signal
Mid-noise regime (~400–850): global spatial structure, pose, and identity features are established; the model is maximally susceptible to identity drift — the corruption of reference character features due to diverging prompt and identity objectives
Low-noise regime (~0–400): macro structure is locked; the model synthesizes local high-frequency details — line art, facial highlights, fabric texture

Treating all three regimes identically is "inefficient and often counterproductive." We therefore separate our guidance into two specialized controllers, each active in exactly the regime where it can do the most work, and neither active where it would cause harm.

Phase 1: Predictive Drift Controller (PDC)

The problem it solves: In the mid-noise regime, both the target prompt and the reference identity want to steer the latent trajectory. They usually disagree. Without intervention, the model gradually abandons the reference character's face, proportions, and line style — which we term identity drift.

Drift quantification. At each denoising timestep (t), we define feature-space drift as the L1 divergence between the noise predictions generated from the current latent under the target prompt versus the same latent under the reference prompt:

$$d(t) = \left| \epsilon_\theta(z_t, t, C_{\text{target}}) - \epsilon_\theta(z_{\text{ref},t},, t,, C_{\text{ref}}) \right|_1$$

where $\epsilon_\theta$ is the denoising U-Net, $C_{\text{target}}$ and $C_{\text{ref}}$ are the text embeddings for the desired output and reference image respectively, and $z_{\text{ref},t}$ is a noisy version of the reference image latent at timestep $t$. This scalar $d(t)$ measures, in real time, how far the current generation is drifting from the reference identity manifold.

PID formulation. We do not apply a fixed correction — we feed $d(t)$ into a discrete Proportional-Integral-Derivative controller:

$$P(t) = d(t)$$ $$\quad I(t) = I(t-1) + d(t)$$ $$\quad D(t) = d(t) - d(t-1)$$ $$u(t) = 1.0 + K_P \cdot P(t) + K_I \cdot I(t) + K_D \cdot D(t)$$

Each PID term does distinct work that no single term can replicate:

P-term reacts to immediate structural divergence — if drift is large right now, intervene more aggressively right now
I-term accumulates historical drift — if misalignment has been persisting for several steps, the integral builds up and forces stronger correction, overcoming the "gravitational pull" of the target prompt that would otherwise gradually erode the identity
D-term anticipates future drift — it dampens the controller when the drift is decreasing, preventing over-correction and overshoot

This PID output $u(t)$ modulates a base blending schedule $\alpha(t)$, yielding an adaptive mixing factor:

$$\alpha_{\text{final}}(t) = \alpha(t) \cdot u(t)$$

The latent is then updated by blending the current latent toward the reference:

$$z'_{t} = (1 - \alpha_{\text{final}}(t)), z_t + \alpha_{\text{final}}(t), z_{\text{ref},t}$$

Why this works: PDC enacts the claim that high error early demands stronger regulation, and diminishing error later can be left alone. This is a mathematical realization of the same insight that underpins PID controllers in robotics, thermostats, and flight stabilization systems — applied, here, to the denoising trajectory.

PID component ablation: Removing the I-term alone degrades DreamSim by ~50% in consistent image customization (from 0.1055 to 0.1586) and ~40% in inpainting (from 0.0724 to 0.1007) — confirming that accumulated drift correction is irreplaceable for single-subject tasks. Removing the D-term most harms multi-subject composition (DreamSim 0.1561 → 0.3578), where the anticipatory damping is essential to stabilize the volatile corrections arising from multiple competing subject boundaries. Removing the P-term primarily degrades structural fidelity in image fusion. No single term dominates all tasks — all three contribute synergistically.

Phase 2: Local Harmonic Field Guidance (LHFG)

The problem it solves: Once PDC has stabilized global structure, the low-noise regime synthesizes high-frequency detail — lace patterns, eye highlights, fine hair strands. The challenge is that injecting reference details at this stage creates hard seams: the reference texture exists as a foreign object pasted into a generated field, producing blending artifacts at every boundary. Standard attention injection and feature copying both suffer from this.

The physical insight. We reframe the problem as a classical PDE. Given two requirements that appear to conflict — (i) identity preservation: anchored positions must exactly match the reference, and (ii) natural blending: unanchored positions must adapt smoothly to injected details — the mathematical structure is precisely a Dirichlet boundary value problem on the latent feature space. We solve the Laplace equation:

$$\nabla^2 F = 0$$

over the interior (Sub-Problem) subject to fixed boundary conditions (Sub-Solution). The Maximum Principle for harmonic functions guarantees that every interior point equals the weighted average of its neighbors — which means blending artifacts are forbidden by construction. Local extrema cannot exist in the interior. Smooth interpolation through the fixed anchors is not an emergent property; it is a mathematical theorem.

Sub-Solution / Sub-Problem partition.

The Sub-Solution (boundary) = positions selected as identity-critical anchors. We compute a per-position importance score $$s = \lVert F_{\text{sol}} \rVert_2$$ from the reference feature map $F_{\text{sol}} \in \mathbb{R}^{B \times C \times H \times W}$, where $F_{\text{sol}}$ is extracted from a U-Net pass conditioned on the reference prompt. The top $k%$ positions by this norm form the anchor mask $M_{\text{anchor}}$. The Sub-Problem (interior) = everything else; these positions are solved by the harmonic equation. Jacobi iteration. The solution field is initialized as: $$u^{(0)} = M_{\text{anchor}} \odot F_{\text{sol}} + (1 - M_{\text{anchor}}) \odot F_{\text{prob}}$$

where $F_{\text{prob}}$ is the current denoising feature map. We then apply the standard four-neighbor Jacobi relaxation:

$$u^{(k+1)}_{\text{prob},i,j} = \frac{1}{4}\left(u^{(k)}_{i-1,j} + u^{(k)}_{i+1,j} + u^{(k)}_{i,j-1} + u^{(k)}_{i,j+1}\right)$$

After every iteration, anchors are hard-reset to their reference values: $$u^{(k+1)} = M_{\text{anchor}} \odot F_{\text{sol}} + (1 - M_{\text{anchor}}) \odot u^{(k+1)}_{\text{prob}}$$

This strictly enforces the Dirichlet condition — identity anchors never drift regardless of how many iterations run. The converged field $F_{\text{harmonic}}$ satisfies the discrete Laplace equation in the Sub-Problem while exactly matching the reference at all anchor positions.

Why Jacobi and not Gauss-Seidel? Gauss-Seidel used in previous approaches requires forward substitution with the full discrete Laplacian matrix $L \in \mathbb{R}^{n \times n}$, giving $O(n^2)$ complexity due to the density of non-zero entries — completely unsuitable for inference-time use. Jacobi relaxation decomposes into straightforward per-position additions that are fully parallelizable, running on GPU in-place. Empirically, only 15 iterations suffice for adequate convergence. Furthermore, LHFG operates on VAE-compressed latent feature maps (typically $128 \times 128$ or smaller), not full-resolution pixel space — dramatically reducing the domain size compared to methods like PixPerfect that operate at pixel resolution. Measured overhead: LHFG contributes an average of 1.587% of total inference time across all tasks.

Phase 3: Strategic Zero-Out

After LHFG completes, we switch off all guidance for the final low-noise steps. The reasoning: PDC has locked global structure and LHFG has anchored fine detail — continued intervention at this point over-constrains the model and produces "pasted-on" artifacts where newly synthesized high-frequency texture fails to integrate with the scene's lighting and atmosphere. Liberating the U-Net in the final phase lets it synthesize detail purely from the text prompt, producing outputs that look organically generated rather than guided.

One solver, many product features

The mathematical abstraction — partition a domain into known Sub-Solution and unknown Sub-Problem, then solve the Laplace equation over the Sub-Problem — is entirely agnostic to what "known" and "unknown" mean. Only the partition definition changes per use case:

Manga generation / colorization / consistent character customization: Reference features form $F_{\text{sol}}$; the anchor mask selects the top $k%$ positions by importance score. The Sub-Problem is the current generated feature map. Solving $\nabla^2 F = 0$ yields outputs that match the reference character at structurally critical positions while allowing free variation in pose, lighting, expression, and — crucially — art style (color palette for colorization, comic proportions for manga, etc.).
Inpainting (virtual try-on, object removal): Instead of importance scoring, the Sub-Solution is derived from a user-provided binary mask. The unmasked region $(M_{\text{mask}} = 0)$ is fixed to reference features as the known context; the masked region $(M_{\text{mask}} = 1)$ is the Sub-Problem. LHFG propagates surrounding context inward via harmonic interpolation — the fill matches the surrounding region's tone, texture, and style by construction. This produces seamlessly integrated try-on garments and clean object removals without the hard-edge artifacts of naive infilling.
Multi-subject composition: Given $N$ subject images $S_1, \ldots, S_N$ and a background $B$, we define two categories of Sub-Solution: (a) subject cores $C_i$, obtained by eroding each subject mask inward by $r$ pixels and fixing $F_{C_i} = F_{S_i}$; and (b) far background $B_{\text{far}}$, obtained by eroding the complement of all combined subject masks by $r'$ pixels and fixing $F_{B_{\text{far}}} = F_B$. When subjects spatially overlap, a z-order priority determines precedence. All transition zones — outer rings of each subject mask, the background's inner ring, and inter-subject overlap regions — become the Sub-Problem. Solving the Laplace equation over these transition zones produces smooth interpolations that harmonize every subject with its neighbors and the background, eliminating hard-paste compositing artifacts. In our experiments, DreamSim in multi-subject composition drops from 0.3260 (next-best: Qwen) to 0.1561 — a 52% improvement — because compositing multiple subjects amplifies the exact identity-drift failure mode that PDC was designed to correct.
Image fusion: The primary input image provides structural anchors (top $k%$ features); PDC partially preserves the secondary input. LHFG blends them without manual masking. Unlike style transfer, image fusion produces never-before-seen results by merging two visual concepts — a mountain range and a forest path, for example — guided entirely by the text prompt.
Inherent style transfer: Because LHFG locks only the structurally salient features of the reference (the Sub-Solution) and leaves the Sub-Problem free to be determined by the base model's learned prior, style transfer is an emergent property of every use case. When the base model is an anime-trained SDXL and the input is a photorealistic photograph, LHFG locks the photograph's spatial composition and identity structures while the unconstrained Sub-Problem positions converge to the anime aesthetic. No ControlNet, no LoRA, no additional training required — just swap the base model.

Adapting the framework to Flux.1 Kontext (Diffusion Transformer architecture)

Flux.1 Kontext is a DiT (Diffusion Transformer) operating with a flow matching objective and packed sequence latents of shape $(B, S, D)$ — fundamentally different from SDXL's convolutional $(B, C, H, W)$ feature maps. The naive PDC formulation does not transfer directly.

Core adaptation: blend velocities instead of latents. At each PDC timestep, we run two transformer forward passes on the same current latent $z_t$:

Edit pass: $v_{\text{edit}} = \text{Transformer}(z_t, t, C_{\text{target}})$ — conditioned on the user's edit prompt
Reference pass: $v_{\text{ref}} = \text{Transformer}(z_t, t, C_{\text{ref}})$ — conditioned on the source (reference) prompt.

Both passes see the identical latent state; they receive different text conditioning. The velocity delta $\Delta = v_{\text{ref}} - v_{\text{edit}}$ captures how much the reference identity wants to redirect the denoising direction at the current state — without injecting any reference pixel content. Three safeguards are applied to $\Delta$ before use:

Low-pass filtering: A 2D Gaussian blur (kernel size 5) on $\Delta$ removes high-frequency components. PDC's purpose is to correct pose and silhouette drift — not to copy textures from the reference. The blur enforces this intent.
Preservation mask: We compute a soft spatial mask $P \in [0,1]^{H \times W}$ from the L2 divergence between $v_{\text{edit}}$ and $v_{\text{ref}}$. Positions where the edit prompt changes content (high divergence) receive $P \approx 0$, suppressing the PDC correction so it doesn't fight the user's intended edit. Positions where the edit prompt preserves content (low divergence) receive $P \approx 1$, allowing full correction.
PID modulation: The adaptive $\alpha_{\text{final}}(t)$ is identical to the SDXL formulation — the PID controller logic is architecture-agnostic. The corrected velocity: $$v'_{\text{edit}} = v_{\text{edit}} + \alpha_{\text{final}} \cdot \text{blur}(\Delta) \odot P$$

Packed latent spatial detour. The Gaussian blur, preservation mask computation, and LHFG Jacobi solver all require 2D spatial structure. Flux's packed format $(B, S, D)$ contains no spatial position information. We unpack the sequence into a spatial grid $(B, C, H', W')$, apply all spatial operations, and repack to the sequence format before returning to the transformer — a bit-exact round-trip.

LHFG on velocity fields. In the low-noise phase, LHFG operates on the velocity field $v(z_t, t)$ rather than the latent $z_t$ itself. A reference velocity field $v_{\text{ref}}$ is precomputed via a single transformer pass at the midpoint sigma of the LHFG phase, using a noisy reference latent conditioned on the source prompt. This reference velocity serves as $F_{\text{sol}}$ in the Jacobi iteration; the current edit velocity is $F_{\text{prob}}$.

Measured overhead. adding PDC+LHFG to Flux.1 Kontext increases wall-clock inference from 8.98s to 12.52s (manga) and from 8.96s to 12.32s (inpainting) — a ~39% increase — with essentially zero memory overhead (33.85 GB → 33.87 GB), because neither module introduces trainable parameters or persistent buffers.

Adapting the framework to CogVideoX (video diffusion, 3D spatio-temporal domain)

CogVideoX is a transformer-based video diffusion model with 5D latent tensors of shape $(B, T, C, H, W)$. The per-frame 2D framework from image generation is insufficient: even if each frame is spatially coherent, small inter-frame variations in PID corrections or harmonic solutions cause identity jitter — features that oscillate subtly frame-to-frame (a jawline that sharpens and softens cyclically), producing temporal incoherence. The noise schedule uses DDIM/DPM rather than flow matching, so sigma-to-timestep mapping is $t/1000$, and phase boundaries are defined on normalized sigma.

Extending LHFG to 3D: The solution upgrades the Jacobi solver from a 2D four-neighbor spatial stencil to a 3D six-neighbor spatio-temporal stencil:

$$u^{(k+1)}_{t,i,j} = \frac{w_t(u^{(k)}_{t-1,i,j} + u^{(k)}_{t+1,i,j}) + w_s(u^{(k)}_{t,i-1,j} + u^{(k)}_{t,i+1,j} + u^{(k)}_{t,i,j-1} + u^{(k)}_{t,i,j+1})}{2w_t + 4w_s}$$

where $w_t$ and $w_s$ control the relative influence of temporal versus spatial neighbors. The Sub-Problem / Sub-Solution formulation is identical to the 2D case — only the domain is now a 3D volume. The consequence is immediate and profound: the Maximum Principle now applies in both space and time simultaneously. There cannot be spatial seams (as in the image case) and there cannot be temporal discontinuities. Identity features anchored at frame 0 propagate smoothly forward through time via harmonic extension, exactly as they propagate across space within each frame. We use equal weighting $w_t = w_s = 1.0$ as the default, which provides strong temporal coherence without perceptible spatial softening within 15 Jacobi iterations.

Extending PDC to the temporal domain — two mechanisms:

Temporal decay weights: Frame 0 receives PDC weight 1.0; the final frame receives weight 0.1, decaying linearly. This controls the identity-animation trade-off: early frames must be locked to the reference identity, but later frames must be free to animate. Without this decay, PDC would pull every frame toward the static reference, suppressing motion.

1D Gaussian temporal smoothing: After PDC applies per-frame corrections, the entire corrected volume is convolved along the temporal axis with a 1D Gaussian kernel (kernel size 3, $\sigma = 1.0)$. This addresses a subtle issue: the PID state evolves over denoising steps, not over frames, so its corrections are correlated in denoising time but uncorrelated in video time. The temporal Gaussian restores the missing inter-frame correlation, damping residual frame-to-frame oscillations introduced by the PID controller.

3D anchor extraction: The edit-aware importance score over the full temporal volume is:

$$s_{t,i,j} = \frac{|F^{(t,i,j)}_{\text{ref}}|}{1 + |F^{(t,i,j)}_{\text{ref}} - F^{(t,i,j)}_{\text{gen}}|}$$

Positions with high reference magnitude and low edit divergence score highest, becoming Sub-Solution anchors. Frame 0 naturally receives the densest anchors (closest to the unmodified reference); later frames receive sparser anchoring proportional to how much freedom the model has used to animate.

Training convergence acceleration — proof the principle transcends images

The same dual-phase error-correction logic that governs diffusion denoising can be applied to gradient-based neural network training, where the "error" is not noise-prediction drift but the gap between current and target loss — also high early, diminishing later.

Phase 1 — PDC as adaptive learning rate: We implement a PID controller on the exponentially smoothed training loss $(\alpha = 0.9)$. The multiplier is: $$\mu = 1 + e + \int e dt - \dot{e}$$ applied as $lr = lr_{\text{base}} \times \mu$. The I-term integrates accumulated loss to escape plateaus (clamped at $\pm 10$); the D-term dampens the learning rate on loss spikes. This yields a fully adaptive schedule requiring no manual warmup or decay tuning — the controller discovers the appropriate schedule from the loss signal alone.

Phase 2 — LHFG in weight space: After PDC stabilizes training dynamics, we snapshot the model weights as $W^{\text{stable}}$ and partition all weight positions by gradient magnitude $|\nabla_W L|$. The bottom $k%$ (assuming small gradient = converged) forms the Sub-Solution and is regularized toward the stable reference: $\mathcal{L}_{\text{anchor}} = \sum{_{i,j \in \text{anc}} |W_{i,j} - W^{\text{stable}}_{i,j}|^2}$. The remaining high-gradient positions form the Sub-Problem and are penalized by the squared discrete Laplacian: $\mathcal{L}_{\text{smooth}} = \frac{1}{|\text{lrn}|} \sum_{i,j \in \text{lrn}} (\nabla^2 W_{i,j})^2$. When this penalty vanishes, every Sub-Problem weight equals the average of its neighbors — satisfying the discrete Laplace equation exactly in weight space.

MNIST results: All three configurations (PDC only, LHFG only, PDC+LHFG combined) consistently outperform the vanilla SGD baseline across training loss, training accuracy, validation loss, and validation accuracy — with the combined variant achieving best performance on all four metrics. Sensitivity analysis shows the PDC-to-LHFG transition timestep $T_{\text{PDC,end}}$ is the single overwhelmingly dominant parameter ($r_s = -0.727$, $p < 0.001$ on training loss), confirming that the duration of the adaptive-learning-rate phase is the primary lever for convergence quality.

Quantitative performance against state of the art

All benchmarks compare against Nano Banana Pro (Gemini 3 Pro Image), FLUX.2-dev, Qwen-Image-Edit-2511, MS-Diffusion, and OmniGen. Four metrics: CLIP-T (text-image alignment), DreamSim (perceptual identity similarity), LPIPS (diversity/perceptual distance), FID (distributional quality).

Consistent image customization: FID 88.38 vs. next-best MS-Diffusion 118.47 (−25.4%); DreamSim 0.1055 vs. 0.1577 (−33.1%)
Inpainting: 1st on all four metrics; DreamSim 0.0724 vs. OmniGen 0.0958 (−24.4%); FID 22.65 vs. 25.48 (−11.1%)
Multi-subject composition: DreamSim 0.1561 vs. Qwen 0.3260 (−52.1%); FID 214.49 vs. Qwen 281.66 (−23.8%)
Image fusion: Best LPIPS (0.5478); 2nd-best DreamSim
Colorization: Best CLIP-T (0.8956) and DreamSim (0.1881)
User study (50 blinded participants, 1–5 Likert scale across Reference Similarity, Prompt Match, Image Quality, Diversity): SoulCanvas scores 4.80–4.84 overall across all five tasks. The closest competitor never exceeds 3.50 on any task. Across all four criteria, in all five tasks, SoulCanvas is ranked first — by a margin that leaves no statistical ambiguity.

Inference speed and memory efficiency

A critically underappreciated result: SoulCanvas is not just the most accurate — it is the fastest and most memory-efficient method tested, averaging 3.65 seconds per image on under 9 GB VRAM. Where the heaviest alternative Qwen-Image-Edit-2511 incurs 80.3x more slowdown and the one with respect to VRAM usage (Flux.2) costs 7.0x more storage. SoulCanvas can generate 72 multi-subject compositions in the same wall-clock time FLUX.2 generates one, and runs on a consumer 12 GB GPU — a hardware tier that excludes both Qwen and FLUX.2 entirely. This efficiency derives from operating entirely within a standard SDXL backbone (24 denoising steps) with lightweight PDC and LHFG modules that require no auxiliary encoders, no batch-size inflation, and no persistent parameter buffers.

Hyperparameter sensitivity and tuning hierarchy

We conducted systematic sensitivity analysis via Tree-structured Parzen Estimator (TPE) sampling across 30–40 independent configurations per task, computing Spearman rank correlations between each hyperparameter and each metric. The key finding: the phase-boundary timestep $T_{\text{phase1}}$ is the single most influential parameter, with the strongest and most frequently significant correlations ($p < 0.001$) across tasks. This confirms that identity preservation is governed primarily by how long PDC is active — not entirely by the specific PID gains. Among gains, $K_I$ (integral term) most consistently tightens identity fidelity across all tasks. All LHFG structural parameters ($k$, $N_{\text{Jacobi}}$) can safely remain at their optimized defaults. The practical tuning hierarchy: set $T_{\text{phase1}}$ first, then $K_I$, then $K_D$ for fine control — everything else is robust to variation.

Language: Python

Diffusion Models: SDXL (multiple base models), Wan2.2, CogVideoX-5b, FLUX.1-Kontext-dev

LLM: Fine-tuned counselling language model

Voice: Qwen3-TTS voice synthesis

Framework: Gradio (interactive web UI), Diffusers, PyTorch

Hardware: Huawei Ascend 910B3 NPU (64 GB HBM) with full backward compatibility to NVIDIA CUDA GPUs

Research: accepted paper at CVPR 2026 with sole first authorship

Challenges we ran into

The central technical challenge was designing a training-free guidance framework that could meaningfully improve diffusion model outputs without touching model weights. This is counterintuitive — the prevailing wisdom in 2024-2025 was that controllable generation requires either fine-tuning (LoRA, DreamBooth) or architectural modification (ControlNet, IP-Adapter). We rejected both.

The deeper challenge was generality. It's relatively straightforward to hand-craft a guidance signal for one specific task. But we needed a single principled framework — not a bag of tricks — whose core mathematical formulation could solve a virtually unbounded class of problems: identity-consistent editing, temporal smoothing in video, multi-subject composition, inpainting, style transfer, and even training convergence acceleration in unrelated deep learning tasks. Finding that unifying principle — that the denoising trajectory itself exhibits a "high drift early, fine correction late" pattern analogous to PID control systems and boundary value problems — took many failed hypotheses.

Other significant challenges

The mid-noise instability gap: In the mid-noise phase, the diffusion models enter a regime where neither global structure nor local detail is locked in. Existing methods (classifier-free guidance, attention manipulation) apply uniform pressure across all timesteps, causing identity drift precisely in this critical window. Designing a mid-phase component to predict and pre-correct spatial drift before it cascades required borrowing from control theory — a field rarely applied to generative models.

Harmonic field computation at inference time: low-noise component models detail refinement as a Dirichlet boundary value problem. Solving Laplace's equation at every denoising step naively would be prohibitively expensive. We needed an efficient approximation (Jacobi iteration with early stopping) that converges in only a few iterations without visible quality loss.

Solo development across the full stack: Building a research contribution (CVPR 2026 paper), a production Gradio application, voice integration, LLM fine-tuning for therapeutic dialogue, and a user-facing therapy product — simultaneously, alone — required ruthless prioritization and an architecture designed for modularity from day one.

Accomplishments that we're proud of

We beat every current state-of-the-art image editing model — qualitatively, quantitatively, and in user preference — without training a single parameter.

Specific accomplishments:

Qualitative superiority over Nano Banana Pro, Flux.2, Qwen Image Edit, MS-Diffusion, and OmniGen across identity-consistent editing, style transfer, and composition tasks. Our outputs maintain sharper identity, cleaner line art, and more coherent global structure.

Zero-cost plug-in improvements to existing commercial-grade models:

Flux Kontext: Our guidance dramatically improves reference image style preservation — the difference is immediately visible to the naked eye, not just measurable in metrics.
CogVideoX: Our temporal consistency guidance eliminates abrupt motion artifacts (e.g., sudden hand-lift postures in generated video) and produces physically plausible smooth motion without any video-specific fine-tuning.

Quantitative benchmarks consistently place us at 1st or 2nd across four standard metrics:

CLIP-T (text-image alignment)
DreamSim (perceptual similarity)
LPIPS (diversity)
FID (distributional quality)
...against the same baselines: Flux.2, Qwen Image Edit, MS-Diffusion, and OmniGen.

User study dominance: Human evaluators rated SoulCanvas as the most favoured across all four evaluation dimensions — Reference Similarity, Prompt Match, Image Quality, and Diversity. Not first in one and second in others — first in all.

Theoretical generality proven: The same core principle (predict drift → correct trajectory) successfully accelerated training convergence in a standard deep learning classification network. This validates our thesis that the framework addresses a fundamental class of problems characterized by "high error early, low error late" — far beyond image generation alone.

Accepted at CVPR 2026 — the premier computer vision conference — validating the scientific rigor of our approach through double-blind peer review.

What we learned

The best frameworks come from asking "what's the same?" not "what's different."

For a long enough time, we treated manga generation, video smoothing, inpainting, and multi-subject composition as separate problems requiring separate solutions. The breakthrough came when we stopped looking at what distinguished these tasks and instead asked: what pattern do they all share? The answer — a noisy-to-clean trajectory where early drift compounds into late-stage artifacts — unlocked a single mathematical framework that solved them all. This lesson extends far beyond AI: the most powerful abstractions emerge from finding unity in apparent diversity.

Training-free doesn't mean effort-free — it means principled.

Not touching model weights forced us to deeply understand why diffusion models fail, rather than brute-forcing corrections through gradient descent. This constraint — initially frustrating — produced a more interpretable, more portable, and more theoretically grounded solution than any fine-tuning approach could have.

Therapy and technology share a common truth: presence matters more than perfection.

Building the therapeutic side taught us that users don't need a flawless AI therapist. They need one that's there — at 3 AM, without judgment, without a waitlist. Similarly, the companion's visual appearance doesn't need to be photorealistic to be emotionally meaningful. A simple anime character that a user designed themselves creates deeper attachment than the most technically impressive photorealistic render they had no hand in creating. The act of creation is the connection.

Modular architecture isn't optional for solo developers — it's survival.

With 10+ features, 6 model classes, and multiple hardware targets (NPU + CUDA), the project would have collapsed under its own weight without strict separation of concerns. The decision to build DPSModel as a standalone base class with lazy loading and auto-invalidation on model switch — made early and maintained religiously — is what allowed one person to ship what looks like a team's output.

Lived experience is an irreplaceable design compass.

Every UX decision — the companion persistence (because consistency builds trust), the creative exercises (because making art was part of my own recovery), and other components — was informed by knowing what it's like to be on the other side. No amount of user research substitutes for having been the user.

Who does this help

Primary: Individuals experiencing loneliness, mild-to-moderate depression, anxiety, or emotional burnout who need an always-available, non-judgmental companion

Secondary: Expressive Arts Therapy practitioners seeking AI-augmented tools for client exercises

Tertiary: AIGC content creators who need a comprehensive, multi-modal generation studio

What's next for SoulCanvas

Near-term (3-6 months):

Clinical pilot study: Partner with university counselling centers to run a controlled trial measuring SoulCanvas's impact on PHQ-9 (depression) and GAD-7 (anxiety) scores as a complement to human therapy, not a replacement. We want peer-reviewed evidence of efficacy.
Mobile-first redesign: Migrate from Gradio to a native mobile experience (React Native + on-device inference for the LLM via llama.cpp). Therapy happens on couches and in beds, not at desks.
Expanded therapeutic modalities: Integrate guided journaling, mood tracking with AI-generated visual metaphors (e.g., your mood as a landscape that evolves over weeks), and CBT worksheet generation tailored to the conversation context.

Medium-term (6-12 months):

Therapist dashboard: Allow licensed practitioners to "prescribe" SoulCanvas exercises to their clients between sessions, review AI-generated session summaries (with client consent), and customize the companion's therapeutic approach per patient.
Multi-modal memory: The companion remembers past conversations, references previous artwork you've made together, and notices patterns ("You've mentioned feeling isolated three times this week — would you like to explore that?").

Long-term vision (1-2 years):

SoulCanvas as a platform: Allow therapists and artists to publish custom "therapy journeys" — guided multi-session creative programs (grief processing through portrait evolution, anxiety management through world-building, identity exploration through character design).
Localization for underserved regions: Deploy lightweight versions optimized for low-bandwidth environments in Southeast Asia, Sub-Saharan Africa, and rural China — where mental health infrastructure is most scarce and the need is greatest.
The companion grows with you: Using longitudinal interaction data (fully private, on-device), the companion's personality, visual style, and therapeutic approach subtly evolve to match the user's healing journey — more playful as depression lifts, more grounding during anxious periods. The endgame is simple: no one should have to heal alone because they can't afford not to.

Impact Statement

SoulCanvas democratizes emotional support and creative healing. In a world where 60% of countries have fewer than 1 psychiatrist per 100,000 people, this project provides immediate, stigma-free therapeutic companionship to anyone with a smartphone — in their own language, wearing a face they chose, speaking with a voice that feels safe.

The therapeutic value is twofold: the companion is the therapy (counselling conversations), and the act of creating the companion is also therapy (Expressive Arts). A user designing their companion's anime appearance, generating a comic of their day, or trying on an outfit they lack confidence to wear in real life — these are all therapeutic acts of self-expression and identity exploration.

For the AIGC community, SoulCanvas pushes the boundary of what's possible with training-free diffusion control. Our CVPR 2026 dual-phased approach improves identity consistency, reduces artifacts, and enables professional-quality results across manga generation, virtual try-on, scene composition, and style transfer — all without fine-tuning.

Who benefits: