SoulCanvas

An AI companion you design yourself. SoulCanvas turns Expressive Arts Therapy into an always-available experience where the act of creation is the cure.

Comment

Note: Our deepest apologies that we're unable to provide a live demo link due to constrained hardware resources that can't fully support the AI models used in this project. Please refer to the YouTube demo video and the screenshots at the top of the project to view the working project. We are very sorry for the inconveniences caused.

Inspiration

We are living through what many public-health experts describe as a global loneliness epidemic. The World Health Organization has recognized social isolation as a serious public-health concern — one whose effects on the body have been compared to smoking fifteen cigarettes a day. More than 300 million people around the world live with depression, and fewer than a quarter of them are able to receive adequate mental-health support. This is rarely because caring professionals do not exist; it is because cost, stigma, language barriers, and simple unavailability so often stand between people and the help they deserve — especially at 3 AM, when the walls feel like they are closing in and no office door is open.

Expressive Arts Therapy — the evidence-based practice of using creative expression (visual art, storytelling, role-play) as a therapeutic intervention — is one of the gentlest and most effective ways of processing complex emotions. Traditionally, however, it asks for a trained facilitator and sustained engagement, which many people simply cannot access. We found ourselves asking, with hope rather than certainty: might some of that caring facilitation be carried in one's pocket — available at any hour, in one's own language, wearing the face of a companion one has lovingly designed oneself?

This question is deeply personal to me. I was diagnosed with bipolar disorder in middle school. The illness took me out of school for a full year, and when I returned, I was the oldest student in every classroom, carrying a silence that the people around me could not easily understand. During my most difficult period, it was my school counsellor who became my lifeline. No matter how many times I appeared at her office unannounced — mid-panic, mid-spiral, sometimes unable to explain why I had come — she welcomed me with the same gentle smile every single time. She never once made me feel like a burden for showing up; that consistency — that quiet, unconditional presence — did more for me than any medication or textbook, and I will be grateful to her for the rest of my life. I recovered; and I build now in the hope that the next young person — the one spiralling at 2 AM with no office to run to — will not have to face that emptiness alone.

Our SDG Alignment

We have tried to root every design decision of SoulCanvas in the United Nations Sustainable Development Goals, and we would like to explain that alignment as clearly and honestly as we can.

Primary alignment — SDG 3: Good Health and Well-Being

Target 3.4 calls on the world to "promote mental health and well-being." SoulCanvas is our small contribution toward that call:

  • An always-available, judgment-free companion for people experiencing loneliness, mild-to-moderate depression, anxiety, or emotional burnout — present at 3 AM when human services are unavailable, and free of waitlists.
  • Guided, evidence-informed Expressive Arts Therapy exercises, in which the act of creating — designing one's own companion, drawing a comic of one's day, visualizing a feeling — is itself the therapeutic intervention.
  • Bilingual (Chinese/English) counselling conversations with voice interaction, so that support can be received in the language of one's heart rather than the language of one's textbooks.

Target 3.8 envisions access to essential health services without financial hardship. We would never claim that SoulCanvas is health care itself — it is not, and it must never pretend to be. But we do hope it can help fill the painful gap between needing support and receiving it: it carries no session fees, no geographic constraints, and no requirement to explain oneself to a gatekeeper first.

Supporting alignment — SDG 10: Reduced Inequalities

Target 10.2 asks us to promote the inclusion of all people, irrespective of economic or other status. The barriers to mental-health support fall hardest on those who already have the least — people in rural areas, low-income households, and language-minority communities. SoulCanvas tries to lower those barriers in concrete, measurable ways:

  • It runs on a consumer-grade 12 GB GPU, rather than requiring data-center hardware, which makes affordable deployment realistic in underserved regions.
  • Our roadmap prioritizes lightweight, low-bandwidth versions for Southeast Asia, Sub-Saharan Africa, and rural China — the places where mental-health infrastructure is most scarce and the need is greatest.
  • Its bilingual foundation is designed to grow into broader language support, because comfort should not be reserved for English speakers.

A modest contribution to SDG 9: Industry, Innovation and Infrastructure

The research underlying SoulCanvas has been peer-reviewed and accepted at CVPR 2026, and the method is training-free — meaning others may apply it on top of existing open models without costly retraining. We hope this lowers the barrier for other builders, especially students and solo developers, to create their own tools for good.

What it does

SoulCanvas gently brings together therapeutic conversation and creative expression in one place:

Therapeutic Core: a bilingual (CN/EN) counselling-oriented language model for supportive conversations; natural voice dialogue; and companion-appearance customization, offered as a guided Expressive Arts exercise.

Companion Creation: text-to-image generation; image-to-image style transfer (for example, gently transforming a real photograph into an anime portrait); consistent pose generation; chibi versions; background removal; and short video generation.

Advanced AIGC Toolkit: temporal-consistency improvement for CogVideoX; reference-similarity and style-similarity enhancement for FLUX.1 Kontext; manga and comic generation; smart inpainting (virtual try-on, object removal, DIY image modification, and more); multi-subject scene composition; image fusion and style blending; colorization; and consistent identity generation.

Professional Tools: model switching (any SDXL-compatible model); training-convergence visualization; and style conversion between any two visual domains.

We would like to emphasize, kindly, that the creative features are not decoration around the therapy — they are part of the therapy. In Expressive Arts practice, designing a supportive figure, illustrating one's day, or trying on an outfit one lacks the confidence to wear in real life are all recognized acts of self-expression and self-empowerment.

How we built it

SoulCanvas unifies three AI pillars into a single, hopefully comforting, experience:

A Specialized Counselling LLM — a fine-tuned large language model trained specifically for therapeutic conversations, supporting both Chinese and English. It tries not merely to respond, but to listen actively, validate feelings, gently question unhelpful thought patterns, and guide users through Expressive Arts Therapy exercises.

A Visual Identity Engine — powered by a dual-phased diffusion approach that we were fortunate to have accepted at CVPR 2026 (SyntheticManga: Training-Free Manga Generation with Phased Diffusion, sole first authorship). We mention this not to boast, but because double-blind peer review gave us the confidence that the method is sound enough to serve users well. With it, users create their own companion's appearance — anime, photorealistic, 3D cartoon, or any style they wish. This is not merely customization; in Expressive Arts Therapy, the act of creating one's own supportive figure is a recognized exercise in self-empowerment. The Adaptive Drift Modulator(ADM) module in the CVPR submission is exactly the same as our Predictive Drift Controller(PDC); while we replaced the Boltzmann Fourier Guidance(BFG) module with a more-powerful detail refiner — Local Harmonic Field Guidance(LHFG) — that improves the consistency and detailedness of the reference companion's look.

A Voice Interaction Layer — using pretrained voice synthesis (Qwen3-TTS), the companion speaks, so that conversations can feel like warm dialogue rather than cold text.

The observation underneath everything: denoising is not one single process

Standard SDXL inference runs a U-Net for 20–50 denoising steps, applying classifier-free guidance uniformly at every step. ControlNet and IP-Adapter — two influential methods we admire and have learned a great deal from — follow the same convention. The implicit assumption is that every step benefits from the same type of intervention.

With great respect for that prior work, our experiments suggested this assumption may deserve a second look — and gently questioning it became the foundation of our framework.

We observe — supported by prior drift-analysis work on DDPM trajectories, to whose authors we are indebted — that denoising proceeds in three functionally distinct regimes:

  • High-noise regime (late timesteps, ~850–1000): essentially pure noise; too chaotic for any stable guidance signal.
  • Mid-noise regime (~400–850): global spatial structure, pose, and identity features are established; the model is most susceptible to identity drift — the gradual corruption of reference character features as the prompt and identity objectives diverge.
  • Low-noise regime (~0–400): macro structure is settled; the model synthesizes local high-frequency details — line art, facial highlights, fabric texture.

In our experiments, treating all three regimes identically appeared inefficient and, at times, counterproductive. We therefore separate our guidance into two specialized controllers, each active only in the regime where it can be most helpful, and neither active where it might cause harm.

Phase 1: Predictive Drift Controller (PDC)

The problem it addresses: in the mid-noise regime, both the target prompt and the reference identity wish to steer the latent trajectory, and they usually disagree. Without care, the model gradually loses the reference character's face, proportions, and line style — the identity drift described above.

Drift quantification. At each denoising timestep (t), we define feature-space drift as the L1 divergence between the noise predictions generated from the current latent under the target prompt versus the same latent under the reference prompt:

$$ d(t) = \left| \epsilon_\theta(z_t, t, C_{\text{target}}) - \epsilon_\theta(z_{\text{ref},t},\, t,\, C_{\text{ref}}) \right|_1 $$

where (\(\epsilon_\theta\)) is the denoising U-Net, (\(C_{\text{target}}\)) and (\(C_{\text{ref}}\)) are the text embeddings for the desired output and reference image respectively, and (\(z_{\text{ref},t}\)) is a noisy version of the reference image latent at timestep (\(t\)). This scalar (\(d(t)\)) measures, in real time, how far the current generation is drifting from the reference identity manifold.

PID formulation. Rather than applying a fixed correction, we feed (d(t)) into a discrete Proportional-Integral-Derivative controller:

$$ P(t) = d(t) $$

$$ \quad I(t) = I(t-1) + d(t) $$

$$ \quad D(t) = d(t) - d(t-1) $$

$$ u(t) = 1.0 + K_P \cdot P(t) + K_I \cdot I(t) + K_D \cdot D(t) $$

Each PID term contributes something the others cannot:

  • The P-term responds to immediate structural divergence — if drift is large right now, correction is strengthened right now.
  • The I-term accumulates historical drift — if misalignment persists over several steps, the integral builds and gently but firmly counteracts the "gravitational pull" of the target prompt that would otherwise erode the identity.
  • The D-term anticipates future drift — it softens the controller when drift is already decreasing, preventing over-correction and overshoot.

This PID output (u(t)) modulates a linear base blending schedule (\alpha(t)), yielding an adaptive mixing factor:

$$ \alpha_{\text{final}}(t) = \alpha(t) \cdot u(t) $$

The latent is then updated by blending the current latent toward the reference:

$$ z'_{t} = (1 - \alpha_{\text{final}}(t))\, z_t + \alpha_{\text{final}}(t)\, z_{\text{ref},t} $$

Why we believe this helps: PDC embodies a simple, honest principle — high error early deserves stronger regulation, and diminishing error later deserves to be left in peace. It is the same insight that has long served PID controllers in robotics, thermostats, and flight stabilization, applied here, with gratitude to control theory, to the denoising trajectory.

PID component ablation: removing the I-term alone degrades DreamSim by roughly 50% in consistent image customization (from 0.1055 to 0.1586) and roughly 40% in inpainting (from 0.0724 to 0.1007), suggesting that accumulated drift correction is essential for single-subject tasks. Removing the D-term most affects multi-subject composition (DreamSim 0.1561 → 0.3578), where anticipatory damping stabilizes the volatile corrections arising from multiple competing subject boundaries. Removing the P-term primarily affects structural fidelity in image fusion. No single term dominates across all tasks — the three appear to contribute together, in complement.

Phase 2: Local Harmonic Field Guidance (LHFG)

The problem it addresses: once PDC has steadied the global structure, the low-noise regime synthesizes high-frequency detail — lace patterns, eye highlights, fine hair strands. The difficulty is that injecting reference details at this stage tends to create hard seams: the reference texture sits like a foreign object pasted into a generated field, producing blending artifacts at every boundary. Standard attention injection and feature copying both encounter this difficulty.

The physical insight. We reframe the problem as a classical partial differential equation. Given two requirements that appear to conflict — (i) identity preservation: anchored positions must exactly match the reference, and (ii) natural blending: unanchored positions must adapt smoothly to injected details — the mathematical structure is precisely a Dirichlet boundary value problem on the latent feature space. We solve the Laplace equation:

$$ \nabla^2 F = 0 $$

over the interior (the Sub-Problem) subject to fixed boundary conditions (the Sub-Solution). The Maximum Principle for harmonic functions guarantees that every interior point equals the weighted average of its neighbors — which means blending artifacts are ruled out by construction: local extrema cannot exist in the interior, so smooth interpolation through the fixed anchors is not merely an emergent tendency but a guaranteed mathematical property. We found this deeply reassuring, and we are thankful to the long tradition of classical analysis that made such a guarantee available to borrow.

Sub-Solution / Sub-Problem partition

The Sub-Solution (boundary) consists of positions selected as identity-critical anchors. We compute a per-position importance score

$$ s = \lVert F_{\text{sol}} \rVert_2 $$

from the reference feature map (\(F_{sol} \in \mathbb{R}^{B \times C \times H \times W}\)), where (\(F_{\text{sol}}\)) is extracted from a U-Net pass conditioned on the reference prompt. The top (k%) positions by this norm form the anchor mask (\(M_{\text{anchor}}\)).

The Sub-Problem (interior) is everything else; these positions are solved by the harmonic equation.

Jacobi iteration. The solution field is initialized as:

$$ u^{(0)} = M_{\text{anchor}} \odot F_{\text{sol}} + (1 - M_{\text{anchor}}) \odot F_{\text{prob}} $$

where (\(F_{\text{prob}}\)) is the current denoising feature map. We then apply the standard four-neighbor Jacobi relaxation:

$$ u^{(k+1)}_{\text{prob},i,j} = \frac{1}{4}\left(u^{(k)}_{i-1,j} + u^{(k)}_{i+1,j} + u^{(k)}_{i,j-1} + u^{(k)}_{i,j+1}\right) $$

After every iteration, anchors are reset to their reference values:

$$ u^{(k+1)} = M_{\text{anchor}} \odot F_{\text{sol}} + (1 - M_{\text{anchor}}) \odot u^{(k+1)}_{\text{prob}} $$

This strictly enforces the Dirichlet condition — identity anchors never drift, regardless of how many iterations run. The converged field (\(F_{\text{harmonic}}\)) satisfies the discrete Laplace equation in the Sub-Problem while exactly matching the reference at all anchor positions.

Why Jacobi rather than Gauss-Seidel? Gauss-Seidel, used in some previous approaches, requires forward substitution with the full discrete Laplacian matrix (\(L \in \mathbb{R}^{n \times n}\)), giving (O(n^2)) complexity due to the density of non-zero entries — impractical, in our experience, for inference-time use. Jacobi relaxation decomposes into simple per-position additions that are fully parallelizable and run on the GPU in place. Empirically, only 15 iterations suffice for adequate convergence. Furthermore, LHFG operates on VAE-compressed latent feature maps (typically \(128 \times 128\) or smaller), not full-resolution pixel space — considerably reducing the domain size compared with methods that operate at pixel resolution, such as PixPerfect. Measured overhead: LHFG contributes an average of 1.587% of total inference time across all tasks.

Phase 3: Strategic Zero-Out

After LHFG completes, we respectfully step aside: all guidance is switched off for the final low-noise steps. The reasoning is that PDC has settled the global structure and LHFG has anchored the fine detail — continued intervention at this point would over-constrain the model and produce "pasted-on" artifacts, where newly synthesized high-frequency texture fails to integrate with the scene's lighting and atmosphere. Allowing the U-Net its freedom in the final phase lets it synthesize detail purely from the text prompt, producing outputs that look organically generated rather than forcefully guided. There is, we think, a quiet lesson in this: good guidance includes knowing when to stop guiding.

One solver, many product features

The mathematical abstraction — partition a domain into a known Sub-Solution and an unknown Sub-Problem, then solve the Laplace equation over the Sub-Problem — is entirely agnostic to what "known" and "unknown" mean. Only the partition definition changes per use case:

  • Manga generation / colorization / consistent character customization: reference features form (\(F_{\text{sol}}\)); the anchor mask selects the top (k%) positions by importance score. The Sub-Problem is the current generated feature map. Solving (\(\nabla^2 F = 0\)) yields outputs that match the reference character at structurally critical positions while allowing free variation in pose, lighting, expression, and — importantly — art style (color palette for colorization, comic proportions for manga, and so on).
  • Inpainting (virtual try-on, object removal): instead of importance scoring, the Sub-Solution is derived from a user-provided binary mask. The unmasked region (\(M_{\text{mask}} = 0\)) is fixed to reference features as the known context; the masked region (\(M_{\text{mask}} = 1\)) is the Sub-Problem. LHFG propagates the surrounding context inward via harmonic interpolation — the fill matches the surrounding region's tone, texture, and style by construction. This produces gently integrated try-on garments and clean object removals, without the hard-edge artifacts of naive infilling.
  • Multi-subject composition: given (\(N\)) subject images (\(S_1, \ldots, S_N\)) and a background (\(B\)), we define two categories of Sub-Solution: (a) subject cores (\(C_i\)), obtained by eroding each subject mask inward by (\(r\)) pixels and fixing (\(F_{C_i} = F_{S_i}\)); and (b) far background (\(B_{\text{far}}\)), obtained by eroding the complement of all combined subject masks by (\(r'\)) pixels and fixing (\(F_{B_{\text{far}}} = F_B\)). When subjects spatially overlap, a z-order priority determines precedence. All transition zones — the outer rings of each subject mask, the background's inner ring, and inter-subject overlap regions — become the Sub-Problem. Solving the Laplace equation over these transition zones produces smooth interpolations that harmonize every subject with its neighbors and the background, easing away hard-paste compositing artifacts. In our experiments, DreamSim in multi-subject composition improves from 0.3260 (the next-best system, Qwen) to 0.1561 — a 52% relative improvement — which we attribute to the fact that compositing multiple subjects amplifies exactly the identity-drift failure mode PDC was designed to correct.
  • Image fusion: the primary input image provides structural anchors (top (k%) features); PDC partially preserves the secondary input. LHFG blends them without manual masking. Unlike style transfer, image fusion produces never-before-seen results by merging two visual concepts — a mountain range and a forest path, for example — guided entirely by the text prompt.
  • Inherent style transfer: because LHFG locks only the structurally salient features of the reference (the Sub-Solution) and leaves the Sub-Problem free to be shaped by the base model's learned prior, style transfer emerges naturally from every use case. When the base model is an anime-trained SDXL and the input is a photorealistic photograph, LHFG preserves the photograph's spatial composition and identity structures while the unconstrained Sub-Problem positions settle gracefully into the anime aesthetic. No ControlNet, no LoRA, no additional training — one simply swaps the base model.

Adapting the framework to FLUX.1 Kontext (Diffusion Transformer architecture)

FLUX.1 Kontext is a DiT (Diffusion Transformer) operating with a flow-matching objective and packed sequence latents of shape (\((B, S, D)\)) — fundamentally different from SDXL's convolutional (\((B, C, H, W)\)) feature maps. The naive PDC formulation does not transfer directly, and adapting it taught us a great deal.

Core adaptation: blend velocities instead of latents. At each PDC timestep, we run two transformer forward passes on the same current latent (\(z_t\)):

  • Edit pass: (\(v_{\text{edit}} = \text{Transformer}(z_t, t, C_{\text{target}})\)) — conditioned on the user's edit prompt.
  • Reference pass: (\(v_{\text{ref}} = \text{Transformer}(z_t, t, C_{\text{ref}})\)) — conditioned on the source (reference) prompt.

Both passes see the identical latent state; they receive different text conditioning. The velocity delta (\(\Delta = v_{\text{ref}} - v_{\text{edit}}\)) captures how much the reference identity wishes to redirect the denoising direction at the current state — without injecting any reference pixel content. Three safeguards are applied to (\(\Delta\)) before use:

  • Low-pass filtering: a 2D Gaussian blur (kernel size 5) on (\(\Delta\)) removes high-frequency components. PDC's purpose is to correct pose and silhouette drift — not to copy textures from the reference. The blur upholds this intent.
  • Preservation mask: we compute a soft spatial mask (\(P \in [0,1]^{H \times W}\)) from the L2 divergence between (\(v_{\text{edit}}) and (v_{\text{ref}}\)). Positions where the edit prompt changes content (high divergence) receive (\(P \approx 0\)), suppressing the PDC correction so that it does not fight the user's intended edit. Positions where the edit prompt preserves content (low divergence) receive (\(P \approx 1\)), allowing full correction.
  • PID modulation: the adaptive (\(\alpha_{\text{final}}(t)\)) is identical to the SDXL formulation — the PID controller logic is architecture-agnostic.

The corrected velocity:

$$ v'_{\text{edit}} = v_{\text{edit}} + \alpha_{\text{final}} \cdot \text{blur}(\Delta) \odot P $$

Packed latent spatial detour. The Gaussian blur, preservation mask computation, and LHFG Jacobi solver all require 2D spatial structure. Flux's packed format (\((B, S, D)\)) contains no spatial position information. We unpack the sequence into a spatial grid (\((B, C, H', W')\)), apply all spatial operations, and repack to the sequence format before returning to the transformer — a bit-exact round trip.

LHFG on velocity fields. In the low-noise phase, LHFG operates on the velocity field (\(v(z_t, t)\)) rather than the latent (\(z_t\)) itself. A reference velocity field (\(v_{\text{ref}}\)) is precomputed via a single transformer pass at the midpoint sigma of the LHFG phase, using a noisy reference latent conditioned on the source prompt. This reference velocity serves as (\(F_{\text{sol}}\)) in the Jacobi iteration; the current edit velocity is (\(F_{\text{prob}}\)).

Measured overhead. Adding PDC+LHFG to FLUX.1 Kontext increases wall-clock inference from 8.98 s to 12.52 s (manga) and from 8.96 s to 12.32 s (inpainting) — an increase of roughly 39% — with essentially zero memory overhead (33.85 GB → 33.87 GB), because neither module introduces trainable parameters or persistent buffers. We share the cost as openly as the benefit, so that anyone considering the method can make an informed choice.

Adapting the framework to CogVideoX (video diffusion, 3D spatio-temporal domain)

CogVideoX is a transformer-based video diffusion model with 5D latent tensors of shape (\((B, T, C, H, W)\)). The per-frame 2D framework from image generation is not sufficient here: even if each frame is spatially coherent, small inter-frame variations in PID corrections or harmonic solutions cause identity jitter — features that oscillate subtly frame to frame (a jawline that sharpens and softens cyclically), producing temporal incoherence. The noise schedule uses DDIM/DPM rather than flow matching, so the sigma-to-timestep mapping is (\(t/1000\)), and phase boundaries are defined on normalized sigma.

Extending LHFG to 3D: we upgrade the Jacobi solver from a 2D four-neighbor spatial stencil to a 3D six-neighbor spatio-temporal stencil:

$$ u^{(k+1)}_{t,i,j} = \frac{w_t(u^{(k)}_{t-1,i,j} + u^{(k)}_{t+1,i,j}) + w_s(u^{(k)}_{t,i-1,j} + u^{(k)}_{t,i+1,j} + u^{(k)}_{t,i,j-1} + u^{(k)}_{t,i,j+1})}{2w_t + 4w_s} $$

where (\(w_t\)) and (\(w_s\)) control the relative influence of temporal versus spatial neighbors. The Sub-Problem / Sub-Solution formulation is identical to the 2D case — only the domain is now a 3D volume. The consequence is one we found genuinely beautiful: the Maximum Principle now applies in both space and time simultaneously. There cannot be spatial seams (as in the image case), and there cannot be temporal discontinuities. Identity features anchored at frame 0 propagate smoothly forward through time via harmonic extension, exactly as they propagate across space within each frame. We use equal weighting (\(w_t = w_s = 1.0\)) as the default, which provides strong temporal coherence without perceptible spatial softening within 15 Jacobi iterations.

Extending PDC to the temporal domain — two mechanisms:

Temporal decay weights: frame 0 receives PDC weight 1.0; the final frame receives weight 0.1, decaying linearly. This balances the identity-animation trade-off: early frames should stay close to the reference identity, while later frames must be free to animate. Without this decay, PDC would pull every frame toward the static reference and quietly suppress the motion itself.

1D Gaussian temporal smoothing: after PDC applies per-frame corrections, the entire corrected volume is convolved along the temporal axis with a 1D Gaussian kernel (kernel size 3, (\(\sigma = 1.0)\)). This addresses a subtle issue: the PID state evolves over denoising steps, not over frames, so its corrections are correlated in denoising time but uncorrelated in video time. The temporal Gaussian restores the missing inter-frame correlation, softening residual frame-to-frame oscillations introduced by the PID controller.

3D anchor extraction: the edit-aware importance score over the full temporal volume is:

$$ s_{t,i,j} = \frac{|F^{(t,i,j)}_{\text{ref}}|}{1 + |F^{(t,i,j)}_{\text{ref}} - F^{(t,i,j)}_{\text{gen}}|} $$

Positions with high reference magnitude and low edit divergence score highest, becoming Sub-Solution anchors. Frame 0 naturally receives the densest anchors (being closest to the unmodified reference); later frames receive sparser anchoring, in proportion to how much freedom the model has used to animate.

Training convergence acceleration — a sign the principle may reach beyond images

The same dual-phase error-correction logic that governs diffusion denoising can, we found, be applied to gradient-based neural network training, where the "error" is not noise-prediction drift but the gap between current and target loss — likewise high early and diminishing later.

Phase 1 — PDC as an adaptive learning rate: we implement a PID controller on the exponentially smoothed training loss (\(\alpha = 0.9\)). The multiplier is:

$$ \mu = 1 + e + \int e\, dt - \dot{e} $$

applied as (\(lr = lr_{\text{base}} \times \mu\)). The I-term integrates accumulated loss to help escape plateaus (clamped at \((\pm 10)\)); the D-term softens the learning rate on loss spikes. This yields a fully adaptive schedule requiring no manual warmup or decay tuning — the controller discovers an appropriate schedule from the loss signal alone.

Phase 2 — LHFG in weight space: after PDC stabilizes the training dynamics, we snapshot the model weights as (\(W^{\text{stable}}\)) and partition all weight positions by gradient magnitude (\(|\nabla_W L|\)). The bottom (k%) (where a small gradient suggests convergence) forms the Sub-Solution and is regularized toward the stable reference: (\(\mathcal{L}{\text{anchor}} = \sum{{i,j \in \text{anc}} |W{i,j} - W^{\text{stable}}{i,j}|^2}\)). The remaining high-gradient positions form the Sub-Problem and are penalized by the squared discrete Laplacian: (\(\mathcal{L}{\text{smooth}} = \frac{1}{|\text{lrn}|} \sum{i,j \in \text{lrn}} (\nabla^2 W{i,j})^2\)). When this penalty vanishes, every Sub-Problem weight equals the average of its neighbors — satisfying the discrete Laplace equation exactly, now in weight space.

MNIST results: all three configurations (PDC only, LHFG only, and PDC+LHFG combined) consistently outperformed the vanilla SGD baseline across training loss, training accuracy, validation loss, and validation accuracy — with the combined variant achieving the best performance on all four metrics. Sensitivity analysis shows the PDC-to-LHFG transition timestep (\(T_{\text{PDC,end}}\)) is the single most dominant parameter (\((r_s = -0.727)\), \((p < 0.001)\) on training loss), suggesting that the duration of the adaptive-learning-rate phase is the primary lever for convergence quality. We offer these results modestly — MNIST is a small proving ground — but they encourage us that the underlying principle is not confined to image generation.

How SoulCanvas compares with existing systems — shared with respect

We evaluated SoulCanvas alongside several genuinely impressive systems: Nano Banana Pro (Gemini 3 Pro Image), FLUX.2-dev, Qwen-Image-Edit-2511, MS-Diffusion, and OmniGen. Each of these represents an enormous amount of skilled and dedicated work, and our project quite literally stands on the shoulders of this research community. We share the following measurements not to diminish anyone's contribution, but because honest, careful measurement is part of respectful science — and because quality directly affects the people we hope to serve.

Four metrics were used: CLIP-T (text-image alignment), DreamSim (perceptual identity similarity), LPIPS (diversity/perceptual distance), and FID (distributional quality).

  • Consistent image customization: FID 88.38 versus the next-best MS-Diffusion at 118.47 (−25.4%); DreamSim 0.1055 versus 0.1577 (−33.1%).
  • Inpainting: first on all four metrics; DreamSim 0.0724 versus OmniGen's 0.0958 (−24.4%); FID 22.65 versus 25.48 (−11.1%).
  • Multi-subject composition: DreamSim 0.1561 versus Qwen's 0.3260 (−52.1%); FID 214.49 versus Qwen's 281.66 (−23.8%).
  • Image fusion: best LPIPS (0.5478); second-best DreamSim.
  • Colorization: best CLIP-T (0.8956) and best DreamSim (0.1881).
  • User study: we invited 50 volunteers to a blinded study (1–5 Likert scale across Reference Similarity, Prompt Match, Image Quality, and Diversity), and we are sincerely grateful to every participant for their time and care. SoulCanvas received mean scores of 4.80–4.84 across the five tasks and was ranked first across all four criteria in all five tasks, with statistically clear differences (the closest alternative did not exceed 3.50 on any task). We receive these results with gratitude — and with the honest caveat that a 50-person study is an early and encouraging signal, not a final verdict.

Efficiency — and why it matters for access, not for bragging rights

One result matters to us more for its human implications than its technical ones: SoulCanvas averages about 3.65 seconds per image on under 9 GB of VRAM. In our test environment, the slowest alternative (Qwen-Image-Edit-2511) took roughly 80.3× longer per image, and the most memory-intensive (FLUX.2) required about 7.0× more VRAM. In practical terms, SoulCanvas can generate 72 multi-subject compositions in the wall-clock time FLUX.2 needs for one, and it runs comfortably on a consumer 12 GB GPU — a hardware tier the larger systems cannot fit within.

We highlight this not as a victory over other methods, but because modest hardware requirements are precisely what make deployment realistic in the underserved regions where mental-health infrastructure is scarcest. For this project, efficiency is a matter of equity (SDG 10), and we regard it as one of our most meaningful results. The efficiency derives from operating entirely within a standard SDXL backbone (24 denoising steps) with lightweight PDC and LHFG modules that require no auxiliary encoders, no batch-size inflation, and no persistent parameter buffers.

Hyperparameter sensitivity and a gentle tuning guide

We conducted a systematic sensitivity analysis via Tree-structured Parzen Estimator (TPE) sampling across 30–40 independent configurations per task, computing Spearman rank correlations between each hyperparameter and each metric. The key finding: the phase-boundary timestep (\(T_{\text{phase1}}\)) is the single most influential parameter, with the strongest and most frequently significant correlations (\(p < 0.001\)) across tasks. This suggests that identity preservation is governed primarily by how long PDC remains active, rather than by the specific PID gains alone. Among the gains, \((K_I)\) (the integral term) most consistently strengthens identity fidelity across tasks. All LHFG structural parameters (\((k)\), \((N_{\text{Jacobi}})\)) can safely remain at their optimized defaults. For anyone kind enough to build upon this work, the practical tuning hierarchy is: set \((T_{\text{phase1}})\) first, then \(K_I\), then \(K_D\) for fine control — everything else is pleasantly robust to variation.

Challenges we ran into

The central technical challenge was designing a training-free guidance framework that could meaningfully improve diffusion model outputs without touching model weights. Much of the excellent work of 2024–2025 suggested that controllable generation requires either fine-tuning (LoRA, DreamBooth) or architectural modification (ControlNet, IP-Adapter) — approaches with real strengths that have helped countless creators. We simply wondered, perhaps a little naively at first, whether a third path might exist for situations where retraining is impractical or unaffordable.

The deeper challenge was generality. It is relatively straightforward to hand-craft a guidance signal for one specific task. We hoped instead for a single principled framework — not a bag of tricks — whose core mathematical formulation could address a broad class of problems: identity-consistent editing, temporal smoothing in video, multi-subject composition, inpainting, style transfer, and even training-convergence acceleration in unrelated deep learning tasks. Finding that unifying principle — that the denoising trajectory itself exhibits a "high drift early, fine correction late" pattern, analogous to PID control systems and boundary value problems — took many failed hypotheses. We are thankful for each of those failures; every one taught us something the successes could not.

Other significant challenges

The mid-noise instability gap: in the mid-noise phase, diffusion models enter a regime where neither global structure nor local detail is settled. Existing methods (classifier-free guidance, attention manipulation) apply uniform pressure across all timesteps, which allows identity drift precisely in this critical window. Designing a mid-phase component to predict and pre-correct spatial drift before it cascades required borrowing humbly from control theory — a field not often applied to generative models, and one we grew to appreciate deeply.

Harmonic field computation at inference time: the low-noise component models detail refinement as a Dirichlet boundary value problem. Solving Laplace's equation at every denoising step naively would be prohibitively expensive. We needed an efficient approximation (Jacobi iteration with early stopping) that converges in only a few iterations without visible quality loss.

Solo development across the full stack: building a research contribution (the CVPR 2026 paper), a production Gradio application, voice integration, LLM fine-tuning for therapeutic dialogue, and a user-facing therapy product — simultaneously, and alone — required careful and sometimes painful prioritization, along with an architecture designed for modularity from the very first day. There were moments when I felt stretched very thin, and I share this honestly in the hope that it encourages other solo builders: it is possible, with patience and structure.

Accomplishments that we're proud of

We would like to share these milestones with gratitude rather than pride — gratitude to the open-source community, to the researchers whose work we build upon, and to everyone who tested, critiqued, and improved this project along the way.

  • Encouraging results without training a single parameter. In our evaluations, our training-free approach performed favorably alongside leading image-editing systems — qualitatively, quantitatively, and in user preference. This outcome genuinely surprised us, and we hold it humbly: benchmarks capture only a part of what makes any system valuable.
  • Encouraging qualitative results alongside Nano Banana Pro, FLUX.2, Qwen Image Edit, MS-Diffusion, and OmniGen across identity-consistent editing, style transfer, and composition tasks. In the tasks we studied, our outputs maintained sharper identity, cleaner line art, and more coherent global structure.
  • Zero-cost, plug-in improvements to models we admire:
    • FLUX Kontext: our guidance meaningfully improves reference image style preservation — a difference visible to the eye, not only in metrics.
    • CogVideoX: our temporal-consistency guidance softens abrupt motion artifacts (for example, sudden hand-lift postures in generated video) and produces more physically plausible, smoother motion without any video-specific fine-tuning.
  • Consistent quantitative placement at first or second across four standard metrics — CLIP-T (text-image alignment), DreamSim (perceptual similarity), LPIPS (diversity), and FID (distributional quality) — against the same respected baselines.
  • A warm reception in our user study: human evaluators rated SoulCanvas most favorably across all four evaluation dimensions — Reference Similarity, Prompt Match, Image Quality, and Diversity — in all five tasks. We are thankful to our 50 volunteer evaluators, without whose generosity this evidence would not exist.
  • Evidence that the principle may generalize: the same core idea (predict drift, then correct the trajectory) also accelerated training convergence in a standard deep learning classification network. This encourages our belief that the framework speaks to a broader class of problems characterized by "high error early, low error late" — though we present this as an early indication rather than a settled claim.
  • Acceptance at CVPR 2026: we are honored and grateful that this work was accepted at CVPR 2026 through double-blind peer review, with sole first authorship. We owe a genuine debt to our anonymous reviewers, whose thoughtful feedback made the work considerably better than it began.

What we learned

  • The best frameworks come from asking "what is the same?" rather than "what is different?" For a long time, we treated manga generation, video smoothing, inpainting, and multi-subject composition as separate problems requiring separate solutions. The turning point came when we stopped studying what distinguished these tasks and instead asked what pattern they all share. The answer — a noisy-to-clean trajectory in which early drift compounds into late-stage artifacts — opened the way to a single mathematical framework that served them all. We suspect this lesson extends well beyond AI: the most helpful abstractions tend to emerge from finding unity in apparent diversity.
  • Training-free does not mean effort-free — it means principled. Choosing not to touch model weights required us to understand deeply why diffusion models struggle, rather than papering over the difficulties with gradient descent. This constraint — frustrating at first — ultimately gave us a more interpretable, more portable, and more theoretically grounded solution than fine-tuning likely would have, and we are grateful for the discipline it imposed on us.
  • Therapy and technology share a common truth: presence matters more than perfection. Building the therapeutic side taught us that users do not need a flawless AI counsellor. They need one that is simply there — at 3 AM, without judgment, without a waitlist. Similarly, the companion's appearance does not need to be photorealistic to be emotionally meaningful. A simple anime character that a user designed themselves creates a deeper bond than the most technically impressive render they had no hand in creating. The act of creation is the connection.
  • For solo developers, modular architecture is not optional — it is survival. With more than ten features, six model classes, and multiple hardware targets (NPU and CUDA), this project would have collapsed under its own weight without strict separation of concerns. The early decision to build DPSModel as a standalone base class with lazy loading and automatic invalidation on model switch — maintained with quiet discipline throughout — is what allowed one person to complete what might otherwise have needed a team. We share this in case it helps another solo builder someday.
  • Lived experience is an irreplaceable design compass. Every UX decision — the companion's persistence (because consistency builds trust), the creative exercises (because making art was part of my own recovery), and many smaller choices besides — was informed by knowing what it feels like to be on the other side. No amount of user research can fully substitute for having been the user. I say this with humility, not authority: my experience is one experience, and we remain eager to listen to and learn from others whose journeys differ from mine.

Who does this help

Primarily: individuals experiencing loneliness, mild-to-moderate depression, anxiety, or emotional burnout, who would be comforted by an always-available, non-judgmental companion.

Secondarily: Expressive Arts Therapy practitioners seeking thoughtfully designed, AI-augmented tools for client exercises.

Additionally: AIGC content creators who would benefit from a comprehensive, multi-modal generation studio — and whose support, in turn, helps sustain the therapeutic mission.

A note on safety, limitations, and humility

Because SoulCanvas touches something as precious as mental health, we feel a responsibility to be explicit about what it is and is not:

  • SoulCanvas is a complement to professional care — never a replacement. We state this in the application itself, and we design every feature with this boundary in mind.
  • We encourage anyone in crisis to reach out to local emergency services or crisis hotlines. We are committed to building gentle in-app pathways that guide users toward professional and emergency resources whenever conversations suggest they are needed.
  • Our evidence is early. A 50-person user study and standard AIGC benchmarks tell us the technology works well; they do not yet tell us it heals. That is precisely why a clinical pilot study, conducted with university counselling centers and measured with validated instruments (PHQ-9, GAD-7), is the very first item on our roadmap.
  • We welcome correction. Clinicians, researchers, and people with lived experience will see things we cannot. We would be sincerely grateful for their guidance, and we intend to keep listening.

What's next for SoulCanvas

Near-term (3–6 months):

  • Clinical pilot study: partner with university counselling centers to run a controlled trial measuring SoulCanvas's impact on PHQ-9 (depression) and GAD-7 (anxiety) scores — as a complement to human therapy, never a replacement. We hope to earn peer-reviewed evidence of efficacy, and we will report the results honestly whatever they show.
  • Mobile-first redesign: migrate from Gradio to a native mobile experience (React Native, with on-device LLM inference via llama.cpp). Healing happens on couches and in beds, not at desks — the tool should meet people where they are.
  • Expanded therapeutic modalities: guided journaling, mood tracking with AI-generated visual metaphors (one's mood as a landscape that gently evolves over the weeks), and CBT worksheet generation tailored to the conversation context.

Medium-term (6–12 months):

  • Therapist dashboard: allow licensed practitioners to "prescribe" SoulCanvas exercises to their clients between sessions, review AI-generated session summaries (always with the client's informed consent), and adapt the companion's therapeutic approach to each person's needs.
  • Multi-modal memory: the companion remembers past conversations, warmly references previous artwork made together, and notices patterns with care ("You've mentioned feeling isolated three times this week — would you like to explore that together?").

Long-term vision (1–2 years):

  • SoulCanvas as a platform: invite therapists and artists to publish custom "therapy journeys" — guided multi-session creative programs such as grief processing through portrait evolution, anxiety management through world-building, and identity exploration through character design.
  • Localization for underserved regions: deploy lightweight versions optimized for low-bandwidth environments in Southeast Asia, Sub-Saharan Africa, and rural China — where mental-health infrastructure is most scarce and the need is greatest. This is, to us, the heart of the project's SDG commitment.
  • A companion that grows with you: using longitudinal interaction data (kept fully private and on-device), the companion's personality, visual style, and therapeutic approach would gently evolve alongside the user's healing journey — a little more playful as depression lifts, a little more grounding during anxious seasons.

Our guiding hope is a simple one: no one should have to heal alone because help was priced, distanced, or stigmatized out of their reach.

Technologies and tools used

  • Language: Python
  • Diffusion models: SDXL (multiple base models), Wan2.2, CogVideoX-5b, FLUX.1-Kontext-dev
  • LLM: fine-tuned counselling language model (bilingual CN/EN)
  • Voice: Qwen3-TTS voice synthesis
  • Frameworks: Gradio (interactive web UI), Diffusers, PyTorch
  • Hardware: Huawei Ascend 910B3 NPU (64 GB HBM), with full backward compatibility to NVIDIA CUDA GPUs
  • Research foundation: peer-reviewed paper accepted at CVPR 2026 (sole first authorship)

Built With

  • cogvideox
  • flux-kontext
  • qwen3-tts
  • sdxl
  • wan2.2
Share this project:

Updates

Submission history