StyleMemory

StyleMemory is an image studio that learns your characters, styles, and feedback, helping each generation stay on-model—without per-character LoRA training.

Comment

Inspiration

Comic artists and indie game developers need one thing text-to-image generation has never reliably given them: the same character, looking the same, across dozens of images.

But generation is stateless. The same prompt produces a different identity every time, and nothing you made last session carries over to this one. I looked at it from the side of someone actually trying to ship a comic: a character whose face drifts between pages breaks the story faster than bad rendering does. At that point you stop reading the characters and start noticing the mistakes.

The usual fix is a LoRA per character — collect 15+ curated images, run a GPU training job, ship a separate model file for every character. It works, but every new character costs you a dataset and a training run, and it still does nothing for art styles, color palettes, or simple feedback like "stop making it so dark."

So I asked a narrower question: how much consistency can you get from memory alone — no training, no reference images, no model file per character?

What it does

Characters, style profiles, and learned preferences all live in MemOS, a local-first memory layer I built as a separate open-source project. Every generation runs through five steps:

  1. Recall what is relevant — whole-word name match first, then a relevance-thresholded text search, plus any learned traits that are always active.
  2. Build an enriched prompt by folding those memories into the positive conditioning.
  3. Generate through ComfyUI with Qwen Image 2.1 Turbo, streaming live progress for each step.
  4. Show its work — you see the exact final prompt that was sent and every memory that shaped it.
  5. Learn from feedback, so a thumbs-down with a note writes a trait back to memory and the next generation improves.

The most important design decision: memory is the only thing that changes between two runs. That makes a same-seed A/B comparison an exact test of what memory contributes — same prompt, same seed, same image size, memory off in one and on in the other. It turns "it remembers your character" into something you can measure instead of just claim.

How I built it

pnpm workspaces, a Hono and TypeScript server, a Vite and React frontend with Tailwind, ComfyUI running Qwen Image 2.1 Turbo as a Q5_0 GGUF, the @mem-os/sdk package for memory, and a JSON index alongside a local image folder. Nothing here needs a paid API.

The whole thing is a pipeline, not a wrapper: recall → prompt construction → diffusion → feedback → re-store. The ComfyUI workflow is read from disk and inspected at runtime to find its node IDs, so nothing is hardcoded — swapping the workflow file changes the pipeline without touching any code.

To check whether memory was really doing anything, I wrote a consistency suite that runs the same prompt as same-seed pairs and reports two independent signals.

The first is pixel divergence, the mean absolute difference between the memory-off and memory-on images, computed over a grayscale projection:

$D = \frac{1}{255 \cdot N_{px}} \sum_{i=1}^{N_{px}} \left| L_i^{\text{off}} - L_i^{\text{on}} \right|$

where the luma value at each pixel is

$L = 0.299R + 0.587G + 0.114B$

Because the seed and image size are identical, the only difference is the prompt conditioning — so (D) directly measures how much memory steered the generation.

The second is the recall rate, the fraction of runs where the memory-on version actually pulled up a memory:

$R = \frac{n_{\text{recalled}}}{N}$

That second number matters because a prompt matching nothing stored produces an identical image, and the score should admit that rather than take credit for it.

Across 20 same-seed pairs (40 generations, 33 minutes of compute) on "Kai skateboarding at sunset":

Metric Result
Memory recall rate 20/20 (100%)
Mean pixel divergence 15.1% (median 15.1%, σ = 2.3 pts)
Range across runs 11.4% – 19.0%
Pixels visibly changed 68.7% mean

The recall rate is what proves memory is doing real work — every single run retrieved the character, so no pair is an accident of matching nothing. The tight spread matters too: with (\sigma = 2.3) points across 20 independent seeds, the effect reproduces reliably rather than being one lucky draw.

You can also see it by eye. The same prompt with memory off produced a generic figure with dark hair in a dark button-down. With memory on, it produced the stored character — electric blue hair, a red windbreaker, ripped jeans, a crouched stance.

Challenges I ran into

CFG 1 ignores negative prompts completely. Qwen Image 2.1 runs at CFG 1, where ComfyUI discards negative conditioning entirely. Learned "avoid" traits had to be rewritten as positive phrasing instead ("too dark" → "bright, well-lit"), and the negative node is only filled for higher-CFG workflows.

The Turbo checkpoint wants 8 steps, not 25. It ships with a distilled 8-step sigma schedule. Forcing that already-distilled trajectory through 25 uniform steps over-iterates the coarse phase and starves the fine detail — I measured it as sharp ground with fur that never finished denoising. Dropping to 8 steps fixed the quality and cut generation time ~2.7×.

Memory recall was contaminating unrelated prompts — the worst bug of the build. A relevance fallback in MemOS returned a constant score of 1.0 for every node regardless of query, so the highest-importance character was injected into every prompt. I only caught it by looking at the output: I asked for "a red fox in a forest" and got a fox wearing the character's jacket, fur melting into the fabric. Fixed by removing the fallback and adding a relevance floor.

ComfyUI's progress events are unreliable. I measured back-to-back runs delivering 16 frames and then 1. A socket-only progress bar would sometimes sit at 0% for an entire generation, so the socket drives the bar when it has data and a measured clock model keeps it moving when it doesn't. It never claims completion on its own — only the authoritative history poll does.

Image quality was a configuration problem before it was a model problem. Two of my three worst image bugs were launch flags and step count. --fast fp8_matrix_mult alone caused visible softness and color drift, and I only found it by reading the actual process arguments instead of trusting what the images looked like.

Accomplishments I'm proud of

  • A working, measurable intervention point. The same-seed A/B turns "it remembers" into something checkable, and the suite made the project's central claim falsifiable.
  • Quarantine. Feedback naming a defining attribute ("hair color is wrong") is held for review rather than written to memory — it's usually a complaint about one character, not a general preference.
  • Version history via supersede(). Editing a character never deletes and recreates; the full chain stays queryable.
  • Progress that doesn't lie. The bar never claims completion; only the authoritative history poll does.

What I learned

  • Measuring beats asserting. The consistency suite caught the recall bug indirectly, and it's the only reason someone else can check my main claim. Without it I'd have shipped the contamination bug.
  • Don't trust a similarity metric you can't explain. I tested LPIPS and DreamSim and rejected both — each needs a model download I don't have, and neither can say whether the memory-on image is better. They measure distance, not quality. The pixel delta is openly a proxy; the recall rate is the half that evidences memory working.
  • Refusing to trust some input is itself a feature. See quarantine above.

What's next for StyleMemory

Semantic recall — the blocker is fixed upstream, and it was never really ours. MemOS's FTS5 query path used to wrap the entire query in double quotes, turning every natural-language search into a phrase match that requires the exact question text to appear verbatim in stored content. Since that never happens, the keyword leg of search silently returned zero rows for prose, and recall leaned entirely on name matching. That tokenization fix — term-level quoting with function-word removal, AND-first with an automatic OR fallback, and a porter unicode61 stemmer so word variants match — shipped today in MemOS v1.6.27. The remaining step is mechanical: bump @mem-os/sdk and re-run the consistency suite. No app code changes required.

Image memory — foundation shipped, generation wiring next. That same release lands image memory in MemOS: an image stored as a caption plus bytes in a content-addressed blob store, with optional CLIP pixel search, wired into retrieval, packs, the CLI, and MCP. The remaining step is attaching an image to a character and conditioning new renders on it through Qwen's multi-reference support.

Two items stay open:

  • Per-workspace memory — separate memory per project, so a comic's cast does not bleed into a game's.
  • Consistency suite in CI — run the same-seed suite as a regression gate and fail if mean divergence drops, so memory regressions are caught before release.

Built with: @mem-os/sdk · ComfyUI · Qwen Image 2.1 Turbo · Hono · React · Vite · Tailwind CSS · pnpm · TypeScript · pngjs

Built With

Share this project:

Updates

Submission history