StitchGeni

Adaptive Knitting Tutor

Comment

StitchGeni — Adaptive Knitting Tutor

Inspiration

StitchGeni was inspired by a very human problem: knitting patterns are often hard to understand, inaccessible, and visually inconsistent. Many patterns rely on abbreviations, dense charts, or assumed prior knowledge. This creates barriers for beginners, children, non-native speakers, and especially visually impaired users.

The idea grew from asking a simple question:
What if knitting instructions could adapt to the person — not the other way around?

Instead of forcing users to interpret unclear diagrams or shorthand, StitchGeni aims to translate visual knitting knowledge into clear, spoken, step-by-step guidance, aligned with visuals that truly match each instruction.

What it does

StitchGeni is an adaptive knitting tutor that:

  • Analyzes knitting images, charts, or textual descriptions
  • Converts them into complete, start-to-finish knitting instructions
  • Adapts patterns for:
    • Specific ages (for example, a 5-year-old or 12-year-old)
    • Garments (hat, jacket, pants, etc.)
  • Produces structured steps, where each step includes:
    • Clear written instructions (no abbreviations)
    • A matching instructional image or diagram
    • Audio narration (text-to-speech)
    • A simplified re-explanation if the user gets stuck
  • Supports multiple languages (English, German, Turkish)
  • Maintains a session gallery so users can revisit previously generated patterns without regenerating them

Accessibility is a core principle: every step is understandable even without seeing the image.

How we built it

StitchGeni was built using:

  • Google AI Studio with Gemini models for:
    • Pattern reasoning and structured instruction generation
    • Image understanding and step-aligned diagram generation
    • Text-to-speech for narrated instructions
  • TypeScript + React for a clean, accessible frontend
  • Gradio-style interaction patterns adapted into a custom UI
  • A strict JSON schema to ensure deterministic, reusable outputs
  • Accessibility-first UI design:
    • Keyboard navigation
    • ARIA labels
    • “Skip to main content” support
  • A modular architecture:
    • geminiService.ts for AI orchestration
    • Component-based UI for patterns, steps, and gallery reuse

Each knitting step is treated as a unit that can be: [ \text{explained} \rightarrow \text{visualized} \rightarrow \text{spoken} \rightarrow \text{re-explained} ]

Challenges we ran into

  • Visual alignment: Generated images did not always reflect the exact instruction. This required refining prompts so visuals are instructional diagrams, not decorative images.
  • Language leakage: Images sometimes contained English text even when Turkish or German was selected, requiring explicit language constraints in prompts.
  • Ambiguous inputs: Knitting images often lack full context (yarn weight, gauge, size), forcing the system to safely mark outputs as approximate.
  • Accessibility balance: Making instructions detailed enough for beginners without overwhelming users.
  • State vs storage: Designing a session gallery without violating privacy or assuming consent for long-term storage.

Accomplishments that we're proud of

  • Built a deterministic, reusable knitting instruction system
  • Achieved true step-to-image alignment, not generic visuals
  • Integrated audio narration for hands-free learning
  • Designed a system that respects privacy by default
  • Created a gallery experience that encourages reuse instead of regeneration
  • Made knitting instructions understandable without abbreviations or charts

What we learned

  • Multimodal AI is most powerful when structure comes first
  • Accessibility improves everyone’s experience, not just edge cases
  • Visual generation must be tightly constrained to be educational
  • Clear schemas are essential when combining text, image, and audio outputs
  • Creative tools still need strong guardrails to be reliable

What's next for StitchGeni

Next steps include:

  • A visual diagram renderer for knitting steps (symbols + stitches)
  • Optional “Üzerimde gör” (see it on me) mode using user-provided photos
  • Smarter pattern similarity detection to avoid duplicates
  • Long-term gallery support with explicit user consent
  • Offline-friendly pattern exports (printable + audio)
  • Expanding language and accessibility support further

StitchGeni is not just about knitting —
it’s about making skilled knowledge adaptive, inclusive, and human.

Built With

Share this project:

Updates