In The Loop

In The Loop is an interactive, XR application that teaches you the foundational concepts behind knitting! Using intuitive animations and a custom ML model, it moves at your pace.

Comment

Inspiration

Textiles have played a huge role in the development of technology throughout human history, but in the modern era, they have largely been overlooked by society. Knitting can offer a great way to unwind, reflect, get in a flow, and get away from the instant reward dopamine economy we live in nowadays. But knitting is mechanically complex to do and can be tricky to learn, that's why this hack would aim to lower the friction with learning how to knit and experience a tradition that is fun and expressive.

What it does

In The Loop teaches the user how to do the knit stitch, using XR. A custom MobileNet neural network (trained through Vultr) is used to track the user's progression through a knit stitch. It can tell if the user has completed a step, and automatically move to the next step instruction.

How we built it

Our project consists of three key components: The user interface (Lens Studio), the 3D animations & models (Blender), and our step identification system (Neural Network). We began by first all learning how to conduct the knit stitch. We were able to break the foundational stitch into four parts: Insert, Wrap the Yarn Around, Pick Up The Stitch, and Throw It Off the Needle. We documented the common mistakes that we made when first learning and the advice that helped members correct their mistakes, so we can use it in our program. Our team used Snap's Lens Studio to develop the user interface and implement the controls. In Blender, we modified a public domain hand asset, made custom needle and yarn assets, and rigged them to create five animations to demonstrate proper posture and technique. For our neural network, we recorded videos of us conducting each step of the knit stitch correctly and incorrectly from several angles, and then aggregated the different frames together into one dataset. We used Roboflow to annotate our frames and split them into training, validation, and testing sets. We then used Vultr to train our datasets and get our model in an ONNX file format for SnapML. Together, these all work together to display to the user how to conduct the current stitch, and validate if the user has done it correctly.

Challenges we ran into

Our biggest challenge we ran into was training and configuring our model. We had issues training a model to accurately understand what each part of a stitch is and what is the right and wrong technique for it. We tried to remedy this by eliminating variables like putting gloves on people's hands to minimize differences from skin tone, using high contrast needles and yarn to make things easy to analyze, and reducing stitches on the needle to minimize overall detail and noise. Integrating all of our systems together into one cohesive package while fixing bugs proved to be a more laborious task than expected, causing us to compromise on some originally planned features, like a live feedback loop with custom-suggestions based on visual input.

Accomplishments that we're proud of

As a team, we were all brand new to Snap Studio and had very limited experience in 3D development software, like Unity. Given that we didn't have much experience going in, we are very proud that we were able to develop an application for the Snap Spectacles. The 3D models and animations are also a unique touch that we are extremely grateful for, especially since it fills a gap in a lot of online resources in learning to knit. Many resources have 2D images and videos, which creates ambiguity in completing certain knitting steps.

What we learned

Through this experience, we learned how to take advantage of 3D environments to best demonstrate and teach new skills. When making a program in 3D, simply slapping a 2D interface into a 3D environment doesn't work. We had to consider the layout around the user, and how to interact with real, physical objects the user is holding. While developing, we also learned how to maximize efficiency using agentic workflows.

What's next for In The Loop

For this project, we'd love to collect for data samples so users can knit along using any needle or without needing any gloves. We also plan on adding support for more stitches (purling, k2tog, ask), teaching other knitting techniques (colorwork, brioche), crocheting, UI refinements, and more. Knitting has such a deep and rich history with so many forms of expression and ways to enjoy it. Truly the ways this can be expanded are near limitless!

Built With

Share this project:

Updates