SpectaculAR Signs

• 

  Onboarding Menu

• 

  Teacher Starting Sign Recording

• 

  Student Learning Sign for "computer"

Inspiration

Our primary inspiration was with research we are involved with at Georgia Tech in sign language education, where we work towards building ethically to empower the over 28 million deaf and hard-of-hearing Americans rather than replace them. Since over 90% of deaf children are born to hearing parents, it is imperative that the friction for early ASL learning is decreased enough that parents can support their children's development and education. We wanted to carry these values while taking advantage of the unique Spectacles development environment, leading to a particularly powerful user experience.

What it does

In SpectaculAR Signs, nearby objects are recognized by Gemini and filtered based on the existing data set so that users, either teachers or students, can record motion data onto Snap Cloud for new signs or test their skills against signs previously recorded by teacher users. This UGC system allows for a more realistically applicable word set that appears in most users' day-to-day, along with supporting community growth and more effective education by relying on individuals within the community we are choosing to support to create content.

How we built it

Using Lens Studio, we were able to create our basic structure with Depth Cache and the hand tracking system, eventually setting up motion data storage and complex input reading and validation to create a comprehensive experience.

Challenges we ran into

We had to handle precision on the Spectacles hand-tracking system in a short period of time, needing to effectively tune and construct our recognizing/recording system in ways that were easily repeatable while being useful for a variety of different kinds of signs. We also faced challenges working with the spatial interface, but found that including our own unique interaction modalities made the overall product feel more complete and far less frustrating.

Accomplishments that we're proud of

We are certainly proud of being able to deliver a viable product that aligns to lessons and information we've learned at the lab here at Georgia Tech, and are especially proud of how our work successfully takes advantage of the unique opportunity that running on Spectacles provides. Managing such a large-scale finished product as a team of two was definitely also an accomplishment.

What we learned

We learned how to effectively design and iterate for the Spectacles with Lens Studio, and had to better understand where we can take advantage of 3D offerings to strengthen systems that would previously have been solved with basic 2D interfaces and potentially camera recognition setups.

What's next for SpectaculAR Signs

We would love to continue creating a larger dataset of signs and collaborating with more creators in the accessibility space to work on Spectacles, to hopefully find how projects like SpectaculAR Signs can fit into a growing ecosystem of technology that supports communities globally.

Built With

• gemini
• snapcloud
• typescript
• ugc