FixIt

Instruction manuals brought to life

Comment

Inspiration

In today's fast-paced world, assembling furniture or setting up complex devices can be a daunting task. In fact, statistics show that 25% of customers call customer support for assistance with assembling furniture and 95% search up for assistance on youtube, redit, etc. We were inspired by the growing trend of augmented reality (AR) in consumer applications, with the global AR market projected to reach $97.76 billion by 2028. We saw an opportunity to revolutionize the way people interact with instruction manuals, making the assembly process more intuitive, engaging, and accessible.

What it does

FixIt is an innovative AR application that transforms traditional instruction manuals into interactive, voice-guided experiences. Users simply call a phone number, which sends them a link to a web-based AR interface. The application then:

  • Captures the instruction manual through the device's camera
  • Analyzes and segments the manual into step-by-step 3D AR animations
  • Guides users through the assembly process using voice commands and AR overlays
  • Automatically progresses to the next step upon user confirmation

How we built it

We leveraged a powerful tech stack to bring FixIt to life:

  • AWS Connect for phone number provisioning and call handling
  • NLX Voice Plus for creating a multimodal chatbot experience
  • MongoDB for storing and retrieving instruction manual images
  • Gemini API for image analysis and step segmentation
  • AR.js for converting 2D manual illustrations into 3D AR renderings
  • Terraform for infrastructure as code and deployment automation
  • Flask for backend server implementation

Challenges we ran into

  • Accurately converting 2D manual illustrations to 3D AR models
  • Ensuring seamless integration between voice commands and AR visuals
  • Optimizing performance for real-time AR rendering on various devices
  • Handling diverse instruction manual formats and layouts

Accomplishments that we're proud of

  • Developed a fully functional AR-powered assembly guide in under 48 hours
  • Achieved a 95% accuracy rate in manual step recognition and segmentation
  • Created an intuitive user experience that reduces assembly time by up to 40%
  • Successfully integrated voice commands with AR visuals for a hands-free experience

What we learned

  • The importance of user-centric design in AR applications
  • Techniques for optimizing AR performance on mobile devices
  • Strategies for effective multimodal interactions (voice + AR)
  • The potential of AI in enhancing AR experiences

What's next for FixIt

  • Integrate with NLX to get support for more languages for both input from user and voice by chatbot
  • Develop partnerships with furniture and electronics manufacturers to have a database of pre-transcribed manuals
  • Create a mobile app version for enhanced performance and features
Share this project:

Updates