SCABO - AR-enabled DIY cardboard toy

Assembling cardboard toys with friends
Talking to cardboard toys
Vietnamese students playing AR games with SCABO toy
SCABO toy at Vietnam school

Inspiration

Our goal is to contribute in the reduction of the usage of new plastic toy and the decrease of thousand to million tones of fossil fuel-based plastic in the toy industry. Based on UN environment report about plastic pollution in 2014, the toy industry is the most plastic intensive industry in the world. Therefore, our team strongly believes that our cardboard toy kits will demonstrate the notable example of combining eco-friendly material with MR/AR technologies to transform the toy industry to become a circular economy.

What it does

DIY cardboard toys with AR platform First, we provide DIY cardboard papers for our customers to assemble their cardboard toys with 3D interactive step-by-step instructions in our Android (several features developed in Flutter and Google Wallet)/iOS application. Then, users play AR games with the cardboard toys in MR environment. Cardboard toys could also be an assistant or a friend, and together they could talk about science subjects.

How we built it

We trained the object detection module to detect 2D position of cardboard toys in the images. Then, we map the 2D position into 3D world space position via ray trace algorithm. Then, our collision detection algorithm is used to detect the collision between physical toys and virtual characters. To render output, we utilize the Sceneform and Scenekit built in Java and Swift for Android and iOS devices.
With the help of Flutter SDK, we were able to integrate Google Wallet API into our application. We also added a Flutter QR Scanner module to scan existing QR, as well as the user' Wallet Pass, to provide them with an interesting sharing experience. These modules are quickly integrated into our architecture as module source code. Then the flutter app communicates with our host application through Flutter Platform Channels.
Here are some of the resources that help us. Module source code Integration, Flutter's Platform Channel, Flutter Screen in Native Host

Challenges we ran into

The main players in the toy manufacturing industry have not changed the materials used to produce their products in the last 40 years. Therefore, our solution should meet the following criteria: inexpensive, recyclable, educational and edutainment to compete with the giants in the industry. Our team has gone from strength to strength by developing more than 20 versions of DIY cardboard robots and cardboard-only toy to let users to assemble their own cardboard toys at homes or at schools. In addition, integrating computer vision tech stack into AR projects was NOT trivial task since the user experience should be smooth while playing AR games (update rate should be at least 30 frames per second - all computation from rendering, object detection and collision detection).

Accomplishments that we're proud of

First, we have mastered the mass production process for our cardboard toy. We also partnering with local recycled waste companies to recycle cardboard waste. Secondly, we have been interviewing with more than 2000 potential users and deliver more than 2000 toy kits to gain feedback from them. After visiting more than 20 schools in Vietnam and Thailand, we understand that user experiences should be entertaining and unique. Therefore, for children or students in remote/local areas, several features of our SCABO app (Android,iOS) can be used without the internet connection. Last but not least, we have different types of cardboard toys for customers with lower prices than other toy manufacturers in the market. Although our prices are 1/3 or 1/5 of the prices of normal toys in the market, we enrich user experiences with AR scenarios - games and edutainment contents.

What we learned

In terms of reducing plastic pollution in the toy industry, we developed 2 features so-called SHARING cardboard toys with YOUR BUDDY and SHARING AR scenarios to YOUR BUDDY to promote the spirit of reduce, re-use, and recycle. The Google Wallet API is best fit for the purposes to let users to share their generic owner pass for the cardboard toy with their friends and share the virtual 3D tower between friends to play AR games. Last but not least, Google Wallet API allows our users to have electronics warranty for electronics parts of the cardboard robot. We are working with recycling partners to allow our customers to use their pass in Google Wallet to collectively convert their recycled waste to our cardboard toys in pilot program in Vietnam and Thailand.

What's next for SCABO - AR-enabled DIY cardboard toy

Our team is going to finish the core loop for our application to motivate users to play and learn in AR. Then, we are planning for a mixed reality hand-free experience with AR glasses for end-users. Our team's goal is to develop our application in commercial AR glasses like Apple Vision Pro, and Meta Quest. In terms of research, our team has been working 3D computer vision technologies like 3D reconstruction and tracking from the scratch. Our goal is let users to scan their plastic toy and play with their plastic toys in the same AR scenarios. Lastly, our goal is to export products to different developing countries in Asia, Africa and South America to support educators and learners to utilize AR/MR technologies for their learning and teaching activities.