Mud Bugs (otherwise known as crayfish) build ‘castles’ of mud towering above blades of grass in once swampy environments. They have poor vision, but relying on instinct and tactile dexterity, are able to collaboratively till and aerate an entire field even in the most adverse circumstances, that would confound modern machinery.
What it does
Mud Bugs allows groups to design an conceptualize a collaborative real-world 3D object, organize a build plan according to environmental constraints (i.e., gravity, balance, construction materials, etc.), and build a prototype in the most efficient way possible (i.e., iterative, parallel, water-fall, etc.) through stages that can be unit tested, with any differences from expectations (e.g., bugs, defects) annotated along the development path, and attached to both the physical and virtual object.
This creates a practical method of multi-user collaboration that highlights the build strategy, uncovers design bottlenecks, and demonstrates associations between form and function that would otherwise remain hidden in project complexity.
How I built it
To test the theory, I used Unity*, and Samsung GearVR* / Oculus* technology to render the AR environment. I used an open-source version of Minecraft* as a framework for modeling simple LEGO*-like structures. I then used LEGO* blocks (recolored with chroma-key paint) to act as a real-world, virtual-world building-bridge substrate, to physically construct, virtually interact with and verify compliance to the build strategy.
Challenges I ran into
- Optical recognition and shape coordination of real-world blocks in a wide-variety of lighting / shading conditions proved problematic and CPU intensive,
- Algorithms for optimal build methods are non-trivial (there really are a million ways to do things, many that are equally viable / efficient),
- Designing an intuitive (3D) navigation method around (4D) spacial and temporal ‘defects’ presents visualization issues since design flaws tend to cluster around ‘bottlenecks.’
Accomplishments that I'm proud of
Making the presentation, multi-user annotation tools and interface navigation intuitive, given the complexity of the project.
What I learned
That AR is a lot more than an interface modality, involving AI, physical simulation, audio dimensionality, and multi-user networking protocols, which any one alone would be hard enough to integrate. Taken together, AR is humbling, and even more demanding in many respects than VR, to be sure.
What's next for Mud Bugs: a Multi-user AR Quality-Control App
It would be great to polish the application and get it working to a level that would allow it to be more than a theoretical framework and working demo. Once done, it would be cool to provide it as a tool to a group of naive user’s and see what they can build together. More importantly, it would be great to test how efficiently the collaborative task is negotiated with Mud Bugs compared to a control group with only LEGOs* to work with.
———————————————- LEGOs and LEGO logo are trademarks of the LEGO Corporation. Minecraft is a trademark of Microsoft. Unity is a trademark of Unreal Technologies. GearVR is a trademark of Samsung Electronics. Oculus is a trademark of Oculus Technologies, a Facebook company.