CAMRAT: Camera-Activated Maze Rat Ability Tester (HackRats)

• 

  Unity Sample Scene Builder

• 

  Unity Game Test - PIZZA GREASE

Event

HackNJIT, Fall 2019

Inspiration

We wanted to create a program that allows a user to control a computer's mouse via the movement of a their hand. A back-and-forth motion would allow the screen to pan right or left, for instance, while the movement of the index finger would trigger a click if pressed down.

Following that train of thought, as well as the idea of "rats" and "mazes", because "mice", we decided to showcase the program via game control. Add to that the hackathon's "pizza" theme, as well as the famous "Ratatouille" film mixed with a dash of "Secret of N.I.M.H.", and we end up with a very strange, dungeon-esque game.

(The original plan was to have a less humanoid rat navigate through a cheese maze to collect pizza slices, but existing 3D assets constrain us to something slightly different.)

What it does

A different type of gesture-based "mouse control". Wiggle your fingers to make a rad rodent run rapidly round and round, collecting pizza and dodging traps.

How we built it

We built this using OpenCV via C++ and Unity 3D via C#. Some changes became necessary in order to more easily attempt to integrate the two parts of the project.

The result? High-contrast pink glove -> webcam -> C++/OpenCV gesture detector -> keyboard input to C#/Unity3D game.

Challenges we ran into

We ran into numerous challenges with this project, starting with the languages we chose to work in (C++ and Unity's C#). As some of our computers were Linux, rather than Windows, we had to also circumnavigate hardware-compatibility. Thus, an issue was attempting to cross-compile a Windows .DLL for use in Unity from Linux environments using CMake, as well as wrestling with OpenCV versioning. Also, being locked out from any modern deep learning-based hand key-point recognizers for lack of a GPU or a Python->.DLL pipeline (has to run at ~10 Hz).

Free assets in Unity 3D also fail to be exactly what you need at any given time, and, as such, we had to make do with what we had and maneuver around the pitfalls inherent in having a rigid plan and design. Being more loose and open to different methodologies has helped, but we are all novices to some degree for some combination of cross-compilation, C++ / C#, OpenCV, and Unity.

Accomplishments that we're proud of

We are proud of the old-school computer vision hand detector we were able to get running, as well as the format of the game itself.

What's next for CAMRAT: Camera-Activated Maze Rat Ability Tester

We plan to integrate the CV hand-controller with the Unity game.

Built With

• c++
• cmake
• dll
• opencv
• unity