The world of Mixed Reality isolates the user from the present world. Dissapoiting experiences with Augmented Reality It leaves no place for the user to re-interact with the real world. Such isolation prevents Mixed Reality from dominating the industry and solving critical problems. Our idea attempts to solve this discountinuity between Virtual Reality and reality by connecting the User, through Mixed Reality, to a robotic arm.

What it does

Our project uses an Inertial Measurement Unity (IMU), which has an accelerometer, gyroscope, and magnetometer, that helps measure fine user movement. By passing it through algorithms including Alpha Beta Filter (G-H Filter), and a Kalman Filter, we are better able to receive a more accurate position of the IMU in 3D space. We use an Arduino to control the IMU and feed the data to Unity where the heavy computation takes place. On the electrical side, another Arduino from different port takes the filtered data from Unity and translates them to motor movements for the robotic arm. Mechanically, we did not have the parts needed to create a sturdy Robotic Arm, so we decided to head for Aluminum cans as they are abundant, recyclable, and available.

How we built it

The IMU is connected to the Arduino. The data from the IMU is read by the Arduino and sent to the computer. In the computer, we parsed the data, ran it through mathematical filters, and displayed an object that matches the orientation of the IMU. The data is then fed to another Arduino that has a CNC Controller Board, which in turn controles the X (Roll), Y (Pitch), and Z (Yaw) of the robotic arm. Attached will be images regarding the project.

Challenges we ran into

As we started this hackathon, the first problem we encountered was making the stepper motors run. Many of the settings of the stepper motor were unknown to us and finding a manufacturer's specification sheet proved to be hard as the manufacturer was not well-known. As we started the project for the first 12 hours, a lot of our time was spent on understanding the technology and searching for example code. The need to learn a relatively low level coding language that interacted with technological components was challenging. Understanding the math behind our code as high schoolers proved to be a difficult, but fun obstacle. The most challenging aspect was our temptedness at trying new things as if we made a mistake, it could cause our electronics to die out or spark. Much care was taken in making sure we were safe when operating with the equipment.

Accomplishments that we're proud of

  • Introduced a partial Kalman Filter into our algorithm that uses state estimation to determine the exact position of the IMU in 3D space. This filter required extensive calculus knowledge and matrices to determine the states.
  • Added an Alpha Beta Filter (G-H Filter) that accounts for the state of the object in relation to the derivative of the position of its state.
  • Implemented Stepper Motors that control the movements of the Robotic Arm using an Arduino.
  • Successfully coded in Python, C# and C++, in Terminal, Unity and the Arduino IDE respectively. We added control methods and scripts on Unity and have a motor control script in C++. Python was used for higher level programming, serial libraries, and matrix calculations, and was a good environment for testing our contraption.
  • Understanding of CNC Control Board and how to operate it. Wiring was connected properly and referenced with online sources.

What we learned

This hackathon offered us the opportunity to delve deeper into the electronics world alongside bettering our skills at software. Our youngest member, a freshmen, spent the time learning to code as well as designing the mechanical aspect of our robotic arm. Both senior members participated in the full development of the project by switching off at intervals to ensure both had knowledge of the situation, of the code, and of its problems.

What's next for Tracking in 3D space, filtering, and Robotic Arm

We plan to perfect our IMU tracking algorithm to be precise and accurate. More research will have to be taken to understand the complexity of incorpating filters into our system. The robotic arm will also have to be improved on, taking into account a full range of problems consisting of insufficient parts, a lack of capital, and a high learning curve. Electrical will need to make sure the wires and system are operating a safe levels and take care of the heating problems with the stepper motor drivers. We plan to continue this project after the hackathon and are looking for any support to further our project.

Built With

Share this project: