What it does

It is a system that offer pilots the opportunity engage in a more immersive and effective flight training experience. The system is in essence an affordable flight simulator that allows completion of training hours toward license certification by allowing the pilot to fly a small scale airplane under REAL CONDITIONS. The pilot is not only able to control the aircraft but is also able to feel as if they themselves are inside the cockpit of the aircraft - this is accomplished by (1) providing a live stream of the environment within the cockpit into a virtual reality headset, (2) allowing the pilot to rotate its head freely and also (3) receive feedback from the aircraft (such as roll, pitch, yaw) which is relayed by the "motion" chair.

How we built it

1) Hacked the MultiWii open source library to introduce two new protocols. One, allow the UAV to send orientation data back to a ground control device. Two, allow the UAV to receive the pilot's head orientation which is used to move the UAV's cockpit camera.

(2) Develop a Flask application which captures a video stream and send device orientation information via web socket

(2.1) Leverage OpenCV 3.3 to capture RF video transmission incoming through an RF USB recevied. (2.2) Capture deviceOrientation (pilot's head orientation) and send it using WebSockets

Challenges we ran into

(1) Thinking about the flight industry and how we could disrupt flight education through the use of technology (2) OpenCV does not support LibV4L right off the bat, so we had to recompile openCV (3) Web sockets and regular WebSockets do not have the sae protocol

Accomplishments that we're proud of

(1) Team work and integration! (2) Technical: Building a WebSocket to SocketServer bridge (3) VideoStreaming python server

What we learned

EVERYTHING. Especially that, HARDware is HARD.

What's next for ImmersiFLY

Open Source Everything!!!

Share this project: