Inspiration
I came up with these idea from two thoughts:
(1) We live in a world in which sensors are everywhere and on everything. In this world, coming up with an embedded system isn't as hard as it used to be. Rather, now the issue is what kind of sensor and how accurate the sensor is to retrieve the data from.
(2) Humans have five major senses: touch, smell, sight, sound, and taste. How much do we know about these? Do the computers know what we are feeling in terms of these senses? The answer is that we only know what we are listening to (a.k.a. Shazam). Why can't we do this for vision?
The closest platform I know of that indirectly solves this problem is Google Glass. However, how do you tell what you are exactly looking at when there are multiple objects within the camera capture? This is where I got the idea for this project; I want to specify where your sight lies on the Google Glass-like camera capture and identify what that exact object is.
What it does
As I briefly mentioned above, it reads the electrical potential difference across your eyes, horizontally and vertically. Then, it maps these readings into a 2D coordinate on the picture that is taken from the camera that lies on pair of glasses. This represents what you are looking at in that picture.
On this version, I identify all known objects within the picture, get the boundaries, and find the closest object boundary to the sight-point. That is the final output of this platform; the predicted object that you are looking at.
How I built it
I started off with the hardware. I knew that the Electroocculogram signals have few mV and 1~30Hz signal range. Therefore, I put an instrumentation amplifier and few high pass and low pass filters to get rid of noise.
This voltage output is then fed into Arduino for ADC, from which I can get the digital serial data.
Finally, I used OpenCV 3.0.0 library to do basic computer vision process. I register few objects that are relatively easy to recognize, and I spot all of them within the camera capture. SIFT and FLANN was used for this purpose. I find the closest object boundary from this list of detected objects.
Challenges I ran into
There were a LOT of challenges due to this project's cross-hardware-software nature:
(1) Hardware debugging. I fried two op-amps and one instrumentation amplifier because I fed in wrong voltage into them. I also had to play around with resistor values and add few filters to get useable data.
(2) Arduino and PySerial interface. Controlling serial data flow from Python's end was quite difficult. Lots of errors popping up, and had to debug a lot.
(3) OpenCV 3.0.0. This newest version of OpenCV is notoriously hard to install. It really should not be. I lost precious team members because OpenCV 3.0.0 refused to work with Windows 10.
(4) Noisy Signal. Body signals are noisy by nature. Coming up with a scheme that will work well with this noisy data was quite hard.
(5) Solo project (unintentionally). As stated from point above, OpenCV 3.0.0 took my team members away, and I had to come up with the entire project myself.
(5+) Lack of sleep. It's hard.
Accomplishments that I'm proud of
I am proud that I was able to complete this project with little time and little manpower. Project could have been a lot more sophisticated otherwise, but I am very satisfied that I was able to pull all the way from hardware data collection onto computer vision identification to decently functional level.
What I learned
(1) For projects dealing with unknown concepts, it is not only important for you to be knowledgeable with the material but also your teammates. Keep them updated and prepared as much as you are.
(2) Body signal is probably the next wave of sensors that should be mastered, and I think there is a lot of potential there. Even with this time limited, undermanned project, I was able to get somewhat of a working product which I think can be used for a lot of different things. Namely, Google Glass, Microsoft Hololense, and Oculus VR.
What's next for Oqulo
As mentioned above, more noise reduction on the hardware side, better runtime optimization of the Python code, and using external object identification API are on the table. Object identification is no new subject and there are people with efficient algorithms and large object image library sitting in their database. Should be very easy to improve object identification performance.
Also, "wet" electrodes are very uncomfortable to keep them on for 2 days straight. Dry electrodes that gently sit on the face will definitely be better.
Log in or sign up for Devpost to join the conversation.