Inspiration
We were inspired by the ease yet excitement of guitar hero and its potential for informing new prosthetic designs, alongside a 1D CNN as a means for decoding neural data into a fulfilling experience.
What it does
The decoder is a 1D CNN that analyzes controller inputs from scifi data. This information is passed onto a guitar hero-style videogame. Both the decoder model and game design are intended to maximize bit rate (indicative of user experience and decoding fidelity).
How we built it
We tested various decoding model architectures (including TCN, RNN, Transformers, EEGNet, etc.) and decided to go with a 1D CNN given its bread-and-butter application to this multi-feature siganling against time. We also iterated on different designs for the game (dance dance revolution, aim trainer, music tiles) and decided that guitar hero gives not only the most optimal bit rate calculation (given its flexibility of input options and ability to pace itself toward user accuracy) but also its dynamicness that allows for 1) an exciting experience that remains IID and 2) a basis for potential prosthetic limb design.
Challenges we ran into
We struggled primarily on friction points between the sci-fi and our laptop: encoding data from the controller to the scifi, streaming the neural data, and deployment on-device (yet were able to suceed). The single challenge we weren't able to overcome was decoding on hard mode, but it was moreso a matter of time constraint and trying to complete deployment rather than model architecture. We believe with a bit more time, we could certainly do hard mode.
Accomplishments that we're proud of
We're really proud of our ability to create a simple yet holistic creation that serves people with disabilities who can't access dyanmic experiences such as video games (and more: see what's next)
What we learned
We learned that friction points between devices can pose real time constraints, but we also learned that sticking with a holistic perspective allowed us to push through and create a truly complete design that serves our bitrate-motivated goal.
What's next for NeuralShred
We believe that the dynamicness of NeuralShred not only informs future games we hope to make to interact with neurotechnology but even inform prosthetics (i.e. such as a person with no hand being able to have their extensor muscles act as a guitar-hero style system to operate a prosthetic hand).
Log in or sign up for Devpost to join the conversation.