-
The launch screen
-
Character selection
-
Place your bets...
-
All bets are placed....
-
Ranking between rounds
-
A final celebration
Inspiration
For this hackerthon, SGX-FX set an interesting challenge for one of their competitions. The goal was to get creative and present trading in a more accessible way. We thought this was a really interesting concept where we could play around with some audio-visual presentation concepts and maybe introduce some game elements. We ended up with the idea of a game where players compete in the market my making investment choices on real world data provided by SGX-FX. In addition the data would be visualized as both a graph as well as played back in sound using generative audio. We also dident want to make just another game on desktop, we wanted something that was really engaging. So we opted to put everything on a Arduino Uno Q to make it portable and standalone and use gesture control so you wouldn't need any extra input devices.
What it does
We built a game that brings the fun to trading. Play against a friend and compete in a volatile market to see who wins bigger, and minimise your risk! All visualised like the floor on wall street, complete with the digital sounds of trading. Toss the keyboard and control everything with your hands using gesture tracking.
How we built it
We used godot as the foundation for the frontend in this project for its modularity and ease of use. For the audio generation to used supercollider, which is a procedural audio generation framework. For the pose estimation we used a combination of multiple frameworks, namely mediapipe for the video processing and tensorflow light for the pose estimation and gesture control using several publicly available models. The project was written in a combination of GDScript, python and slang, a language used for supercollider scripts.
Challenges we ran into
We ran into a couple of roadblocks, with the biggest one likely being the challenge of getting the gesture recognition to function on the Arduino Uno Q. The board is plenty powerful itself, being able to run the godot UI with absolute no problem. However, performing hand tracking and gesture recognition really pushed the limits of the board. We tried a number of different models with only minor success as the more commonly used f32 models were really pushing the memory bandwith of the board. Eventually we found success by using a version we quantized to int8 itself, and while this incurred a very minor performance loss is massively sped up the model's performance. Furthermore we used all available hardware threads by giving each thread one frame to work on. This increased latency a little but made the gestures significantly smoother and less jarring to use.
Another issue we faced was making the music generated by the synthesizer was generally a little monotone and bland. We wanted to have more unique tracks, so first we introduced multiple voices and 7 'moods' matching performance metrics like 'bullish', 'panic', 'calm', 'recovery', etc. In addition we added a independent storm voice which represents instability in the ticker. This wasent quite enough though, since we also needed to identify when to use which mood and what voices. In a lot of randomly chosen points for the ticker nothing much was happening, so we couldn't really identify what moods to use. Our solution was to pre-filter our data for sections that would be exciting, like a key turning point within them or a lot of variance. This also helped the mood algorithm, which has a much easier time identifying timestamps for such turning points and selecting the appropriate style.
Accomplishments that we're proud of
Were quite proud of the procedural sound generation used in this project. This was our first time using supercollider, and it seems like an awesome framework for this. The music turned out great and actually reflects the trend in the ticker really well.
Were also very happy with how well the gesture control works. Once we solved the performance problems it ran very smoothly and predictable. Given prior experiences with platforms like the Xbox kinetict, our expectations weren't particularly high. But pose estimation has come a long way from then and is much more accurate nowdays, and it was a good fit for this project.
What we learned
Bring extra equipment like keyboards and mice. We had a hard time programming the Uno Q because we weren't able to see any output or input anything. We ended up logging in via serial USB, then setting up tailscale and connecting remotely from our notebooks through SSH. Next time at least some small equipment would help us to not have to fiddle with that as much.
What's next for BNCL TradeOff
Were not planning on continuing this project directly, however we are really excited to use what we have learned from this in other small or large projects. Especially the sound design is something that could be carried over well into future projects, and its likely this hackerthon will see more of in the coming years. If you are interested in playing the game, its all open source on our github.
