Chess Lens

UI Screenshot
Original image
Detected chessboard projected into a 2D plane
Image of a bishop on a single square
Image of a pawn on a single square
Hedera Smart Contract Deployment

Inspiration

Elite chess tournaments utilize special boards to digitally capture moves and create a virtual representation of the live match. Our mission is to make this high-level experience accessible to every chess enthusiast. With ChessLens, all it takes is opening the app and pointing the camera at the chessboard to seamlessly bring this immersive experience to the average viewer.

What it does

ChessLens, a mobile app tailored for ardent chess enthusiasts, allows the viewing of Stockfish move lines and advantages during in-person games. The app not only presents a chronological list of past moves but also highlights the optimal Stockfish moves from the current position. ChessLens excels at providing clear digital representations of live, in-person chess games. The app updates automatically as long as you have your camera pointed at the game board. Furthermore, with Hedera integration, it supports saving statistics in the blockchain and gives users incentives to use the app with limited time NFT trophies.

How we built it

We used Flutter/Dart for the frontend. We wanted to create a fully responsive frontend that would be available on both iOS and Android. We integrated with the phone camera and . We receive data from the backend when the board updates and we use that to create a new board, add the played move to the movelist and display new advantage and stockfish lines for the new position.

We used Python, Flask, and Docker for the backend API. The backend calls a self-trained computer vision model and also gets the position evaluation from Stockfish 16. We also used Python's chess libraries In order to detect valid moves and to update the UI on every move change.

We used OpenCV and other libraries to detect a chessboard, and project it onto a 2D plane. We then sectioned the image into 64 individual images. We then trained a Convolutional Neural Network to classify the pieces on each of the squares using Keras.

We also integrated Hedera into our app, creating a coin to keep track of how many en passants have been seen by the app, and storing the data on the blockchain. Users who have enough coins get control over an NFT. The Hedera integration was done in Express.js, where we created an api using the Hedera SDK to allow for cross platform communication with our flutter app.

Challenges we ran into

Computer Vision was very difficult. We had to construct our own massive dataset of chessboard images to achieve good performance. We ended up taking around 1000 photos of chessboards. We had to construct a data pipeline to process the images into training and validation sets.

Accomplishments that we're proud of

-Creating a dataset of tens of thousands of chessboard images,

Writing a fully functioning frontend while learning Dart at the same time.
Learning blockchain integration with Hedera, along with SDK and Smart Contract syntax.
- Writing a deployed backend