Inspiration
The inspiration behind this project is to create a simple and interactive system that can understand handwritten text using Artificial Intelligence and Machine Learning. Handwritten characters can vary greatly from person to person, making them difficult for computers to recognize accurately. We wanted to build a system that can recognize handwritten digits automatically and demonstrate how deep learning can be used in real-world applications.
What it does
The project detects and recognizes handwritten digits drawn by a user on an interactive digital canvas.
The user can draw a digit using a mouse, finger, or stylus. The system processes the drawing, converts it into a suitable grayscale image, resizes it to 28×28 pixels, normalizes the pixel values, and sends it to the trained CNN model.
The model predicts the digit from 0 to 9 and displays the prediction along with its confidence score. The system also provides the probability distribution for all ten digits.
How we built it
We built the project using a complete Machine Learning pipeline.
- Dataset: MNIST handwritten digit dataset containing 60,000 training images and 10,000 test images.
- Preprocessing: Images are converted to grayscale, resized to 28×28 pixels, normalized between 0 and 1, and appropriately inverted when required.
- Model: A Convolutional Neural Network (CNN) is used to extract features and classify handwritten digits.
- Training: The model uses the Adam optimizer and cross-entropy loss with techniques such as Batch Normalization, Dropout, Early Stopping, and learning-rate reduction.
- Backend: Flask is used to create a REST API for real-time prediction.
- Frontend: HTML5 Canvas provides an interactive drawing interface.
- Testing: PyTest is used to test the model and API endpoints.
The trained model is saved as a Keras model and loaded by the Flask server for real-time inference.
Challenges we ran into
One of the main challenges was handling the difference between a user's handwritten drawing and the images used for training.
We faced challenges such as:
- Converting the canvas image into the correct format.
- Resizing the drawing to the required 28×28 pixel size.
- Converting images to grayscale.
- Normalizing pixel values correctly.
- Handling differences between black-on-white drawings and the MNIST image format.
- Preventing overfitting during CNN training.
- Creating reliable communication between the frontend and Flask backend.
- Handling invalid or malformed image payloads.
The preprocessing pipeline addresses these issues by decoding the Base64 image, converting it to grayscale, resizing it, normalizing it, and adjusting the image format before prediction.
Accomplishments that we're proud of
We are proud to have developed a complete end-to-end handwritten digit recognition system rather than only training a machine learning model.
Our major accomplishments include:
- Building a CNN-based handwritten digit recognition model.
- Creating an interactive drawing canvas.
- Achieving real-time prediction through a Flask REST API.
- Displaying prediction confidence for all ten digits.
- Implementing image preprocessing and normalization.
- Adding automated testing for the model and API.
- Maintaining a modular project structure with separate training, preprocessing, backend, frontend, and testing components.
The frontend supports mouse, touch, and stylus input and provides real-time confidence visualization.
What we learned
Through this project, we learned how to develop a complete Machine Learning application from data preparation to deployment.
We learned:
- How CNNs recognize visual patterns and handwritten digits.
- The importance of image preprocessing for accurate predictions.
- How data augmentation can improve model generalization.
- How Batch Normalization and Dropout help improve model training.
- How to build REST APIs using Flask.
- How frontend applications can communicate with Machine Learning models.
- How to perform automated testing using PyTest.
- The importance of separating different components of a production-oriented ML application.
This project helped us understand that building an AI application involves much more than training a model; data processing, backend integration, user experience, testing, and deployment are equally important.
What's next for Detect the Human Written Text
In the future, we plan to extend the project beyond handwritten digit recognition and make it capable of recognizing more complex handwritten text.
Possible improvements include:
- Recognizing handwritten alphabets.
- Recognizing complete handwritten words and sentences.
- Supporting multiple characters in a single image.
- Improving recognition accuracy for different handwriting styles.
- Adding multilingual handwritten text recognition.
- Deploying the application as a cloud-based service.
- Developing a mobile-friendly version.
- Adding more advanced deep learning models for complex handwriting recognition.
- Providing downloadable or editable recognized text.
Our ultimate goal is to transform the current handwritten digit recognition system into a complete handwritten text recognition platform that can convert human-written text into digital text in real time.
Log in or sign up for Devpost to join the conversation.