CodeCapture - VS Code Extension

Inspiration

While in recent times, the world has started moving towards pro-CS education, the fact is that buying computers is a distant dream for most students and educational institutions across the globe even today. In most developing countries, the ratio of CS students versus the number of computers available is highly skewed and most students are still learning programming via pen-and-paper. Therefore, we decided to develop an extension for Visual Studio Code, one of the most commonly used code editors in the world, to be able to capture code from scanned images of handwritten text and save it in a file of the right extension on the computer.

What it does

The CodeCapture extension scans the image selected by the user for handwritten C code and gives us the formatted code in a separate '.c' file in our directory to be checked and tested. This allows us to test and check the code developed by multiple users in a short period of time.

How we built it

Upon coming up with the idea, we divided the extension into three major sections: Picking an image, calling the Computer Vision Read API, and using the response received to write to an output file. For picking an image, we used the VS Code Extension API to have a native dialog asking the user to choose an image from local storage. To call the Azure Computer Vision Read API, axios was used to make both the POST and GET Http requests. The response received from the API was mapped to a string in a line by line format and was populated in a new file using the VS Code API.

Challenges we ran into

One challenge that we ran into was that we were not able to give formatted code in the output. As a temporary solution, we recommend the users to use a beautification extension for the time being.

Another challenge we faced was not being able to write to a file using node.js, we spent a few hours trying to work around the 'fs' npm package, after which we stumbled upon the VS Code API for the same and it made the task rather simple.

Accomplishments that we're proud of

This was our first ever experience creating a Visual Studio Code extension. Apart from that, not only did we create a functioning ML-enabled extension, but we also created a socially beneficial application that can help students and educators worldwide in improving the programming learning experience.

One major accomplishment that we achieved was using the Azure Computer Vision Read API for scanning the handwritten code. As we are all amateur hackers, creating an ML-enabled extension was definitely a Herculean task for us, which we successfully implemented.

What we learned

Our biggest learning was creating our first VS Code extension. Apart from that, we also learned how to use Computer Vision API (handwritten text detection - Read API) functionality using the powerful features available in Azure Cognitive Services. Also, since only one out of the 4 of us had prior full-stack development experience, it was also a great opportunity for the lot of us to learn JavaScript.

What's next for CodeCapture

We will be working on implementing in-built code formatting functionality to help the users beautify the output they need before it is saved in the file. We will also be working on adding multi-language support for languages such as Java, Python, etc. to make it more intuitive user experience.

Built With

• azure
• computer-vision
• javascript
• json
• vscode-api
• vscodeextension-api