PyCV: Future of accessible IDE

The future of coding is touchless. With our Interactive IDE, you can code, get performance analysis, suggestions, and use GUI based debugger. Forget your physical limitations and start coding.

Comment

Inspiration

The inspiration for this project came from the desire to make programming more accessible to everyone, regardless of their abilities. We wanted to create an IDE that would be more inclusive and accessible to people with disabilities.

What it does

PyCV is a touchless IDE that incorporates features of text-to-speech, speech recognition, hand sign language detection, and performance analysis. It also visualizes data structures and algorithms and makes it easier to debug.

How we built it

Building the base by integrating: ​ Framework: Flask, Frontend: HTML, CSS, JS, Backend: Python, Database: SQLite.​ Created a login system by saving user credentials in SQLite database and uses sessions by implanting cookies to improve user-experience.​ Created coding space with an IDE view​ using CSS. Running the code and displaying outputs, error, performance analysis like time and space complexity of the code.​ Extracting code from images and pdf to display onto the editor.​ Developing text to speech and speech recognition for accessibility.​ Build hand recognition using ASL datset and training a model using RandomForest algorithm ​ Using eye computer vision to navigate using embedded on-screen keyboard in the IDE​

Challenges we ran into

One of the biggest challenges the team faced was building a system that was robust and accurate. We also had to deal with the challenges of working with a variety of different technologies and datasets. Integrating Code Llama and language server python (LSP) for advanced intellisense and suggestion at the backend using Python​. Managing the full-stack flask framework development. Extraction of data from PDF, Images, and handwriting.​ Training and testing the Hand Recognition model.​ Developing eye navigation using Computer Vision were one of toughest parts of the projects.

Accomplishments that we're proud of

We are proud of the fact that we were able to create an almost touchless IDE that is both accessible and powerful. Though our machine learning models and code snippets for computer vision ran into many errors and are not super efficient but we are proud that we were able to use and learn from this Hackathon experience about different ML models.

What we learned

We learned a lot about building accessible and robust systems. We learnt about how to handle and manage full stack applications efficiently. This project helped us to explore the realms of Computer vision and Machine Learning.

What's next for PyCV: Future of accessible IDE

The team plans to continue developing PyCV and add new features, such as eye movement detection and code llama integration for performance analysis. We also plan to make the system more accessible to people with different disabilities.

Additional thoughts

I am very impressed with the work that I and the team has done on PyCV. It is a truly innovative project that has the potential to make a real difference in the lives of people with disabilities. I am excited to see what the future holds for PyCV and I am confident that I along with the team will continue to develop and improve it.

Share this project:

Updates