Canary : Early model training failure detection site

This shows training by mobilenet 28709 images , and how this training is harmfull for my computer
these are the data of the images being trained by model
This image refers to the suggestion of epochs and training batch

Inspiration

Machine learning model training costs a lot of time and processing power. Programmers tend to understand that something went wrong with their model only after spending hours on training it, which costs GPU time and delays further experiments. We aimed to address this issue by developing a solution that would act as a "canary in a coal mine," warning about failed training.

What it does

Canary is an AI-based early warning system that helps monitor machine learning training. It monitors metrics associated with training like training loss, validation loss, accuracy, among others, to determine whether there are any signs of:

Overfitting Underfitting Loss divergence Learning plateau Training instability

Based on these metrics, the platform identifies and creates:

Early alerts Confidence/ risk scores Dashboards Suggestions to optimize training

This process happens during the training phase rather than after the completion of training.

How we built it

Canary has been developed using a cutting-edge full-stack technology stack.

Frontend: React.js Tailwind CSS Charts for visual representation of training metrics Backend: FastAPI Python REST APIs for receiving metrics data and prediction Artificial Intelligence & Analytics: Bayesian Change Point Detection for identifying any unusual changes in the learning pattern Analysis of training curves Training risk score engine for predicting failure probability Rule-based recommendation engine for providing recommendations

The backend analyzes the training logs that are uploaded and then evaluates the overall model’s health before displaying it on the dashboard.

Challenges we ran into

Some of the toughest problems involved:

The definition of "training failure," because each model had its own way of learning. Constructing meaningful risk scores that measured the health of the model, not just threshold numbers. Predicting reliably using Bayesian statistics in the presence of noise. Formulating helpful and understandable suggestions rather than just detecting problems. Putting together an interface to present difficult statistical information in a comprehensible form.

Accomplishments that we're proud of

Built a fully functional end-to-end AI Monitoring Platform. Implemented a successful identification of various failure training patterns in early stages. Created an interactive dashboard with actionable insights instead of just metrics. Applied the combination of statistics and ML engineering practices for creating an explainable monitoring tool. Developed a project which can save GPU computations and speed up the process of ML development.

What we learned

This project taught us:

Bayesian statistics for detecting changes. The significance of testing ML models, and not just training them. How to turn unrefined model measurements into useful information. Using FastAPI to build scalable APIs. Linking front-end visualization with back-end analytics tools. Creating interpretable AI systems, in addition to accurate ones.

What's next for Canary : Early model training failure detection site

Our future roadmap includes the following:

Direct integration with TensorFlow, PyTorch, and Hugging Face training pipelines. Live monitoring using APIs rather than uploading logs manually. Detection of new failure types like exploding/vanishing gradients and data drift. Hyperparameters tuning recommendations automatically. Diagnostics for teams consisting of dashboards, experiments tracking, and collaboration functionalities. Integration with other solutions like Weights & Biases and MLflow. Monitoring of GPU usage and other resources along with training diagnostics. Prediction of the likelihood of early termination of a training run to save computations. Development of Canary as an AI model observability solution to help create robust machine learning systems.