Starting a Deeplodocus session
Full Architecutre of Deeplodocus
Plot of the history
Summary of a model in Deeplodocus
Training a network using Deeplodocus
Simplified architecture of Deeplodocus
Deeplodocus was inspired from CFD software which are mostly config files based. They offer default parameters but still force the user to understand the details of the tasks they want to achieve. Deeplodocus is also inspired from web framework which offer flexible but well defined project structures thanks to architectural patterns such as the Model-View-Controller (MVC) pattern. With these ideas in mind, we created Deeplodocus, a config files based Deep Learning platform bringing methodology and structure to deep learning projects.
What is Deeplodocus
In one line, Deeplodocus is a modular, flexible and accessible environment for the structured and rapid development of deep learning projects.
The challenge of training, testing and deploying artificial neural networks involves the completion and integration of numerous complex sub-routines. Efficient data loading and augmentation, model design and optimisation procedures are all non-trivial tasks that require structured management throughout any deep learning project. However, approaching each one of these tasks from scratch is not always necessary—especially given the current range of machine learning libraries—and instead the development of only select modules is often be more efficient. Therefore, Deeplodocus is a flexible environment that enables users to seamlessly integrate any number of existing and custom modules to rapidly build powerful deep learning projects.
Not just another abstract API, Deeplodocus is a new approach to structuring deep learning projects. It allows the level of abstraction in each domain of a project to be tailored to suit the user requirements. Users can quickly compile new deep learning pipelines solely from pre-implemented modules without writing a single line of code, or investigate novel cutting-edge techniques by taking control of every low-level operation.
Built on PyTorch, Deeplodocus offers comprehensive control of high-level configurations and parameters, whilst maintaining maximum flexibility through modularity to accelerate the rapid-prototyping of deep learning techniques.
How to install Deeplodocus
Deeplodocus is available on PIP. To install it, simply enter:
pip install deeplodocus
What we added for the Hackathon
Deeplodocus was already functional before the hackathon, however we added several new features which improved the stability and extended the capabilities of the platform. Moreover, most the new features are directly PyTorch focused. Among them we have :
- New data management system
- New output transforms system (PyTorch GPU transforms)
- New admin command to automatically create transform files
- New documentation
The new data management system is brand new, more flexible and now fully compatible with PyTorch datasets. Deeplodocus automatically wraps any PyTorch dataset using a SourceWrapper assuring the compatibility with the Deeplodocus system. Additionally, users can now create their own custom data Source classes and load them into the Deeplodocus pipeline.
Deeplodocus now handles Output Transforms. It offers more flexibility for visualizing the outputs or computing additional results such as a confusion matrix at the very end. Output transforms are performed by batch using the GPU and the PyTorch transformation operations.
New admin commands
Deeplodocus now accepts new admin command from the terminal. You can now create transform files using the commands:
The new documentation is available at http://deeplodocus.org
Challenges we ran into
- Making everything compatible with the previous version of Deeplodocus
- Fixing bugs related to the new PyTorch version (1.2.0)
- Making tests
What we learned
- Inner structure of PyTorch Datasets
- Inner structure of GPU transforms
- Using entry points in Python
- Using git and working on different branches
- Using readthedocs.org and mkdocs
What's next for Deeplodocus
- Multiple models (such as GAN)
- Making it compatible with Reinforcement Learning environment
- Distributed Learning
- User Interface in the browser
- Multiple languages support (with English as default)
- More flexible transforms
- Create a PyTorch model from a YAML file
Start a Deeplodocus project
To create a new project, simply enter :
deeplodocus new-project <project_name>
Then you can modify the config files and start your session using :