NeuralBridge

problem
old approach: ONNX to TensorFlow
old approach: TensorFlow to ONNX
old approach: ONNX to PyTorch
old approach: PyTorch to ONNX
market

Challenge 🏆

🏆 NeuralBridge is a brand new idea made for the #ai track in GitLab Innovation Pitch Competition.

What is NeuralBridge? 🤖

🦾 Easy-to-use, one-liner AI model conversion tool between AI frameworks to ensure faster model development and portability with Open Neural Network Exchange.

Inspiration 💡

📐 We are two freelance deep learning developers. Constantly working on different projects from time to time gives us experience in various fields of AI development. For example we had projects with NASA, the US Department of Veterans Affairs and Seagate. We use PyTorch, TensorFlow, Keras and OpenVINO frameworks daily to fit the needs of the actual Entruster.

🏗 Maintaining this knowledge is hard and consumes a lot of time and energy. We know a lot of developers who are familiar with only one of the mentioned major frameworks. During our work, we can build the models based on their needs. However, this kind of attitude is the exception and not the thumb rule.

🤝 That’s why we wanted to create a solution that helps developers, who know one framework only, to be able to load and work with AI models in another framework easily without a lot of extra knowledge.

Real world problem 📈

3️⃣ The topic of the problem consists of three major parts:

🤷 The root of the problems is that developers usually are familiar with one framework. This is enough to work as an employee, since most of the big companies build in one framework only. These frameworks change constantly, so keeping knowledge up-to-date consumes a lot of time and energy. According to Stackoverflow’s Survey [1], 8.41% of professional developers use Tensorflow, while 7.89% of developers use PyTorch. There are more than 27 million developers worldwide [2], but only 4.553 developers have a TensorFlow Developer certification [3].

🖋 Famous models are written in one framework only. On HuggingFace 93% of models are PyTorch exclusive [4] and 4% of models are TensorFlow exclusive [5]. Only 3% of models are available for both communities without conversion [6]. Popularity can be measured in stars and forks. On GitLab, Pytorch exclusive repositories have more start and forks than TensorFlow exclusive repositories [7][8].

👔 ONNX is an existing solution to make model conversion available for developers. The problem with that is the developer should convert the original model manually to the desired framework. This means they have to be familiar with both the source and the target frameworks. Furthermore, maintaining PyTorch and Tensorflow frameworks up-to-date in the same environment is challenging due to the different requirements of these packages. An update command in the environment regularly meets with unsolvable conflicts. Using ONNX in the code interferes with the coding standards like PEP 8 or PEP 20 and degrades the readability of the code. We know, styling is usually the least important cornerstone for most developers during coding, but a lot of companies expect to write clean and well structured code. As Oscar Wilde said, “A well-tied tie is the first serious step in life.”

What it does ❓

4️⃣ NeuralBridge offers 4 main features at the moment:

Runtime conversion ⏱

⏱ With the use of the function convert_model() user can convert models from one framework into another runtime.

Saving model or model weights and biases 💾

💾 During model conversion models can be saved as well by setting the appropriate parameters. Models can be saved in the format of the output framework or as an Open Neural Network Exchange model.

Exporting ONNX 🚢

🚢 Since ONNX conversion is an intermediate stage of the main workflow of NeuralBridge, exporting the model in ONNX is available as well. There is an intuitive way of giving the name for the model from the class name but the user can set another name too.

Loading existing model into different framework from file ⏳

⏳ There are separated functions to load models into the desired framework. These functions are framework agnostic, that means they can load models with or without the need of conversion. This is a useful and convenient feature to let users work with any model from any background in the framework they are the most familiar with.

How it works 🔬

🦠 The most easy scenario is the following:

model_in_other_framework = convert_model(model_in_one_framework)

👀 This one line of code can automatically recognize the given model and converts it from one framework to the other. If a PyTorch model is given, the output will be TensorFlow and vice-versa.

✂ There are several options to further customize the conversion process but the basic use case is so easy.

🌌 Under the hood there is a massively engineered automation process to keep everything that simple. First NeuralBridge figures out what is the given model’s input type and concludes what should be the expected output type. After that the model gets converted into Open Neural Network Exchange standard. By default ONNX output is not saved, but the user can turn this feature on as well. NeuralBridge can also save output model or model weights. It can figure out a name to save based on the class name in case the user doesn’t give a different name.

USP 📌

One-line solution 🐣

🐣 The greatest advantage of NeuralBridge is it fits well into the existing code and AI development workflow.

Conforms with existing standards 🕊

🕊 NeuralBridge fits the coding standards. It is enough to change the model loading function with our wrapper function.

Easier to use than ONNX 🦉

🦉 Implementing ONNX into a code consumes a lot of extra knowledge and requires new lines into the code. Our wrapper function needs one line code only. That length is equivalent to the standard loading function. There is no need to create a new mindset to remember converting the model from one framework to another and vice versa.

Supporting runtime model conversion 🦅

🦅 There is no need to preconvert the models. Our solution converts the models on the fly.

Accepts ONNX and raw models 🦜

🦜 If a developer already has a converted ONNX model, it is loadable into the workflow on the fly like PyTorch or Tensorflow models. This provides more flexibility to switch to NeuralBridge easily.

Effortless model switching 🦚

🦚 Developers can try new models easily without switching environments and frameworks, since NeuralBridge converts models runtime. It is possible to load a pretrained model to train further and save the new model into a new framework. This method provides flexibility to build, train or use AI models.

How we built it 🔨

🛬 We built NeuralBridge with a strict top-down approach. We searched for the most painful use cases of the cross-framework model use and we ordered them in row by importance and the level of difficulty. Our conclusion was that model conversion itself is the most important case but there are other significant problems as well. Therefore we implemented model conversion during the hackathon and we made a long roadmap to develop NeuralBridge much much further in the future.

🐍 Since Python is the most common programming language for AI development it was obvious to use it as a coding language.

🧪 We researched all the requirements and practice of the model conversion and we made a detailed list of requirements and feature-needs for ourselves. After that we wrote the code and made the detailed documentation as well.

🎯 Since installing PyTorch, TensorFlow and ONNX into the same Python environment is almost that exhausting than model conversion, we decided to offer NeuralBridge as a Python module to simplify this process for everyone.

Challenges we ran into 💪

2️⃣ There were two main challenges:

🔀 The first one is that the philosophies of PyTorch and TensorFlow are very different. Therefore providing a widely usable and yet automated solution requires deep knowledge and understanding of the differences and their consequences.

🔄 The second one is that converting PyTorch to ONNX requires dummy input in case the model is not yet converted into script model type. To fulfill this need at the moment the shape and datatype of the model are parameters in case of a conversion from PyTorch. In the future we can fully skip this because we plan to ‘figure out” the sizes and data type automatically.

Accomplishments that we're proud of 😎

😎 This hackathon inspired us to make something very useful and to give a solution for the AI developer community that solves an everyday problem which is very painful for almost all the developers. We had to investigate the question of cross-framework model conversion very deeply to implement our idea. We had to try very different solutions out to find the one that suits the most. It was amazing to see how significant the question we found and how challenging its solution is. And it was very high to see we can solve it.

😎 We are new to GitLab and new to its CI/CD conception. We find it very useful by the way since it lets us build very complex deployment processes quite easily. This finding led us to the conclusion that we have to build a CI/CD solution from our idea as well. We will fulfill this task right after the hackathon. NeuralBridge as a CI/CD tool lets the users build variants of their model for different frameworks and different precision very easily.

What we learned 📘

📗 We learned how to use GitLab efficiently. GitLab was totally new to us. Of course, we daily use git services during AI development, but GitLab’s services are great. Since we are deep learning developers, we tried to stick to this topic and aim our project for the #ai track. However, the CI/CD catalog made a lot of impression for us. That’s why we plan to offer NeuralBridge as a CI/CD service on the GitLab platform.

📕 We faced problems during the development. Creating a multi framework solution always consists of the possibility of failures and errors. Each framework has its own requirements that makes mixing those frameworks into one environment really hard. Developers usually use multiple virtual environments or containerized services. During the hackathon we learned how to put different frameworks into one environment without conflicts.

📙 During the hackathon we got a deep understanding about ONNX. Using ONNX is not easy, but it is much simpler than building a solution based on it. We had to learn what ONNX does under the hood with the AI models to be able to create a working solution.

📔 We searched for statistics to prove the real market need for our solution. The results were shocking for us. We already experienced the problem that most of the developers are familiar with one framework only and the most famous AI models are available in only one framework. However, we underestimated the size of the problem. The revelation of statistics made our eyes open. Now we know, this is a project that we must develop further to help the developer community. We hope, in the future, other multi-framework developers will help us to make this bubble bigger.

What's next for NeuralBridge ⏱

6️⃣ We plan 6 improvements:

CI/CD 📓

📓 The next step after this hackathon is to offer NeuralBridge as a CI/CD service on the GitLab platform to add the possibility of automated model conversion for the GitLab users. The automated conversion helps AI developers to offer their solutions for a wider market since this way the use of their models is not limited to the development framework only. This is good for the developers and good for the end-users at the same time.

Model validation 📏

📏 Validating the converted model against the original one is a common and needed coding practice in cross-framework situations. However, automated conversion ensures valid results, the need to offer automated and easy-to-use verification is understandable. Therefore we plan to add simple-to-use automated validation functionality as well.

Model difference ↔

↔ Having different model versions is very common in the modern Artificial Intelligence development processes. Since NeuralBridge requires a very deep understanding of the real model structure at the automation level, it is obvious to use this out and offer more. Model differences can discover and show differences between different model versions. It can handle model structure and model weights or biases as well. Therefore the progress of the layers can get measured separately. Aside from the coding skills making this functionality perfect and very easy-to-use requires data visualization as well.

Model comparison ⚖

⚖ Aside from cross framework conversion there is also data type conversion on the roadmap of NeuralBridge. The use of 16 bit floating point or integer precision instead of double precision requires significantly less resources but in return it means some accuracy loss. Real-world business decisions often cost a lot of time since detailed accuracy comparison is required between the different model data types. To spare development time and to offer the possibility of faster decision making, we plan to add fully automated accuracy comparison between model data types for NeuralBridge.

Input testing and conversion 🧰

🧰 Since NeuralBridge offers both model framework and data type conversions in long-term, implementing automated input validation and conversion sounds also a good idea. This functionality is very useful from the end-users as well, since AI users usually have less knowledge about AI in general or about the used model as its developers. We plan to offer a function that gives suggestions for correcting input issues instead of just throwing detailed error messages.

More frameworks 🎨

🎨 At the moment **NeuralBridge”” can load and export models from and to 3 frameworks; PyTorch, TensorFlow and ONNX. We plan to add all the used frameworks to really forget the exhausting problem of model conversion and to offer all solutions to the users of all frameworks. Achieving this requires a lot of time, a lot of development and a lot of self-development for us as well. In case our project is worthy enough to receive additional support from the organizer we could reach this goal.

Building a community 🧑‍🤝‍🧑

🧑‍🤝‍🧑 In the future we want to build a developer community where people who are familiar with more frameworks help us to build this project. We hope that a strong community can help newbies to go ahead faster. Developers usually spend a lot of time on experimenting with new model types, new layers, different initializations and hyperparameter tuning. For a newbie it is really frustrating that their desired pretrain model is available in a different framework. NeuralBridge can help to tear down this barrier.

Fun facts 🌳

🌳 Since NeuralBridge is a one-liner solution, using it requires fewer lines of code and characters than ONNX. Fewer characters, less typing. Less typing, less noise pollution.

🌳 Furthermore, fewer code makes faster interpretation. Less time spending for the compilation means less energy to run the code.

🌳 As you see, NeuralBridge is might be the greenest project in the GitLab Innovation Pitch Competition.

That's one small step for NeuralBridge… 🦶

🎠 NeuralBridge is a really cool name. We had a lot of bad ideas for this project like: BrainBot, MindModel, MindMonument, TinyTick, TorchFlow or CuteConverter. With the concept of ModelMultiFrameworkConverterSolution we might win the prize for the longest name of the competition. The second concept was neuralink, but it is already occupied and unfortunately we did not make a successful deal with Elon. 😊 So we wanted a name that is easy to remember, trendy and to be melodic.

🚑 We tried to do our best during the hackathon to make a solution that solves a real problem. Furthermore, we delivered a cool name for the project. Please support us and our project!