In the NLP community, there are well-established standards to solve NLP problems like Stanford Core NLP, SpaCy and others. However, there is nothing for building dialogue systems. Our inspiration is to develop a framework for building dialogue systems that include all state-of-the-arts (SOTA) NLP components.
Specifically, our goal is to provide AI-application developers and researchers with:
- set of pre-trained NLP models, pre-defined dialog system components (ML/DL/Rule-based) and pipeline templates;
- a framework for implementing and testing their own dialog models;
- tools for application integration with adjacent infrastructure (messengers, helpdesk software, etc.);
- benchmarking environment for conversational models and uniform access to relevant datasets.
What it does
Nowadays, the framework contains a set of SOTA NLP models including:
- Named-Entity Recognition (NER) (paper by us)
- Open-Domain Question Answering (ODQA)
- Text Classification
- Neural Morphological Tagging
- Automatic spelling correction
- Sequence-To-Sequence Dialogue Models and others
More at demo
How I built it
I am one of the developers of DeepPavlov. The framework is being built by students and researchers of the Moscow Institute of Physics and Technology (MIPT). It's open source and free to use. DeepPavlov is built on top of TensorFlow machine learning frameworks. Other external libraries can be used to build basic components.
Challenges I ran into
Meanwhile, I am the only programmer who is responsible for migrating to TensorFlow 2.0. Specifically, I have to show that all our models can be migrated to TensorFlow 2.0. Broadly speaking, there are a lot of challenges on the way to develop an industry-level framework for building chatbots. The major challenge is to integrate different NLP components into a common architecture. In many cases, while developing new models for the framework it's difficult to reproduce results reported in papers.
During migration, I faced challenges that I didn't resolve. Our Named-entity-recognition (NER) model that was implemented by using biLTSM and CRF, cannot be fully migrated because CRF is yet to be added to tensorflow/addons issue.
Accomplishments that I'm proud of
Despite the fact that I've just started the migration to TensorFlow 2.0, I've successfully converted all Keras-based models. This covers all text classification models, particularly, sentiment analysis, intent recognition (based on snips, ag news, DSTC 2). In terms of the whole framework, we are going to release new Bert-based models including open-domain-question answering (ODQA), Named-entity recognition (NER), and others.
You can interact with the models either via command line or via Python code. The functionality of the text classification models I demonstrate on the attached video.
What I learned
During development, I've participated in building autoFAQ component that transfers incoming questions into predefined answers by applying machine learning models.
While working on the migration, I've realized that TensorFlow is a very easy-to-use framework, however, if you really want to benefit from using TensorFlow you have to follow its philosophy. Hopefully, I will learn a lot of common practices and important details at TensorFlow Connect. This knowledge will help us to adapt our framework to the TensorFlow philosophy.
What's next for DeepPavlov
Our main intention is to develop an industry standard for building dialogue systems. There are a lot of things that are left to do, including text summarization, coference resolution, parsing, and others While working on migration, I plan to integrate tf.data into DeepPavlov instead of using custom data readers and refactor the existing models to follow the TensorFlow 2.0 style.