DeepPavlov

python code
running results

Inspiration

In the NLP community, there are well-established standards to solve NLP problems like Stanford Core NLP, SpaCy and others. However, there is nothing for building dialogue systems. Our inspiration is to develop a framework for building dialogue systems that include all state-of-the-arts (SOTA) NLP components.

Specifically, our goal is to provide AI-application developers and researchers with:

set of pre-trained NLP models, pre-defined dialog system components (ML/DL/Rule-based) and pipeline templates;
a framework for implementing and testing their own dialog models;
tools for application integration with adjacent infrastructure (messengers, helpdesk software, etc.);
benchmarking environment for conversational models and uniform access to relevant datasets.

What it does

Nowadays, the framework contains a set of SOTA NLP models including:

Named-Entity Recognition (NER) (paper by us)
Open-Domain Question Answering (ODQA)
Text Classification
Neural Morphological Tagging
Automatic spelling correction
Sequence-To-Sequence Dialogue Models and others

More at demo

How I built it

I am one of the developers of DeepPavlov. The framework is being built by students and researchers of the Moscow Institute of Physics and Technology (MIPT). It's open source and free to use. DeepPavlov is built on top of TensorFlow machine learning frameworks. Other external libraries can be used to build basic components.

Challenges I ran into

Meanwhile, I am the only programmer who is responsible for migrating to TensorFlow 2.0. Specifically, I have to show that all our models can be migrated to TensorFlow 2.0. Broadly speaking, there are a lot of challenges on the way to develop an industry-level framework for building chatbots. The major challenge is to integrate different NLP components into a common architecture. In many cases, while developing new models for the framework it's difficult to reproduce results reported in papers.

During migration, I faced challenges that I didn't resolve. Our Named-entity-recognition (NER) model that was implemented by using biLTSM and CRF, cannot be fully migrated because CRF is yet to be added to tensorflow/addons issue.

Accomplishments that I'm proud of

Despite the fact that I've just started the migration to TensorFlow 2.0, I've successfully converted all Keras-based models. This covers all text classification models, particularly, sentiment analysis, intent recognition (based on snips, ag news, DSTC 2). In terms of the whole framework, we are going to release new Bert-based models including open-domain-question answering (ODQA), Named-entity recognition (NER), and others.

You can interact with the models either via command line or via Python code. The functionality of the text classification models I demonstrate on the attached video.

What I learned

During development, I've participated in building autoFAQ component that transfers incoming questions into predefined answers by applying machine learning models.

While working on the migration, I've realized that TensorFlow is a very easy-to-use framework, however, if you really want to benefit from using TensorFlow you have to follow its philosophy. Hopefully, I will learn a lot of common practices and important details at TensorFlow Connect. This knowledge will help us to adapt our framework to the TensorFlow philosophy.

What's next for DeepPavlov

Our main intention is to develop an industry standard for building dialogue systems. There are a lot of things that are left to do, including text summarization, coference resolution, parsing, and others While working on migration, I plan to integrate tf.data into DeepPavlov instead of using custom data readers and refactor the existing models to follow the TensorFlow 2.0 style.