The majority of our time is spent waiting on others. Waiting for email replies, phone calls, and slack responses keep us from being productive in ways that would stretch us. For instance, creating a expense report might take 1 minute, but waiting for approval from your boss, co-workers and accounting could take hours. Over the course of a day, we have many of these "tasks" that need to be juggled in our heads.

What AutoFlow does

Auto Flow automatically generates custom workflow on the fly that creates the sequence of actions desired. To do this we have a number of different components, "services" that can interface with each other, along with a custom parsing language that can be used to robustly create such flows from user input.

An example query the user could say to the application would be: "Message Keith, 'Hey, did we get flights approved? If Keith replies 'yes' on twilio then message Alex on slack "Flights are approved". This would create the appropriate sequence of events, such as waiting for a response from Keith, and then subsequently messaging Alex on slack.

Alt text

How we built it

Each component of the service does a different task.

  1. The speech processing component calls google's NLP API to convert speech into a text format which can be parsed.

  2. The NLP parser module then breaks down the text. We developed a custom semi-rule based approach for this system. The commands issued by the user are instructional by design. The system first chunks the instruction into modules that contain action phrases, this is done using conditional keyword matching such as looking for if ... then ... else syntax. Next each action phrase is examined using a mixture of NLTK and rule-based parsing in order to determine the Verb e.g "message keith", Subject "Hello", and the relevant service "on Slack"

Alt text

  1. The tree module then takes the relevant instructions and builds a parse tree in relevant JSON format which can be understood by the API engine.

Alt text

  1. The API module then does a depth first parse of the JSON tree which contain instructions, targets, and messages. It then calls the Stdlib APIs which performs the relevant calls with the required messages. This module also runs a web hook server in order to listen to the responses.

Alt text

Challenges we ran into

Reverse API calls. Design of NLP parser. Design of Query Language. Documentation of certain APIs

Accomplishments that we're proud of

We got the system to mostly work!

What we learned

We learned how to communicate between JavaScript and python applications. How to call APIs the right way and developing speech-to-text. That you have to say things really slowly in a quiet room for google speech to parse correctly. (For now)

Sleep. Who is sleep.

What's next for Auto Flow

We believe this project is just the starting point. With more data we could move from a more rule-based NLP to deep learning NLP which could make the system more robust. Also we would like to add the ability for users to define their own APIs and actions, potentially from a marketplace.

We envision a automation system similar to IFTTT but with user defined workflows on-the-fly! Maybe could extend to such a functionality.

Share this project: