The majority of our time is spent waiting on others. Waiting for email replies, phone calls, and slack responses keep us from being productive in ways that would stretch us. For instance, creating a expense report might take 1 minute, but waiting for approval from your boss, co-workers and accounting could take hours. Over the course of a day, we have many of these "tasks" that need to be juggled in our heads.
What AutoFlow does
Auto Flow automatically generates custom workflow on the fly that creates the sequence of actions desired. To do this we have a number of different components, "services" that can interface with each other, along with a custom parsing language that can be used to robustly create such flows from user input.
An example query the user could say to the application would be: "Message Keith, 'Hey, did we get flights approved? If Keith replies 'yes' on twilio then message Alex on slack "Flights are approved". This would create the appropriate sequence of events, such as waiting for a response from Keith, and then subsequently messaging Alex on slack.
How we built it
Each component of the service does a different task.
The speech processing component calls google's NLP API to convert speech into a text format which can be parsed.
The NLP parser module then breaks down the text. We developed a custom semi-rule based approach for this system. The commands issued by the user are instructional by design. The system first chunks the instruction into modules that contain action phrases, this is done using conditional keyword matching such as looking for if ... then ... else syntax. Next each action phrase is examined using a mixture of NLTK and rule-based parsing in order to determine the Verb e.g "message keith", Subject "Hello", and the relevant service "on Slack"
- The tree module then takes the relevant instructions and builds a parse tree in relevant JSON format which can be understood by the API engine.
- The API module then does a depth first parse of the JSON tree which contain instructions, targets, and messages. It then calls the Stdlib APIs which performs the relevant calls with the required messages. This module also runs a web hook server in order to listen to the responses.
Challenges we ran into
Reverse API calls. Design of NLP parser. Design of Query Language. Documentation of certain APIs
Accomplishments that we're proud of
We got the system to mostly work!
What we learned
Sleep. Who is sleep.
What's next for Auto Flow
We believe this project is just the starting point. With more data we could move from a more rule-based NLP to deep learning NLP which could make the system more robust. Also we would like to add the ability for users to define their own APIs and actions, potentially from a marketplace.
We envision a automation system similar to IFTTT but with user defined workflows on-the-fly! Maybe Stdlib.com could extend to such a functionality.