In the past it was difficult for us to discuss our AWS infrastructure in Slack. A lot of effort was placed in order to get the whole team to see the same information from AWS. Usually we copy pasted the results from the AWS CLI commands or attached screenshots of the AWS console. One day it hit us - this doesn’t have to be so hard. That’s how Opsidian was born.
What it does
Opsidian makes it easy for developers to query their AWS using natural language or AWS CLI commands. This allows teams to:
- Collaborate easier on AWS infrastructure without leaving Slack.
- Run more atomic commands than the AWS CLI - making it easier to discuss on Slack.
- Use natural language to let devs who are not AWS experts query and participate in tackling infrastructure challenges without remembering AWS CLI command syntax.
- Use multiple AWS profiles/accounts.
The bot captures user input queries and uses them to improve our NLP model. This way the bot will get much smarter over time and will gain a better understanding of what the users are asking about.
For security reasons we allow read-only commands to be performed. We currently have coverage of the following AWS services: Kinesis, SQS, ECS, EC2, API Gateway, RDS, S3, DynamoDB, ELB, EMR, ElastiCache and Lambda. You can see our full command and AWS service coverage at http://opsidian.ai/commands.
How we built it
We used AWS Lambda, AWS API gateway, AWS Kinesis and AWS Dynamodb. Opsidian is written in python, for deployments we use Gordon. We use wit.ai for natural language processing and command recognition.
Challenges we ran into
We've started developing our own natural language processing solution. We have build a text corpus from Amazon Web Services documentation and trained an LSI model to find commands that are most similar to what the user has typed in. Unfortunately, data from the documentation was not enough to train a good enough model - and that's because our commands are not identical to aws-cli commands (our commands are more atomic). We have created a wit.ai app and we have been teaching it how to recognise services, commands and resources. We are gathering input data and we use it to improve wit.ai's understanding. When we have a big enough data set we plan to implement our own NLP solution.
Accomplishments that we're proud of
We are proud of how easy it is to use Opisidian without the need to remember exact commands. We also think that our atomic commands approach makes a lot of sense and simplifies collaborating on AWS infrastructure.
What we learned
- We've learned that Serverless is a perfect solution for writing bots.
- Even though we are experienced developers, we have learned how comfortable it is to use natural language instead of remembering or checking the syntax of commands.
- We have learned (yet again) that machine learning and natural language processing are very difficult to use without having access to big data sets.
What's next for Opsidian
Our next step is to gather real world feedback on how the bot is used both in small teams as well as in larger organisations. As a first step we have approached Nordcloud, an AWS Premiere consulting partner to test and give us input on the bot by piloting it on different real world deployments with their managed services team. Initial feedback has been encouraging.
Based on early feedback we plan to extend our AWS service coverage to include Cloud Watch, Billing and IAM. We might also include integration to third party infra management tools (e.g. DataDogs). We plan to start a company to commercialise Opsidian and introduce a paid for version of the bot with extended functionality.