AI Supported Multi-Modal Integration with AWS infra

Solvent Intelligent Workspace integration into AWS for automation.

Comment

Inspiration

At Codesolvent we believe augmenting human abilities with AI capabilities via well thought out and deeply integrated workspace environments is one of the best ways to leverage current capabilities of AI and deliver real value.

And by well "thought out" we mean "not chatbots".

https://codesolvent.com/botworx/intelligent-workspace/

What it does

This project showcases leveraging AI to interact with the AWS infrastructure paired with an Intelligent Workspace environment that puts humans at the center of AI supported work.

Specifically, it allows a user to:

  • Ask personal assistant to Upload a PDF file to AWS.
  • The personal assistant leverages a Solvent aws-cli-agent that connects to AWS
  • Solvent aws-cli-agent uploads pdf to AWS
  • AWS extracts text from pdf and returns result.
  • Solvent aws-cli-agent Saves the result locally.
  • Personal assistant invokes human-in-the-loop tool to allow user to update the text.
  • User updates the resulting text
  • User tells personal assistant to upload the updated text back to AWS.
  • Personal assistant invokes Solvent aws-cli-agent to perform upload
  • User tells personal assistant to send the same text via email.
  • Personal assistant sends the email with attachment

How we built it

  • Create an AWS state machine that uses AWS Lambda to extract PDF using the AWS Textract service.
  • Create an AWS bucket to hold the uploaded PDF.
  • Install AWS CLI on Solvent instance.
  • Create an AI session within Solvent Intelligent Workspace to use Bedrock for LLM service.
  • Create a function calling tool with Solvent to interact with the CLI.
  • Create an agent within Solvent that uses the AWS CLI tool via function calling.
  • Connect the Solvent AWS CLI agent to a primary personal agent to act as an assistant.

Challenges we ran into

Learning the necessary AWS permission tricks required to make the services interact with one another :)

Accomplishments that we're proud of

End-To-End multi-modal integration that reminds you of what you might have seen in a Sci-fi movie, except this one is real work being done via a real AI system.

What we learned

Beyond autonomous agents, there is huge potential for AI to augment human abilities if we build the right workspace environments.

What's next for AI Supported Multi-Modal Integration with AWS infra

More integration examples into other systems and leveraging more of the capabilities of the Solvent platform to show how it can add value to other platforms such as AWS.

This demo showcases for instance possibilities for AWS customers to use AWS services with ease and without being experts of the AWS Console.

Contact me (ekemokai at codesolvent.com) if you want live access to instance running this project

Built With

Share this project:

Updates

Submission history