Search for Collaborative Platforms

Easily search for documents and reports shared on your chat server using keywords from the text in the documents.

Comment

Inspiration

Earlier I built a discord bot that can automatically detect sharing of sensitive data in chat and tokenize values but allow de-tokenization with proper authority. Sensitive data includes passwords, secrets, certificates, keys, URLs in chat, and user-uploaded attachments. I used Skyflow Vault to store all the required information for storing a message and then tokenizing the sensitive part of the data like the attachment CDN URL or the message. The bot also has integration with VirusTotal APIs to detect and report any malicious uploads. Check out more

To this bot, I wanted to add a feature where a user can search for specific text in the documents shared by the community on the server. The default search feature in these collaborative/communication platforms is very limited, it is the same case with discord. Having used discord many times before to work on course projects with my friends, I know the number of project reports, articles, research papers, etc that are shared. The amount of data shared in the case of proprietary business communication would be much larger, so having a robust search feature that includes information from documents would be very useful to the community.

What it does

The bot makes use of discord slash commands to communicate with the AWS serverless platform that makes use of multiple AWS services, the slash commands are:

  1. /sync [history (int)] - Scan the last 'history' number of messages and look for attachments, then upload, followed by text analysis and entity detection. Store and index the keyword: file data in a DB.

  2. /search [keyword (string)] - Search for 'keyword' or similar words in the DB, and then retrieve the documents and their related information.

How we built it

The bot interacts with the main lambda function through slash commands and interaction events. Whilst the bot responds to the command, it invokes helper functions(one for each command sync and search) asynchronously to interact with other AWS services and perform heavier workloads. This way we can decouple tasks and respond quickly to commands and avoid timeouts. The helper lambda functions have the following tasks:

  1. DataHandler: handles the /sync command workflow.

    • Inputs an SQS queue that is subscribed to an SNS topic using the output generated by the main lambda.
    • Retrieves attachment from discord CDN URL and then uploads the file to an S3 bucket.
    • Detects text in the document(S3 bucket) by using AWS Textract.
    • Perform entity detection on this text to detect any key phrases and entity information.
    • Store key phrases and the corresponding file-related data in YugaByteDB. Such that all files with the occurrence of keyword can be queried.

  2. SearchHandler: handles the /search command workflow.

    • Inputs a string, channel-related information from the output of the main function.
    • Queries YugaByte table to fetch related keywords
    • All file-related info is extracted and sent as a text message embed using the discord APIs. The link to the message and the attachment are provided in this message.

Yugabyte Integration

After building the bot earlier I planned on improving the performance of the search algorithm. Switching to a structured database from a NoSQL database improved the performance. YugabyteDB's interface made the onboarding process very easy and was easy to navigate.

What's next for AWS powered Search

Possible improvements include:

  1. Option to allow admin to auto-sync attachments/documents without any command. Scheduling the lambda to execute every 'x' minutes could do the trick.
  2. Replicate the bot for use in other popular collaboration platforms like Slack or Teams.
Share this project:

Updates

Submission history