email archiving compliance platform

The Problem

Compliance teams in the highly regulated financial industry struggle to keep up with the amount of regulatory compliances, one of them being recordkeeping rules MiFID II Article 16 (EU/UK) and SEC Rule 17a-4 (US) where financial institutions such as investment services, banks, trading-related companies need to maintain comprehensive records of any financial transacitons, communications and other business activities for a period of time.

And just communications data alone could amount to nearly 50% of the data that needs to be stored compliantly which are usually scattered across all platforms and is a massive headache to keep track of and maintain. On top of storing the data, compliance teams need to make sure those data are easily accessible (search) when needed for a legal case or investigation and to also make sure that the data is tamper-proof.

The Solution

This secure archiving solution provides compliance teams with a robust platform for archiving, storing, indexing and searching the data to stay compliant with the regulatory rules of MiFID II Article 16 (EU/UK) and SEC Rule 17a-4 (US). The platform centralizes the data into one single source of truth (AWS Aurora Database) and provide a convenient dashboard and UI to manage the data alongside the retention policy and also keep the logs of every single action done within the platform so that they can audit it easily.

The search functionality uses the full-text search feature provided by AWS Aurora Postgres that could help streaming the workflows of searching through needle in the haystack for any use cases such as eDiscovery or investigation from days to just minutes.

So: a regulated archive of record — searchable, tamper-evident, and built on a database designed for exactly this.

Database Choice

I chose AWS Aurora Postgres for the database because of several reasons:

1. Supports relational schema as needed for the use case with the relations between messages, cases and attachments entities.
2. Providers pgcrypto to enable tamper-proof evidence and also WORM immutability through Postgres triggers.
3. GIN index for full-text search + ts_headline capability to enable a seamless search experience for the user with text highlighting
4. Aurora decouples storage from compute which mean the cost to store these data which 70% will be cold data will be very cheap for the enterprises and they are able to make use of this platform effectively with minimum cost.

What's next

• Improve knowledge search with Graph-RAG so that the system is able to know which thread is this message referring to.
• Add another source of communication messages, maybe even multimodal such as calls, video meetings, MSTeams/Slack chat with images.
• Add semantic search capability with pgvector

Built With

• amazon-web-services
• aurora
• nextjs
• vercel