Data Engineering case closed - Anya & Vasilina

Inspiration

We wanted to turn scattered investigation data into a reliable, defensible timeline instead of relying on manual Excel reconciliation.

What it does

Our pipeline combines badge access, device logs, and transaction data into one searchable employee activity timeline while preserving data integrity.

How we built it

We designed an AWS-based pipeline using batch ingestion, Kafka/MSK, DMS/CDC, S3 raw and curated storage, Spark normalization, Parquet, and Athena.

Challenges we ran into

The main challenges were inconsistent schemas, timestamp differences, late or duplicate events, and maintaining chain of custody.

Accomplishments that we're proud of

We created a clear end-to-end architecture that supports both investigation queries and evidence traceability.

What we learned

We learned how ingestion, normalization, storage, serving, and data integrity work together in a real data engineering pipeline.

What's next for Data Engineering case closed - Anya & Vasilina

Next, we would add stronger monitoring, automated contradiction detection, and more advanced integrity controls such as signed audit logs.

Built With

• excalidraw
• slides