What it does
The project extracts, transforms, and loads (ETL) YouTube data from Kaggle, processes it, cleans anomalies, and then offers analytical visualizations. This entire workflow, from data ingestion to visualization, is orchestrated on AWS.
How I built it
Initialization:
- Started with setting up an AWS account, ensuring to follow the best practices for security.
- Created an IAM account with administrative privileges for secure and role-based access.
Data Ingestion & Formation:
- Uploaded the YouTube dataset from Kaggle into an S3 bucket, essentially kickstarting the formation of a data lake.
- In this data lake, I managed different states of data - raw, cleaned, and processed, ensuring a robust data lifecycle.
Data Organization with AWS Glue:
- Deployed AWS Glue crawlers to catalog the YouTube dataset, which helped in organizing and accessing the data efficiently.
- Made use of the Glue Studio to design and manage the ETL jobs, streamlining data transformations.
Transformation & Cleaning:
- Addressed the challenge of the YouTube dataset being in JSON by converting it to Apache Parquet, which is more storage-efficient and query-friendly.
- Faced with JSON inconsistencies, I harnessed the power of AWS Lambda combined with Glue DataBrew and AWS Data Wrangler to refine and preprocess the data.
Analytics & Visualization:
- For data querying, I used Amazon Athena, which allowed SQL-like querying on the processed data.
- Final insights, patterns, and trends were beautifully visualized in an easy-to-grasp manner using Amazon QuickSight.
Challenges I ran into
One significant challenge was handling the JSON errors (JSON Serde) in the initial dataset. Ensuring secure and optimized access to AWS resources, while navigating the breadth of services, was a learning curve.
Accomplishments that I'm proud of
- Successfully constructed a scalable ETL pipeline on AWS from scratch.
- Achieved a seamless transition from raw data to insightful visualizations, capturing the essence of YouTube's vast dataset.
What I learned
- In-depth knowledge of AWS's data services, particularly AWS Glue, Lambda, Athena, and QuickSight.
- Best practices for cloud-based data processing and the significance of data lake formation.
- The intricacies of IAM roles, security policies, and the power of Lambda layers in processing tasks. ## What's next for AWS ETL Pipeline for YouTube Data Analysis Future enhancements could include:
- Integrating machine learning models to predict YouTube trends. - Implementing real-time data processing and analysis.
Built With
- amazon-web-services
- athena
- data
- github
- glue
- json
- kaggle
- lambda
- pandas
- python
- quicksight
- s3