P&G Data Mart
Build a highly scalable data mart (Terabyte scale) for one of the CPG giant for various analytics use cases related to loyalty management programs (Dashboards and Statistical models).
Pull data from multiple diverse data sources, including
- The client’s own Loyalty Management Platform- Reward data which is Azure SQLServer database
- Customer Portal database shared as weekly extracts over SFTP
- App transaction data over SFTP
- Advertisement data from Amazon S3 bucket
- User activity from Google Analytics
and co-located in Amazon Redshift database.
Our 2-step approach for this ETL pipeline is:
- Create two schema on Amazon Redshift
- Staging – to store data pulled out from various sources.
- Datamart – to store the processed data from staging schema
- Create Spark Job
- Implement Spark connectors to pull data from all the above sources and load it into the staging schema
- Implement Spark job to transform the above data as per the business logic
- Load the data into data mart (Amazon Redshift)
Techniques Used:Spark MapReduce, Spark SQL, Amazon EMR, Amazon Redshift, Amazon Lambda, Scala
- Designed and optimized the data mart schema, a star schema, using Amazon Redshift.
- Setup the Big Data cluster on Amazon cloud – EMR cluster.
- Architected and developed the data pipeline using Big Data tech stack – Spark MapReduce, Spark SQL, Scala and AWS Lambda script.
- Validating the data mart against the source data sets and business rules.