Challenges
A US-based Fortune 500 mortgage lender wanted to move their Netezza-based enterprise data warehouse to a modern data lake architecture to reduce licensing and maintenance costs, meet data consumption and business SLAs, and achieve faster time-to-market. They were looking to rearchitect and migrate workloads to an AWS-based enterprise data lake with the following capabilities:
- Scalable services and customized applications in line with current and future business needs
- Ability to move reporting/analytical consumption loads to a scalable AWS vending layer
- Robust disaster recovery
- On-demand cloud-native ETL
The client wanted to leverage cloud-native ‘as-a-service’ offerings to reduce management complexities and switch to a more economical pay-as-you-go pricing model.
Auto-migrated ~1 TB historical data and up to 4 GB incremental data per day
Solution
To meet the client’s business needs, the Impetus team designed an enterprise-scale hybrid cloud solution and leveraged Impetus’ Workload Transformation Solution to automatically translate Netezza logic to the target environment.
Highlights:
- Identified expensive and poorly performing data processing entities through ML-based assessment
- Migrated historical data and developed a reliable mechanism for incremental updates to S3
- Migrated 100+ NZSQL files with approximately 20 queries and 5 Informatica ETL workflows
- Created structured data lake zones on S3 (landing, preparation, and insights) for quality check and data augmentation for downstream consumers
- Automatically translated consumption queries (i.e., SQL, stored procedures, Shell, and Scheduler) into Spark SQL/Hive QL
- Ingested data from S3 to Redshift for reporting and analytics
- Performed SCD Type 1 and Type 2 for slowly changing dimensions
- Replicated the Netezza snapshotting process on AWS
- Used Autosys and Oozie as on-premises/cloud orchestrators
- Used AWS Lambda as an event-driven notification engine
- Used Amazon EMR to process vast amounts of data across scalable Amazon EC2 instances
- Used Amazon RDS to store metadata and workflow information
- Evaluated and performed tool benchmarking for consumption queries on the cloud
- Provided on-site support and services to ensure a seamless migration
50% time and cost savings compared to manual migration
Impact
Our comprehensive solution helped the client sunset Netezza and realize the following benefits:
- Lowered operational expenses by freeing up capacity in Netezza
- 50% time and cost savings compared to manual migration
- Improved data availability across the enterprise, enabling innovation
