Case Study

Automated migration from Netezza to AWS for a Fortune 500 mortgage lender

Lowered operational cost and improved data visibility across the enterprise


Challenge

A US-based Fortune 500 mortgage lender wanted to move their Netezza-based enterprise data warehouse to a modern data lake architecture to reduce licensing and maintenance costs, meet data consumption and business SLAs, and achieve faster time-to-market. They were looking to rearchitect and migrate workloads to an AWS-based enterprise data lake with the following capabilities:

  • Scalable services and customized applications in line with current and future business needs
  • Ability to move reporting/analytical consumption loads to a scalable AWS vending layer
  • Robust disaster recovery
  • On-demand cloud-native ETL

The client wanted to leverage cloud-native ‘as-a-service’ offerings to reduce management complexities and switch to a more economical pay-as-you-go pricing model.

Auto-migrated ~1 TB historical data and up to 4 GB incremental data per day

Solution

To meet the client’s business needs, the Impetus team designed an enterprise-scale hybrid cloud solution and leveraged Impetus’ Workload Transformation Solution to automatically translate Netezza logic to the target environment.

Solution highlights:

  • Identified expensive and poorly performing data processing entities through ML-based assessment
  • Migrated historical data and developed a reliable mechanism for incremental updates to S3
  • Migrated 100+ NZSQL files with approximately 20 queries and 5 Informatica ETL workflows
  • Created structured data lake zones on S3 (landing, preparation, and insights) for quality check and data augmentation for downstream consumers
  • Automatically translated consumption queries (i.e., SQL, stored procedures, Shell, and Scheduler) into Spark SQL/Hive QL
  • Ingested data from S3 to Redshift for reporting and analytics
  • Performed SCD Type 1 and Type 2 for slowly changing dimensions
  • Replicated the Netezza snapshotting process on AWS
  • Used Autosys and Oozie as on-premises/cloud orchestrators
  • Used AWS Lambda as an event-driven notification engine
  • Used Amazon EMR to process vast amounts of data across scalable Amazon EC2 instances
  • Used Amazon RDS to store metadata and workflow information
  • Evaluated and performed tool benchmarking for consumption queries on the cloud
  • Provided on-site support and services to ensure a seamless migration
50% time and cost savings compared to manual migration

Impact

Our comprehensive solution helped the client sunset Netezza and realize the following benefits:

  • Lowered operational expenses by freeing up capacity in Netezza
  • 50% time and cost savings compared to manual migration
  • Improved data availability across the enterprise, enabling innovation