Case Study

Single source of truth from 500+ data feeds — Fortune 500 firm implemented an enterprise data lake on cloud (AWS)

A scalable, one-click data ingestion solution for data pipelines and use cases with built-in robust security, governance, and metadata management


Challenge

A Fortune Global 500 insurance brokerage and risk management company wanted to have a single source of truth and a comprehensive data repository to enhance cost efficiency, performance, and security. They also wanted to migrate their use cases as-is without compromising on data accuracy, SLAs, and existing production applications and business users.


Requirements

The brokerage firm wanted a self-service ingestion platform that could directly be used by business users. They had specific governance and compliance needs and wanted to integrate with enterprise tools to reduce time-to-market. They wanted a solution that would be able to dynamically handle load fluctuation depending on accounting SLAs, priority, and cost.

Additionally, the firm was looking for a cloud-based data lake that would track project costs, provision user authorization for business data access needs, and help them to:

  • Enable robust network policies for on-cloud systems adhering to compliance specifications
  • Ensure audit certification and compliance
  • Enable automated detection, masking, and encryption of PII, PHI, and SPI data
  • Ensure high availability and operational readiness of production systems

Impact

Supporting ingestion of 500+ batch and real-time data feeds on the platform with multi-TB data volume

Reduced release cycle time from 8 weeks to 2 weeks

Saved OpEx of USD 50K per month with continuous monitoring and optimization

Impetus Technologies Inc. migrated all the existing applications to the cloud with zero downtime for the business. A fully secured data lake on the AWS platform with enterprise governance and security compliance was implemented with the following capabilities:

  • Data pipelines connecting Teradata, Oracle, and MS SQL to AWS S3
  • Self-service data ingestion capabilities to automatically pick and process files via a web interface
  • Enabled Consume layer with Tableau, Presto, Jupyter NoteBook, and Hue
  • Voltage-enabled encryption and Ranger powered access policies
  • Automated analysis of data quality profiles and trends
  • High availability of all critical components
  • Single-click deployment for data pipelines and platform
  • End-to-end DevOps for data platform and use cases
  • Serverless data pipelines leveraging Lambda, Step Functions, EMR, and CloudWatch

Moving from disparate sources to a unified data lake created a single source of truth for the insurance firm, enabling a unified, clear and present view of their business.

 


You may also be interested in…

 

linkedin