01
Business needs
A Fortune 500 global insurance brokerage firm wanted to build a secured data platform on AWS to enable agile insurance product development and adhere to the guidelines published by the company’s information and security team. The firm had struggled to build an agile platform, and leverage analytics to help customers find insurance programs for risk management in the digital economy. It was looking for a solution that could:
- Provide persona-based regulated access control with audit checks for early threat detection and fraud prevention
- Create self-service dashboards to manage infrastructure provisioning, ingestion, tracking, debugging, and other capabilities
- Centralize all products via a single endpoint and provide a collaborative experience for customers and brokers
- Improve customer experience and onboarding with 24×7 availability of integrated services
- Unlock multiple business use cases by integrating BI tools and leveraging effective and timely insights
02
Technical needs
The insurance brokerage firm wanted to create a unified cloud data platform for onboarding new use cases and building modernized analytics applications. The firm also wanted to focus on data security, compliances, threat protection, and auditing by leveraging a modern data platform, offering the following engineering capabilities:
- Automated on-demand batch and streaming ingestion pipelines
- Automated detection, masking, and encryption of PII, PHI, and SPI data
- Automated metadata management pipelines
- Data quality assurance certification and data enrichment
- Application migration (on-premises to cloud-based re-platform)
- Audit certification and compliance
- A self-service data science workbench with automated model creation, deployment, and training capabilities
Onboarded 1000+ data source integrations, reduced time to insights to near-real-time, and created a single collaboration workspace for brokers, clients, and carriers
03
Solution
To build a unified data platform on AWS with cutting-edge technology, parameters like security compliances, cost, serverless integration capabilities, and other vital factors were considered.
The Impetus team strategically evaluated and regressed the requirements against its in-house models to ensure best practices around operational excellence, security, and performance efficiency. A self-service, secure and governed data platform on AWS was implemented with the following highlights:
- Onboarded 1000+ feeds spanning 1300+ client integrations and 60+ TB of data
- Provided a unified view of business, operational, and technical metadata
- Enabled serverless, configuration-driven automated zero-code ingestion pipelines to support batch and streaming data
- Provided self-service data ingestion for end-to-end management and monitoring
- Automated metadata management pipelines
- Automated data quality profiling and trend analysis
- Enabled automated detection of PII data using technologies like Spark-NLP, EDC, and Macie, and anonymized it using Voltage-provided Format Preserving Encryption (FPE) capability
- Established data tribe to formulate and ensure persona-based compliance governance
- Defined and governed guidelines to onboard, offboard, and access data in the data lake
- Enabled single-click deployment for data pipelines and platforms
- Enabled CloudTrail, CloudWatch, and GuardDuty across cloud environments to monitor, detect, and analyze anomalies, threats, and suspicious activities
- Integrated HashiCorp Vaults for improved secret and credentials management
- Achieved more than 40% cost deduction for AWS services like EMR, Glue, S3, EC2, Elastic Cache, etc. with automation and optimization techniques
- Migrated 60+ data quality pipelines from EMR to Glue
- Onboarded 100+ Spring Boot, Apigee, Angular, Node.js, and Java applications as Docker containers with a single Jenkins pipeline
- Defined a tagging policy and enforced automated tagging for cost tracking
- Automated error handling and recovery for known application and system failures
- Performed multiple timely audits to catch and rectify any process gaps and anomalies
- Enabled 100% migration of applications from a Docker Swarm-based traditional platform to a modern Amazon EKS-based platform
Saved millions of dollars annually in OpEx with continuous monitoring and optimization
04
Impact
Moving from disparate sources to a unified data platform created a single source of truth for the insurance brokerage firm, enabling an integrated, clear, and present view of its business. It enabled the firm to:
- Attain comprehensive self-serve capabilities – from ingestion to intelligent insights
- Built a single collaboration workspace for customers and brokers
- Reduce time to insights from weeks/months to near-real-time
- Create a highly robust system with 99.8% success rate for all data processing jobs
- Generate new revenue streams by delivering new products
- Digitize cargo, pet, personal, auto, workers’ compensation, and DSP insurance
- Save millions of dollars annually in OpEx with continuous monitoring and optimization
- Save cost via AWS-led optimizations (related to S3 lifecycle policies, EMR optimization, CW logs retention policies, EC2 instances, and more)
- Get 2K+ new client registrations for a leading product and extend customer base from the US to 18+ countries
Created a highly robust system with 99.8% success rate for all data processing jobs
Modern serverless solution design and AI/ML analytics helped the insurance brokerage firm unlock new potential and established it as a pioneer in the digital space.
