Accelerated migration of customer digital journey platform to Amazon MSK - Impetus

Accelerated migration of customer digital journey platform to Amazon MSK

Confluent to Amazon MSK migration for a global market leader in B2B digital sales to support data volume surge



A digital customer journey experience company was facing frequent outrages with Kafka on open-source Confluent, which led to message loss and impacted client businesses.  

Confluent was unable to process from 4 different Kafka-dependent consumers, handle sudden surge in data volumes, and process data in near-real-time for analytics. Moreover, there was difficulty in scaling, installing upgrades, managing brokers, and monitoring failures of Confluent brokers.  

To address these challenges, the company wanted to migrate their customer digital journey platform to Amazon Managed Streaming for Kafka (MSK) brokers as a self-managed service to: 

  • Handle Kafka servers with high availability of data to server management efforts 
  • Get out-of-the-box metrics to monitor the health of the brokers 
  • Automatically scale up in case of a sudden surge in data volumes 

Released 2 DevOps resources to focus on other operations



The Impetus team migrated 5 open-source Confluent environments to Amazon MSK in ~2 months, with provision to upscale/downscale volume.  

A high-level architecture diagram of the solution is given below:


  • Redirected all producers and consumers to MSK brokers 
  • Retained other Kafka components like Schema-registry, connector services, and Kafka-rest on open-source Confluent binaries on AWS EKS
  • Used Kafka MirrorMaker to migrate data from Confluent to MSK brokers

Completed migration of 5 environments in ~2 months



Transitioning to Amazon MSK with automated server provisioning, cluster configuration, and failure handling helped the client:

  • Release 2 DevOps resources to focus on other operations 
  • Ensure zero outrages and no downtime with automated load balancing 
  • Support data volumes in TBs, while open-source Confluent supported up to 500GB  
  • Ensure zero data loss during the maintenance window


Amazon MSK advantage

  • No memory shortage due to broker unavailability 
  • Secure data storage and scaling to support load changes 
  • Continuous monitoring and alarms for broker usage 
  • Regular insights of CPU and volume usage  
  • Easy access to MSK brokers from AWS Lambda Functions, Amazon EC2, and AWS EMR 
  • Smooth switchover across multiple servers

Choose a lab aligned to your Data & AI journey

Address your desired use case across critical analytic dimensions

  • Explore architecture options with experts

  • Ensure strategic alignment of business and technology

  • Architect an ideal solution for a pressing problem

  • Validate new or refactor existing architecture

  • Develop a prototype with expert guidance

  • Establish a roadmap to production

Learn more about how our work can support your enterprise