01
Challenges
A digital customer journey experience company was facing frequent outrages with Kafka on open-source Confluent, which led to message loss and impacted client businesses.
Confluent was unable to process from 4 different Kafka-dependent consumers, handle sudden surge in data volumes, and process data in near-real-time for analytics. Moreover, there was difficulty in scaling, installing upgrades, managing brokers, and monitoring failures of Confluent brokers.
To address these challenges, the company wanted to migrate their customer digital journey platform to Amazon Managed Streaming for Kafka (MSK) brokers as a self-managed service to:
- Handle Kafka servers with high availability of data to server management efforts
- Get out-of-the-box metrics to monitor the health of the brokers
- Automatically scale up in case of a sudden surge in data volumes
Released 2 DevOps resources to focus on other operations
02
Solution
The Impetus team migrated 5 open-source Confluent environments to Amazon MSK in ~2 months, with provision to upscale/downscale volume.
A high-level architecture diagram of the solution is given below:
Highlights
- Redirected all producers and consumers to MSK brokers
- Retained other Kafka components like Schema-registry, connector services, and Kafka-rest on open-source Confluent binaries on AWS EKS
- Used Kafka MirrorMaker to migrate data from Confluent to MSK brokers
Completed migration of 5 environments in ~2 months
03
Impact
Transitioning to Amazon MSK with automated server provisioning, cluster configuration, and failure handling helped the client:
- Release 2 DevOps resources to focus on other operations
- Ensure zero outrages and no downtime with automated load balancing
- Support data volumes in TBs, while open-source Confluent supported up to 500GB
- Ensure zero data loss during the maintenance window
04
Amazon MSK advantage
- No memory shortage due to broker unavailability
- Secure data storage and scaling to support load changes
- Continuous monitoring and alarms for broker usage
- Regular insights of CPU and volume usage
- Easy access to MSK brokers from AWS Lambda Functions, Amazon EC2, and AWS EMR
- Smooth switchover across multiple servers
