Real-time baggage tracking for a major American airlines with Amazon MSK - Impetus

Real-time baggage tracking for a major American airlines with Amazon MSK

Migrated Bag Manager application to enhance customer experience and reduce lost baggage claims


Business needs

One of the major American airlines wanted to move their Bag Manager application from Teradata to AWS to:

  • Reduce baggage data processing time for immediate consumption and analytics 
  • Add more checkpoints for bags in transit, for example, scanned, added to storage, loaded/unloaded, etc. 
  • Mobilize airlines’ ground staff for better management of scheduled flights 
  • Track changes in the state of the bags 

Processed ~60 multi-level nested JSON messages per second with an average size of 80 KB



The Impetus team leveraged Amazon Managed Streaming for Kafka (MSK) to process ~60 multi-level nested JSON messages per second with an average size of 80 KB. The solution used the all-in-one data pipeline platform Gathr for consuming data from Amazon MSK. Apart from processing 20 GB of data per day with ~1,500,000 records, Gathr also helped to:

  • Parse and process the message to a structured format for the consumption layer 
  • Implement CDC with SCD type 1 strategy

A high-level solution architecture diagram is given below:


  • Designed and implemented a separate pipeline to move historical data to Redshift for deep analytics 
  • Used hash function at row level to improve SCD-1 performance 
  • Implemented a utility to merge and compress the small files generated by Gathr on S3 
  • Employed Gathr health checks to monitor data anomalies and duly report to the users

Reduced baggage data processing time from 20 minutes to real-time



Migrating the baggage tracking application to Amazon MSK helped the airline to:

  • Reduce baggage data processing time from 20 mins to real-time for immediate consumption and analytics
  • Save cost and time by routing baggage to the right destination in case of misconnects or itinerary change
  • Resolve issues quickly in case of misconnects/itinerary changes/flight delays
  • Enhance customer experience with real-time baggage tracking

Choose a lab aligned to your Data & AI journey

Address your desired use case across critical analytic dimensions

  • Explore architecture options with experts

  • Ensure strategic alignment of business and technology

  • Architect an ideal solution for a pressing problem

  • Validate new or refactor existing architecture

  • Develop a prototype with expert guidance

  • Establish a roadmap to production

Learn more about how our work can support your enterprise