Delivering excellence: An airline’s critical modernization initiative involving 6 billion transactions

A unified view of enterprise data is an essential building block for making an informed decision. However, a single version of truth continues to elude the enterprise due to multiple data sources functioning in silos.

At Impetus, we aim to simplify the complexities arising from hundreds of point solutions and enable intelligent decision-making by creating a single source of truth. This blog talks about our partnership with a leading American airline to successfully retire its proprietary Opera-based Customer Signal Hub (CSH) and Trip Narrative applications.

Why was this project unique?

This migration had a history of failed attempts. Before we worked on the project, three vendor partners were unsuccessful in modernizing the Customer Signal Hub (CSH) and Trip Narrative applications because of the following reasons:

  • Lack of planning and execution
  • The migration required the re-architecting of many components. However, lack of proper feasibility analysis led the vendors to believe that as-is migration with a ‘lift and shift’ approach would work.

  • Lack of transparent communication
  • The execution required close collaboration with various stakeholders. The previous vendor partners took this as a project to be delivered and did not collaborate with the client’s team to build trust and align them at every phase.

After multiple failed migration attempts, the airline partnered with us to replace its third-party customer data analytics solution with an in-house, cost-effective, performant, and extensible solution. They wanted to retire the Opera CSH platform to reduce cost and replace Opera’s Signal Hub platform with an agile solution that provides quick, impactful customer insights. The airline was looking to:

  • Save $2.5 million annually in Opera licensing cost
  • Reduce latency to deliver actionable customer insights
  • Reduce third-party dependency to ensure access to insights
  • Ensure a single source of truth for all customer and related datasets

Challenges

As transformation partners, every project we deliver poses a unique set of challenges. However, this migration was particularly complex because of the following reasons:

  • Migrating huge volumes of high-velocity data, including:
    • 160+ batch and real-time datasets from 10+ data sources
    • 6 billion transactions for 108 million customers and ~3 TB+ historical data
    • ~200 GB of daily operational incremental data for signals
  • Information accessibility: The information to drive customer data analytics, actionable insights, and effective customer engagement was stored in the Opera black box, which was not accessible to business users and SMEs.
  • No single source of truth: There were no written business requirements. Moreover, numerous data stores, divisional data marts, and multiple tools led to inconsistent data versions.
  • History of failed migration attempts: Many industry leading partners made multiple unsuccessful attempts to migrate CSH from Teradata to Hadoop.
  • Stringent delivery timelines: Since their Opera license was expiring, the airline had a strict timeline of six months for delivering the solution

Key drivers of success

Considering the critical delivery aspects, we banked on a highly competent data analytics team and our proven execution process driven by customer success, technical expertise, and in-depth knowledge to ensure the project’s success. Some of the drivers that helped us in successful delivery are:

  • A highly experienced and committed data analytics team, including architects and engineers with experience and acumen to simplify complex processing logic
  • Delivery in small, testable releases (one release per week)
  • Frequent collaboration with functional owners for a quick turn-around on gaps and issues. For example,
    • For migrating Teradata workloads to a Spark-based component, TPumps and Mloads need to be redesigned and re-implemented (they cannot be migrated as-is). We had to revisit the requirements, SLAs, and business expectations for multiple scenarios.
    • While migrating Spark-based real-time ETL pipelines to Palantir, we faced multiple challenges with the platform. Close collaboration, clear communication of risks and implications helped us migrate seamlessly.
    • Opera-based CSH was a black box for the client, who was discovering new aspects every day with the progress of the implementation. Accepting that we cannot have all the answers before we start helped us move one step at a time, with known and calculated risk.
  • World-class delivery processes, which involved:
    • A thorough evaluation to design a robust and scalable solution backed by AWS-based Palantir foundry with the following features:
      • Ability to recover without any data loss/data corruption in an event of failure
      • Built-in replay and reconciliation mechanism to ensure data quality and sanctity
      • Real-time alerting and monitoring framework to raise an alarm in case of application, platform, data quality, and other issues
    • Strict adherence to coding standards
    • Use of rigorous testing and sanity suite with automated test cases
    • Complete UAT and end-user testing support
    • Well-planned and closely monitored parallel run
    • Adequate documentation, user training, and knowledge transfer

Close collaboration with the business and product owners, a motivated and focused team, and a fail-fast approach gave us time to recalibrate and successfully deliver this project.

From the customer’s desk

"This is a major accomplishment that would help serve our customers better through access to their travel details with minimal latency, apart from a yearly cost saving of ~$2.5M by insourcing the asset from a third-party vendor. An incredible example of resiliency and teamwork amidst the global crisis!" — Managing Director – Data Engineering

 

"We want to thank the Impetus team for the tremendous effort they have consistently delivered to make this project a success. Impetus team’s contentiousness and attention to detail have been driving factors that have made this delivery a success." — Data Engineering, Portfolio Delivery Lead – CSD, AO, Cargo

 

These client testimonials reflect our relentless commitment to ensuring customer success. This is just one example of how Impetus has helped several Fortune 1000 enterprises successfully modernize their legacy data stores. While the needs and challenges of each migration project are different, the core strategies and drivers for success apply to all. To know more about how we can help you determine the right migration strategy, connect with us.

Author
Shilpi Saxena
Director Of Engineering