A unified view of enterprise data is an essential building block for making an informed decision. However, a single version of truth continues to elude the enterprise due to multiple data sources functioning in silos
At Impetus, we aim to simplify the complexities arising from hundreds of point solutions and enable intelligent decision-making by creating a single source of truth. This blog talks about our partnership with a leading American airline to successfully retire its proprietary Opera-based Customer Signal Hub (CSH) and Trip Narrative applications.
Why was this project unique?
This migration had a history of failed attempts. Before we worked on the project, three vendor partners were unsuccessful in modernizing the Customer Signal Hub (CSH) and Trip Narrative applications because of the following reasons:
Lack of planning and execution
The migration required the re-architecting of many components. However, lack of proper feasibility analysis led the vendors to believe that as-is migration with a ‘lift and shift’ approach would work.
Lack of transparent communication
The execution required close collaboration with various stakeholders. The previous vendor partners took this as a project to be delivered and did not collaborate with the client’s team to build trust and align them at every phase.
After multiple failed migration attempts, the airline partnered with us to replace its third-party customer data analytics solution with an in-house, cost-effective, performant, and extensible solution. They wanted to retire the Opera CSH platform to reduce cost and replace Opera’s Signal Hub platform with an agile solution that provides quick, impactful customer insights. The airline was looking to:
- Save $2.5 million annually in Opera licensing cost
- Reduce latency to deliver actionable customer insights
- Reduce third-party dependency to ensure access to insights
- Ensure a single source of truth for all customer and related datasets
As transformation partners, every project we deliver poses a unique set of challenges. However, this migration was particularly complex because of the following reasons:
Migrating huge volumes of high-velocity data
- 160+ batch and real-time datasets from 10+ data sources
- 6 billion transactions for 108 million customers and ~3 TB+ historical data
- ~200 GB of daily operational incremental data for signals
The information to drive customer data analytics, actionable insights, and effective customer engagement was stored in the Opera black box, which was not accessible to business users and SMEs.
No single source of truth
There were no written business requirements. Moreover, numerous data stores, divisional data marts, and multiple tools led to inconsistent data versions.
History of failed migration attempts
Many industry leading partners made multiple unsuccessful attempts to migrate CSH from Teradata to Hadoop.
Stringent delivery timelines
Since their Opera license was expiring, the airline had a strict timeline of six months for delivering the solution
Key drivers of success
Considering the critical delivery aspects, we banked on a highly competent data analytics team and our proven execution process driven by customer success, technical expertise, and in-depth knowledge to ensure the project’s success. Some of the drivers that helped us in successful delivery are:
A highly experienced and committed data analytics team included architects and engineers with experience and acumen to simplify complex processing logic.
Delivering one release per week increased testability of releases.
Frequent collaboration with functional owners allowed for a quick turn-around on gaps and issues.
For migrating Teradata workloads to a Spark-based component, TPumps and Mloads need to be redesigned and re-implemented (they cannot be migrated as-is). We had to revisit the requirements, SLAs, and business expectations for multiple scenarios.
While migrating Spark-based real-time ETL pipelines to Palantir, we faced multiple challenges with the platform. Close collaboration, clear communication of risks and implications helped us migrate seamlessly.
Opera-based CSH was a black box for the client, who was discovering new aspects every day with the progress of the implementation. Accepting that we cannot have all the answers before we start helped us move one step at a time, with known and calculated risk.
World-class delivery processes
A thorough evaluation to design a robust and scalable solution backed by AWS-based Palantir foundry with the following features:
- Ability to recover without any data loss/data corruption in an event of failure
- Built-in replay and reconciliation mechanism to ensure data quality and sanctity
- Real-time alerting and monitoring framework to raise an alarm in case of application, platform, data quality, and other issues
- Strict adherence to coding standards
- Use of rigorous testing and sanity suite with automated test cases
- Complete UAT and end-user testing support
- Well-planned and closely monitored parallel run
- Adequate documentation, user training, and knowledge transfer
Close collaboration with the business and product owners, a motivated and focused team, and a fail-fast approach gave us time to recalibrate and successfully deliver this project.