Business needs
An American Fortune 1000 railcar products and services provider wanted to upgrade their Databricks workspaces on AWS and Azure to Databricks Unity Catalog, addressing challenges like fragmented view of data and AI estate, disjointed access management tools, and limited monitoring and observability. The growing number of AI use cases heightened the urgency to migrate.
Key objectives included:
- Improve access control and data governance for multiple user personas within the Databricks ecosystem
- Unify data assets under a single governance framework for centralized data visibility to improve collaboration
- Minimize business disruption during legacy metastore migration to the Databricks Unity Catalog
- Enable seamless cross-platform data sharing across cloud regions
- Improve monitoring and observability by ensuring comprehensive oversight of data access and activities across all workspaces
- Simplify the migration process, minimizing manual efforts
Delivered the 14-week migration project on schedule, achieving a 3x reduction in costs
Solution
The Impetus team leveraged its Unity Catalog Migration Accelerator (UCMA) to facilitate the railcar provider’s transition to Databricks Unity Catalog, ensuring a secure, efficient, and automated migration.
Why Impetus Unity Catalog Migration Accelerator (UCMA)?
- Assessment dashboard for clear asset visibility
- Fully automated migration of tables, views, clusters, jobs, and notebooks
- Thorough validation of all workspace assets for accuracy and consistency
The solution approach involved upgrading:
Tables and views:
- Prepared scripts for the Unity Catalog upgrade
- Migrated tables and views to Unity Catalog
- Created storage credentials, external locations, and UC Catalogs
Users and groups:
- Migrated local workspace groups, users, and ACLs to the account level
- Validated the migration of users and groups
Jobs:
- Prepared scripts for the Unity Catalog upgrade
- Modified job/notebook code for namespace changes, cluster policies, and mount points
- Verified job functionality post-migration, including metadata validation and performance
Mount point:
- Identified and created external locations and volumes for mount points
- Replaced mount points in code programmatically and validated changes
Interactive clusters:
- Checked and upgraded cluster Databricks Runtime (DBR)
- Upgraded cluster policies to enforce Unity Catalog
Figure 1: Solution architecture
Solution highlights
- Performed an in-depth assessment for five workspaces across AWS and Azure
- Migrated local workspace groups with automated permission configuration
- Upgraded mount points to external locations for seamless data access
- Updated clusters and jobs to Unity Catalog-supported DBR versions for compatibility
- Ensured accurate migration of tables and views to Unity Catalog
- Performed static code analysis and upgraded notebooks for post-migration functionality
- Validated the migrated Databricks objects, table counts, and job performance
- Executed a seamless switch-over post-validation, ensuring uninterrupted operations
Impact
The migration to Databricks Unity Catalog revolutionized the client’s operations, driving significant improvements:
- Unified governance and automation: Centralized metadata, user management, and access controls while automating permissions management across all workspaces
- Cost and time efficiency: Completed the 14-week migration on schedule, achieving a 3x cost reduction
- Accelerated AI/ML outcomes: Cleared AI/ML backlogs, unlocking faster insights and data monetization
- Enhanced performance: Improved query response times with materialized views in Unity Catalog
- Seamless cross-cloud sharing: Enabled frictionless data transfer between AWS and Azure via Delta Sharing
