While cloud adoption continues to accelerate, with 36% of enterprises spending more than $12 million per year on public clouds, businesses are looking for ways to optimize their cloud spend. According to Flexera 2021 State of the Cloud Report, organizations waste 30% of cloud spend, making controlling cloud cost the top challenge in public cloud adoption.
Gartner predicts that through 2024, nearly all legacy applications migrated to public cloud infrastructure as a service (IaaS) will require optimization to become more cost-effective. This blog details how we helped a Fortune Global 500 insurance brokerage and risk management company halve their cloud cost by optimizing their spend.
01
Identifying the primary challenge
A few months after implementing an AWS-based data lake for a Fortune 500 firm, we discovered that their in-house IT team was unable to control cloud consumption costs. They were using AWS Cost Explorer, which monitored cloud costs for different teams in silos, without optimizing usage.
The realization prompted us to explore why the cost spiraled with increasing amounts of data. To optimize cost, we adopted a “shift-left” approach, which is widely used for maintaining code quality.
02
Optimizing cloud cost through visibility
Our focus was on empowering the Dev teams, the primary consumers of cloud services and resources, to gain more visibility into the costs based on their usage. This, in turn, would make them more accountable and enable them to monitor costs effectively.
How the Impetus team optimized cloud cost
- Defined tagging policy and guidelines for AWS resources with tag keys like application, business unit, environment, etc.
- Updated the automation scripts to ensure all created resources were tagged.
- Created custom rules to scan all resources and report non-tagged resources to the management.
- Created a tagging strategy for various services to filter resources for cost analysis.
- Created alerts for services reaching/exceeding threshold
- Enabled automated termination of long-running EMR clusters
- Enabled termination of idle EMR clusters on weekends
- Compared historical price patterns and peaks with existing usage to effectively plan an auto-scaling mechanism
- Reported idle/stopped resources and security vulnerabilities
- Created a tagging solution for all services and components
- Created dashboards for billing and custom dashboards for tagged resources
- Designed a strategy to gather information from all accounts in a single billing dashboard on AWS. This helped to compare consumption across Dev, Stage, Prod, and Labs accounts. The report was further analyzed to optimize Dev spending and formulate stricter policies on resource provisioning and termination
- Implemented lifecycle rules on Amazon S3 buckets to automatically move old data to lower storage tiers, resulting in reduced storage costs
- Used spot instances for EMR core nodes, which are up to 50% cheaper than on-demand instances
03
Results of the customized solution
The customized cost analysis solution with built-in controls provided specific access to Dev teams to continuously monitor and track cloud cost usage for taking corrective actions based on their consumption.
Increased visibility and control
- Connection with AWS, Azure, and GCP accounts to collect and analyze cloud costs in one place
- Quick insights categorized by region, services, tags, and more
- Simplified workflows to manage the resource tags and maintain tag hygiene for accurate cost analysis
- Rule-based alerts for any spikes and recommendations to improve cost-efficiency
- Reduceed multi-cloud wastage by analyzing spends against budgets and forecasting costs and usage
The solution helped the Fortune 500 insurance firm reduce their monthly cloud expenditure from $120000 to $60000.
04
Key takeaways
Monitoring cloud costs continuously and optimizing resource utilization are critical for reducing cloud spend and realizing the benefits of the cloud. While every cloud provider offers reports and dashboards to track resource consumption and costs, correlating the data, identifying provisioning inefficiencies, controlling virtual sprawl, and analyzing cloud spend across multiple providers can be challenging. The implemented solution (a custom cost explorer) simplifies hybrid and multi-cloud cost analysis by consolidating expense data from different cloud platforms and accounts. It helps businesses optimize their cloud expenditure and predict future costs.
