Cloud transformation is ranked as the cornerstone of innovation and digitalization. The legacy IT infrastructure to run the business operations—mainly data centers—has a deadline to shift to cloud-based services. Agility, innovation, and time-to-value are the key differentiators cloud service providers (CSP) claim to help organizations speed up digital transformation projects and business objectives. The public cloud is increasingly becoming the preferred platform to host data analytics–related projects, such as business intelligence, machine learning (ML), and AI applications. However, the reality is that the “move to cloud” is a turbulent flight for many of them. The main challenges are pointed out as a lack of resources/expertise, security, and from a different perspective, cloud cost management.
The decentralized resource provisioning approach, foundational to cloud solutions and infrastructure, faces a big challenge when it comes to governing the costs of on-demand resource allocation/utilization. Besides the advantages and changes in the technical aspects cloud brings, it imposes a paradigm shift from the traditional way of managing costs and procurement. The traditional approach prioritizes capital expenditure (CapEx), whereas cloud economics favors operating expenses (OpEx).
Innovation and digital transformation can be accelerated in the cloud, taking advantage of the pay-as-you-go model, with no upfront payment and faster time to market. A wide range and virtually unlimited resources and services are available on demand, and IT spending fluctuates based on consumption. Now development teams and engineers can spin up IT resources by themselves, just with a click of a button. The FinOps framework is helping organizations to obtain the best ROI for their cloud transformation. New cool technologies like GPU are enabling more use cases, especially in the Genearative AI (GenAI) and machine learning areas, and the cloud can easily provide those resources on an on-demand basis.
Why FinOps?
Cost management is pointed out as the main challenge companies are facing during their cloud journey. According to the Flexera 2023 survey1, 82 percent of enterprises and SMBs pointed to cost management as the main challenge they are trying to overcome. As cloud security is not a big concern anymore and it is more mature (pointed out by 79 percent of all enterprises as a challenge), the market has seen an increase in the adoption of cloud services in the last few years. For the “as a service” model, the cloud offering is more consolidated, mature, and easy to adopt, which opens up the door for more migration to the cloud and consequently, more cloud infrastructure and platform consumption. It is not strange to hear and know about companies adopting a cloud-first strategy in the short term. According to Gartner Hyper Cycle2, Cloud computing solutions are situated on the plateau of productivity, so more and more companies are starting that journey.
FinOps3—Cloud Financial Operations—is a framework that allows companies to take the most of their investments in the cloud; in other words, maximize ROI. By promoting cross-functional integration and communication, different areas and departments contribute to keeping costs aligned with the business objectives. Technology, finance, and operations collaborate to bring financial accountability to cloud spend. But FinOps is not only about cost management and control. It also improves the procurement process with cloud providers, negotiating better discount rates and plans, maximizing the cloud investment.
Cultural Shift
As organizations seek agility and innovation, cloud computing is the most appropriate way to prototype and launch new products and services, reducing the time to market. That same agility generates inefficiencies and loss of governance, leading to under-estimated cloud requirements, most of the cases resulting in the waste of cloud resources. The traditional “lift and shift” approach is not appropriate when building cost-effective cloud architectures. According to the latest Oomnitza report4, more than half of the surveyed companies recognize they have wasted 10 percent or more of their cloud budget on underutilized, unmanaged, and unattended cloud resources, which reflects the need for a discipline to manage the cloud and integrate financial, operation, and engineering teams.
FinOps—a project part of the Linux Foundation—breaks down the walls that separate technology, business, and finance to improve the cloud unit economics. It’s not just a set of procedures or a piece of software to monitor cost and trigger alerts. It requires a profound cultural change in how to manage and coordinate IT resources, business needs, procurement, and finance to take advantage of the variable cloud model. Transparency is one of the main pillars allowing visibility on how, when, and who uses the cloud resources. The idea is not to spot overspending and find culprits, but instead to improve the usage and investments. Real-time consumption data and reports will allow the teams to make informed decisions on the current spending and also improve financial forecasting and planning. All that combined will improve financial awareness and accountability for cloud transformation.
Cloudera FinOps Capabilities
CDP is a cloud-native platform helping companies accelerate cloud adoption to run their data and analytics workloads. Replication Manager, Observability, and Data Catalog are examples of tools that are part of the Control Plane suite, helping companies to leverage the cloud as their primary infrastructure or as an extension of their data centers for data analytics initiatives and projects.
Besides the technical aspects of moving to the cloud, Cloudera provides tools and features to make sure the workloads are running properly and making the most of the cloud infrastructure. FinOps, as a cloud management discipline, proposes a journey consisting of three phases to help organizations improve their cloud investments. This is how unique Cloudera capabilities are helping customers in the FinOps journey and avoiding cloud overruns.
Inform
Visibility and accountability for workload spending.
This is the first phase of the FinOps process. It is about training all stakeholders with the information and knowledge necessary to make informed decisions about using the cloud. It is about empowering business organizations with better visibility, allocation, benchmarking, and budgeting. Proper time-accurate visibility of the cloud is necessary to make intelligent decisions due to customized pricing, discounts, and the on-demand and elastic nature of the cloud. A team’s productivity can be optimized through accurate features of benchmarking.
Consumption Report
CDP Public Cloud gives visibility to all stakeholders of Cloudera services executing. The Consumption Report offers an aggregated view of how the related credits are being used, but also administrators to drill down by services, date/time, and environment.
Resource tagging
CDP Public Cloud allows administrators to easily add tags to the Data Service and resources the platform deploys on the company’s cloud tenant. Afterward, those tags are also used to track resource usage, assign usage to cost centers/departments, and trigger automation policies. Tags can be defined at tenant, environment, and service level, which adapts to different needs and strategies, and at the same time is propagated to the CSP infrastructure.
Optimize
Reduce cloud spending and increase cloud efficiency.
Optimize the next phase of the FinOps process and focus on discovering cost-saving opportunities. In what areas of your organization can you adjust resources and take advantage of the benefits of discounts based on current use? After the businesses are empowered, the natural stage is to optimize their cloud footprint. For that, cloud service providers offer several resources, including reserved capacity—also known as Saving Plans—which can improve the ROI of cloud investments.
Cloudera Observability
Cloudera Observability is an observability tool that continuously discovers and collects performance telemetry across applications and infrastructure components running in CDP environments. It performs real-time correlations that indicate existing and potential problems and includes prescriptive guidance to address them. Platform administrators are using these insights and recommendations to optimize job performance and execution, which has a direct impact on cloud infrastructure usage.
Chargeback
The Cloudera Observability Financial Governance Chargeback feature collects CPU, memory, and resource usage data from your environment, allocates those charges to your custom cost centers, and visually displays the results. It provides in-depth visibility into the workload resource costs of your environment’s infrastructure that can be used for planning, budgeting, forecasting, and optimization.
Operate
Continuously evaluate and automate cloud financial management practices.
Operating is the final phase of the FinOps process, as organizations continually evaluate their performance concerning business objectives and look for ways to improve their FinOps practices. After defining optimization efforts, automation allows organizations to implement policies that will continually adjust cloud resources to control costs without impacting performance. It is essential to analyze business goals and the steps taken to achieve them.
Compute resource allocation
Cloudera Data Services includes different configurations and parameters to make sure compute allocation dynamically scales up and down accordingly to achieve the right performance/cost balance. Platform administrators can set the right amount of compute resources to meet workload needs, alongside auto-scale threshold, making sure to allocate resources when they’re needed, helping to control and manage cloud spending.
Spot instances
Recently Cloudera added the support for spot instances to facilitate cloud cost savings for workloads that are not SLA-bound. Spot instances are infrastructure provided with high disocunts—80 to 90 percent—that can be reclaimed at any time by the cloud provider. For fault-tolerant workloads such as Apache Spark, utilizing spot instances can provide significant cost savings. Because spot instances can be requested at any time by the CSP (impacting job performance) Cloudera recommends only using them for workloads without strict SLA requirements.
Cloudera <3 FinOps
The features listed above are a partial list of the FinOps capabilities CDP Public Cloud provides. Cloudera is constantly innovating on new functional and integration features and adding and expanding FinOps capabilities to ensure companies have the best ROI for their cloud investments. Besides the out-of-box features, our Cloud Solutions Architect and Customer Success teams proactively guide and assess customers through the journey to provide a safe and cost-effective flight to the cloud.
Cloudera FinOps capabilities span all deployment form factors of CDP cloud-native platform, helping companies to consistently address the technical and financial challenges of moving to cloud.
Now data teams can decide where and when to execute their data analytics workloads, optimizing cost and taking advantage of each infrastructure option. Predictable and persistent workloads can still run on-premise and improve the investments made on the traditional hardware—bare metal and virtualization. On the contrary, seasonal and transient workloads can be easily moved to the public cloud infrastructure. Data is also driving the decisions of where and when to deploy critical workloads seemingly in a hybrid cloud.
Next Steps
Learn more about our brand-new Observability tool that is revolutionizing the way companies take advantage of Lakehouses and help to be FinOps friendly:
Cloudera Observability product page
Blog: Beyond Monitoring: Introducing Cloudera Observability
References:
1- https://info.flexera.com/CM-REPORT-State-of-the-Cloud-2023-Thanks?revisit#challenges
2 – https://www.gartner.com/en/newsroom/press-releases/2022-08-04-cloud-platform-hc-press-release
3 – https://www.finops.org/
4 – https://www.oomnitza.com/resources/2023-snapshot-survey-saas-cloud-spend-optimization-and-automation/
Editor's Choice
