Driving Engineering Effectiveness through FinOps Principles

There’s  a common misconception that any attempts at optimizing cloud costs can be a major inhibitor for developer innovation and creativity. This line of thought stems from the idea of implementing checks and balances to cloud service consumption, which can restrict a developer when building innovative solutions.

This article addresses the challenge mentioned above and discusses how organizations migrating to the cloud or transitioning to the cloud can optimize costs without stifling innovation.

The solution to unexpected increase in cloud costs

As you go through your cloud journey, technological and time constraints drive you to make decisions that are at odds with the most efficient implementation. The classes of problems include:

1. Incorrectly identifying and using resources: An inability to quickly re-architect your software to take advantage of the key cloud-native services can lead to increased costs.

2. Inefficient allocation and scaling of resources: Insufficient tagging, inefficient management of development environments and failure to right-size your infrastructure can lead to wasted  resources.

Both these problems result in higher spending as you scale your business. 

Thoughtworks FinOps offering can help you solve this problem and explores how you can deliver more business value for your cloud spend. We identify four key components in our strategy:

1. Clearly reporting the allocation, usage, utilization and cloud cost economics.
2. Providing actionable recommendations.
3. Performing a remediation process to clarify your organizational priorities. 
4. Ensure that the organizations can retain the new ways of working both culutrally and technically.

The first two elements — reporting and recommendations — can be handled by many tools and platforms in the industry today. The most challenging, but possibly most important step in addressing your cloud cost problems is developing a strategy to implement the remediation process and sustain this strategic direction.

FinOps to improve engineering effectiveness

Each step of a cloud native development lifecycle can provide solutions to the problem.

Developers are really looking for a more structured use of cloud resources. Making sure you provide them access to these services using a true FinOps Platform that doesn’t only report but also makes recommendations, and which remediates and sometimes instigates the development cycle is the solution to the problem.

A typical cloud native development lifecycle goes through planning, defining, designing and delivering the applications.  Empowering your developers at each step to make decisions that best suit them with a relevant structural framework to avoid bleeding costs will give them the autonomy needed to address this challenge themselves.

Key considerations here are:

• Budgeting and chargeback: Every engineer is aware of and wants to do the right thing by using the available resources. Instead of making this an ad hoc process, provide budgets at the domain level. Start with relevant Wardley mapping, and align this budgeting process to your domain-driven design. One of the core principles of the Thoughtworks FinOps framework is to get your Finance, Product and Technology teams to work together. This consideration enables that.

• Relevant observability  (effective showback): The real challenge in many FinOps journeys is a lack of coherent and actionable data. The problem of increased cloud usage comes from the fact that the current usage and utilization data is not easily available to your engineers. Providing easy access to this data on how they are consuming the cloud resources and services will be an essential component of how you go about solving this problem. This will lead to your development team looking at alternative architectural decisions, considering lower cost options when that is an option and consistently building the applications to be more efficient. FinOps reporting and recommendation tools typically provide you with these “what-if” scenarios. However, integrating that data into the observability platform your developers use makes it easier to correlate usage and utilization alongside cost projections; this will enable  developers to make more informed decisions about how they use cloud resources.

• Automated data governance: The basic tenets of any platform approach are to hide the complexity of the underlying technologies and provide self-serve capabilities that give developers more autonomy and independence in their work. Those principles aren’t that different in this scenario either. To enable better autonomy and increase self-serve automated lightweight governance should be implemented across all cloud environments in a consistent manner. The governance should be built around sensible defaults that the developers should be able to change and still be able to stay compliant. This will ensure that the developer autonomy is preserved and unintended costs are avoided. This obviously requires higher levels of autonomy from the developers, but most organizations trust their developers with their most valuable assets, so the lightweight governance is merely used as a check and balance to ensure that any unintentional errors are avoided. The Internal Developer Portal (IDP) used within your organization can be an effective alternative to do your showback and automated governance, fitting in nicely with your engineering effectiveness initiative.

• Reducing waste: Most organizations we work with have a culture of FinOps that is driven through finance and technology, which is great. However, bringing more of your product lifecycle knowledge into this mix is no longer an optional activity. The waste reduction we are referring to here isn’t so much related to those times when you’re operating the product at scale, but instead to experimentation and development phases. You might be surprised that we strongly recommend flexible architectural (software and infrastructure) governance that can be shaped by the results of experimentation. This additional level of governance built into a higher level of data visualization platform described above can positively impact the behavior of developers.

The following illustration demonstrates how your development life cycle can coincide with the FinOps actions that will most help improve the effectiveness and efficiency of your development team:

In the planning phase, choosing suitable pricing models happens in parallel or as a post-experimental activity. It certainly shouldn’t be ignored, because these decisions will have long-term ramifications on your solutions. 

In the define and design phase, you definitely want to be doing all the necessary activities to integrate reporting, improve architectural rigor and ensure that resources are right-sized and tagged.

In the delivery phase, taking a data-driven approach to build models that learn from prior experiences and align with business goals as you implement continuous delivery can neatly tie this all together.

Conclusion

Based on our experience, the core FinOps principles and cloud cost optimization can greatly improve your engineering effectiveness and culture. Implementing the right FinOps principles can provide engineers visibility into the cost of doing business and encourage them to think of alternative ways to architect solutions that make the most sense for the business. 

The following two tables summarize how various key engineering metrics and FinOps metrics can be improved by using a combined approach in your organization.

Organizations can increase their engineering team’s effectiveness by being intentional about how to go about approaching FinOps across the board in your Cloud Native Development lifecycle. They should take this approach instead of banking on tactical remediation techniques for the most egregious of your cloud cost problems. Doing so does not have to come at the expense of additional business investment opportunities. Thoughtworks' FinOps offering allows you  to incorporate these principles into your existing delivery ecosystem and when paired with our engineering effectiveness capabilities, your organization can make impactful progress.