Kubernetes has emerged the de-facto container orchestrator for on-premise and cloud infrastructure setups. And, in their scramble to implement the fairly new infrastructure, organizations seem to be struggling.
It’s imperative for technologists eager to containerize applications and adopt Kubernetes, to know where to begin and how, to know about the maintenance and scaling of infra, and also be able to handle tech complexities and cost investments along this journey.
Organizations adopting the Kubernetes ecosystem require a robust change program. And, the biggest change required is a mindshift or shift in mindset.
The mindshift should be along two critical perspectives – the organisation’s view point and the development teams’ view point. Based on Thoughtworks’ extensive experience of working on huge programs with large enterprises, we are sharing a few guidelines on how to make the needed shift.
In part 1 of this 2 part series, we will focus on the organization.
Experimentation
Organizations benefit from a ‘fail-fast’ culture that minimizes uncertainty and builds confidence by encouraging experimentation.
Organizations that consistently re-iterate, reflect, try new things and re-assess are more suited to build an adaptive architecture that's expandable, and easy to rollback without risk.
Lean governance
In spite of adopting new tools and technologies, organizations fail to adopt new team structures and governance processes. The subsequent growing portfolio of systems affects the quality and time taken to complete work.
Organizations should establish that building the right culture isn't the ‘platform-building IT team’s’ responsibility alone. The entire organization should come together to break silos, cross-collaborate and ensure innovation at the intersection of technology and business.
Autonomous cross-functional teams
Organizations should promote autonomous teams who will have the freedom to establish their own internal goals and work practices. This motivates them to produce reliable and high quality output.
Automate the pain away
Automation is a key principle of evolutionary architecture. Adoption of new architecture and managing the change effectively can involve provisioning end-to-end ‘infrastructure as code’ using automation tools (like Terraform and Ansible). It could also include adhering to software development practices like version control, testable infrastructure with continuous integration and continuous delivery that bake security in.
In organizations that nurture such an environment, failed experiments lead to lean builds with safety nets, iterative cycles and space for exploration.
Kubernetes guarantees powerful infrastructure capabilities such as scalability, resiliency, availability and more. Such capabilities were home only to digital native companies until recently.
However, the powerful infra capabilities come with its own complexities. Infact, most implementations suffer runaway complexity, especially in the case of higher abstractions – ingress controllers, endpoint slicing, dynamic volume provisioning etc.
Kubernetes promises a single-command rollout of deployment, but managing the production grade-cluster is tough work. Expected challenges are centered on upgrades, maintenance and security.
On-cloud
Cloud-based Kubernetes cluster eliminates the control over master nodes and its performance. Our recommendation is to watch the Kubernetes API server’s performance when deploying components like custom Kubernetes operators. Additionally, ‘forced’ Kubernetes upgrades have to be factored in when planning releases.
We’d also advise keeping an eye out for deprecated API and its usage. As upgrades are automatic, debt will have to be cleared as soon as possible else, upgrades will cause production downtime.
On-premise
Kubeadm is production ready and can be used to set up clusters. However when on-premise, infrastructure components like on-demand storage and load balancers along with Kubernetes’ internal network will require additional setting up.
To extend the Kubernetes (K8s) clusters, open source tools like Rook Operator for ceph distributed storage solution, MetalLB for software load balancing, Canal for networking inside K8s clusters are available with different interfaces. However, this only adds levels of complexity to both the K8s cluster setup and maintenance on-premise.
On-security
Kubernetes increases the surface attack on one’s architecture. With each K8s cluster setup, there are additional components like Helm or Prometheus running on the cluster. And in a containerized environment, every new application ushers in a new set of threats.
Our recommendation is checking tasks off, including scanning images, ensuring their credibility, limiting access to nodes, isolating resources by namespace and network, managing secrets, establishing resource quotas, ensuring everything runs in the correct security context and logging everything.
Organizations have to invest in in-house capability to make the best use of such a powerful and feature-rich tool. We are talking about skilled experts who can administer, build, operate, manage and expand the platform. For instance, experts will know not to use separate Nginx or API gateways for path based routing, they could simply leverage an ingress controller.
We’d also like to take this moment to clarify a very common doubt that technologists have – yes, core concepts of Kubernetes like StatefulSets and Operators allow for deployment of databases inside the Kubernetes cluster. Infact, official Helm charts and Operators are available for most of the databases like Postgresql, MongoDB, Redis, Cassandra.
Our recommendation is to seek external help from specialized IT service companies for the initial period of heavy lifting, and take the needed time to build in-house capability.
Kubernetes is a free and open-source tool, but organizations sometimes fail to factor in the additional costs of adopting K8s. There’s the usual cost of core infrastructure components (like
cluster nodes, load balancers and storage volumes), of managed services, of cluster maintenance and operations. Additional to this is the cost of hiring experts and upskilling the existing team.
Kubernetes builds world-class infrastructure capabilities with commodity hardware, and our recommendation is to avoid heavy investments on base infrastructure capabilities that K8s are known to handle well.
Else, organizations will waste resources on building the same capabilities. For instance both RAID solutions at base infrastructure and Ceph setup for Kubernetes cluster provide storage solutions.
Our observation is that indirect cost-saving benefits usually appear in the form of better quality product, zero down-time, frequent deployments and better customer satisfaction.
In conclusion, organizations that adopt the 4C framework better understand the quantum of change and are proactive about creating a short-term and long-term plan to adopt a new age infrastructure.
In part 2 of this series, we will look at the needed mindshift from the development team’s perspective.
It’s imperative for technologists eager to containerize applications and adopt Kubernetes, to know where to begin and how, to know about the maintenance and scaling of infra, and also be able to handle tech complexities and cost investments along this journey.
Organizations adopting the Kubernetes ecosystem require a robust change program. And, the biggest change required is a mindshift or shift in mindset.
The mindshift should be along two critical perspectives – the organisation’s view point and the development teams’ view point. Based on Thoughtworks’ extensive experience of working on huge programs with large enterprises, we are sharing a few guidelines on how to make the needed shift.
In part 1 of this 2 part series, we will focus on the organization.
The organizational lens
Organizations should leverage the 4C framework – Culture, Complexity, Capability and Cost when adopting containers and Kubernetes.
Culture:
ExperimentationOrganizations benefit from a ‘fail-fast’ culture that minimizes uncertainty and builds confidence by encouraging experimentation.
Organizations that consistently re-iterate, reflect, try new things and re-assess are more suited to build an adaptive architecture that's expandable, and easy to rollback without risk.
Lean governance
In spite of adopting new tools and technologies, organizations fail to adopt new team structures and governance processes. The subsequent growing portfolio of systems affects the quality and time taken to complete work.
Organizations should establish that building the right culture isn't the ‘platform-building IT team’s’ responsibility alone. The entire organization should come together to break silos, cross-collaborate and ensure innovation at the intersection of technology and business.
Autonomous cross-functional teams
Organizations should promote autonomous teams who will have the freedom to establish their own internal goals and work practices. This motivates them to produce reliable and high quality output.
Automate the pain away
Automation is a key principle of evolutionary architecture. Adoption of new architecture and managing the change effectively can involve provisioning end-to-end ‘infrastructure as code’ using automation tools (like Terraform and Ansible). It could also include adhering to software development practices like version control, testable infrastructure with continuous integration and continuous delivery that bake security in.
In organizations that nurture such an environment, failed experiments lead to lean builds with safety nets, iterative cycles and space for exploration.
Complexity:
Kubernetes guarantees powerful infrastructure capabilities such as scalability, resiliency, availability and more. Such capabilities were home only to digital native companies until recently. However, the powerful infra capabilities come with its own complexities. Infact, most implementations suffer runaway complexity, especially in the case of higher abstractions – ingress controllers, endpoint slicing, dynamic volume provisioning etc.
Kubernetes promises a single-command rollout of deployment, but managing the production grade-cluster is tough work. Expected challenges are centered on upgrades, maintenance and security.
On-cloud
Cloud-based Kubernetes cluster eliminates the control over master nodes and its performance. Our recommendation is to watch the Kubernetes API server’s performance when deploying components like custom Kubernetes operators. Additionally, ‘forced’ Kubernetes upgrades have to be factored in when planning releases.
We’d also advise keeping an eye out for deprecated API and its usage. As upgrades are automatic, debt will have to be cleared as soon as possible else, upgrades will cause production downtime.
On-premise
Kubeadm is production ready and can be used to set up clusters. However when on-premise, infrastructure components like on-demand storage and load balancers along with Kubernetes’ internal network will require additional setting up.
To extend the Kubernetes (K8s) clusters, open source tools like Rook Operator for ceph distributed storage solution, MetalLB for software load balancing, Canal for networking inside K8s clusters are available with different interfaces. However, this only adds levels of complexity to both the K8s cluster setup and maintenance on-premise.
On-security
Kubernetes increases the surface attack on one’s architecture. With each K8s cluster setup, there are additional components like Helm or Prometheus running on the cluster. And in a containerized environment, every new application ushers in a new set of threats.
Our recommendation is checking tasks off, including scanning images, ensuring their credibility, limiting access to nodes, isolating resources by namespace and network, managing secrets, establishing resource quotas, ensuring everything runs in the correct security context and logging everything.
Capability:
Organizations have to invest in in-house capability to make the best use of such a powerful and feature-rich tool. We are talking about skilled experts who can administer, build, operate, manage and expand the platform. For instance, experts will know not to use separate Nginx or API gateways for path based routing, they could simply leverage an ingress controller. We’d also like to take this moment to clarify a very common doubt that technologists have – yes, core concepts of Kubernetes like StatefulSets and Operators allow for deployment of databases inside the Kubernetes cluster. Infact, official Helm charts and Operators are available for most of the databases like Postgresql, MongoDB, Redis, Cassandra.
Our recommendation is to seek external help from specialized IT service companies for the initial period of heavy lifting, and take the needed time to build in-house capability.
Costs:
Kubernetes is a free and open-source tool, but organizations sometimes fail to factor in the additional costs of adopting K8s. There’s the usual cost of core infrastructure components (likecluster nodes, load balancers and storage volumes), of managed services, of cluster maintenance and operations. Additional to this is the cost of hiring experts and upskilling the existing team.
Kubernetes builds world-class infrastructure capabilities with commodity hardware, and our recommendation is to avoid heavy investments on base infrastructure capabilities that K8s are known to handle well.
Else, organizations will waste resources on building the same capabilities. For instance both RAID solutions at base infrastructure and Ceph setup for Kubernetes cluster provide storage solutions.
Our observation is that indirect cost-saving benefits usually appear in the form of better quality product, zero down-time, frequent deployments and better customer satisfaction.
In conclusion, organizations that adopt the 4C framework better understand the quantum of change and are proactive about creating a short-term and long-term plan to adopt a new age infrastructure.
In part 2 of this series, we will look at the needed mindshift from the development team’s perspective.
Disclaimer: The statements and opinions expressed in this article are those of the author(s) and do not necessarily reflect the positions of Thoughtworks.
