Continuous Deployment is about making a decision to do things right. It is a clear goal and a proof that the changes across all levels were successful. The primary obstacle is thinking that we can get there without drastic changes in the application’s architecture, processes, and culture. Tools are the least of our problems.
I spend a significant chunk of my time helping companies improve their systems. The most challenging part of my job is going back home after an engagement knowing that the next time I visit the same company, I will discover that there was no significant improvement. I cannot say that is not partly my fault. It certainly is. I might not be very good at what I do. Or maybe I am not good at conveying the right message. Maybe my advice was wrong. There can be many reasons for those failures, and I do admit that they are probably mostly my fault. Still, I cannot shake the feeling that my failures are caused by something else. I think that the root cause is in false expectations.
People want to improve. That is in our nature. Or, at least, most of us do. We became engineers because we are curious. We like to play with new toys. We love to explore new possibilities. And yet, the more we work in a company, the more we become complacent. We learn something, and then we stop learning. We shift our focus towards climbing company ladders. The more time passes, the more emphasis we put on defending our positions which often mean the status quo.
We become experts in something, and that expertise brings us to glory and, hopefully, it lands a promotion, or two. From there on, we ride on that glory. Look at me, I’m a DB2 expert. That’s me, I set up VMWare virtualization. I brought the benefits of Spring to our Java development. Once that happens, we often try to make sure that those benefits stay intact forever. We won’t switch to NoSQL because that would mean that my DB2 expertise is not as valuable anymore. We won’t move to Cloud, because I am the guru behind VMWare. We will not adopt Go, because I know how to code in Java.
Those voices are critical because they are being voiced by senior people. Everyone needs to listen to them, even though the real motivations behind those voices are selfish. They are not based on actual knowledge, but often on repeated experience. Having twenty years of experience with DB2 is not truly twenty years of improvement, but rather the same experience repeated twenty times. Yet, twenty years has weight. People listen to you, but not because they trust you, but because you are senior and management believes in your capabilities to make decisions.
Combine voices from the old with management’s fear of unknown and their quest for short-term benefits. The result is often status quo. That worked for years, why would we change it to something else. Why would I trust a junior developer telling me what to do? Even if a claim for change is backed by the experience from giants like Google, Amazon, and Netflix (just to name a few), you are likely to get a response along the following lines. “We are different”. “That does not apply here.” “I’d like to do that but regulations, which I do not truly understand, prevent me from changing anything.”
Still, sooner or later, a directive to change comes along. Your CTO might have gone to the Gartner meeting where he was told to switch to microservices. Too many people spoke about Agile for upper management to ignore it. DevOps is a huge thing, so we need to employ it as well. Kubernetes is everywhere, so we’ll start working on a PoC soon.
When those things do happen, when a change is approved, you might be ecstatic. This is your moment. This is when you’ll start doing something delicious. That is often the moment when I receive a call. “We want to do this and that. Can you help us?” I usually (not always) say yes. That’s what I do. And yet, I know that my engagement will not produce a tangible improvement. I guess that hope dies last.
Why am I so pessimistic? Why do I think that improvements do not produce tangible benefits? The answer lies in the scope of required changes.
Almost every tool is a result of specific processes. A process, on the other hand, is a product of a particular culture. Adopting a process without making cultural changes is a waste of time. Adopting tools without accepting the processes behind them is a futile effort that will result only in wasted time and potentially substantial license costs. In infrequent occasions, companies do choose to accept the need to change all three (culture, processes, and tools). They make a decision, and sometimes they even start moving in the right direction. Those are precious cases that should be cherished. But they are likely to fail as well. After a while, usually a few months later, we realize the scope of those changes. Only the brave will survive, and only those committed will see it through.
Those who do choose to proceed and indeed change their culture, and their processes, and their tools, will realize that they are incompatible with the applications they’ve been developing over the years. Containers work with everything, but benefits are genuinely tremendous when developing microservices, not monoliths. Test-driven development increases confidence, quality, and speed, but only if applications are designed to be testable. Zero-downtime deployments are not a myth. They work, but only if our applications are cloud-native, if they follow at least some of twelve factors, and so on.
It’s not only about tools, processes, and culture, but also about getting rid of the technical debt you’ve been accumulating over the years. By debt, I don’t necessarily mean that you did something wrong when you started, but rather that time converted something awesome into a horrible monster. Do you spend fifty percent of your time refactoring? If you’re not, you’re accumulating technical debt. It’s unavoidable.
When faced with all those challenges, giving up is the expected outcome. It’s human to throw down the towel when there’s no light at the end of the tunnel. I don’t judge you. I feel your pain. You’re not moving forward because the obstacles are too big. Still, you have to get up because there is no alternative. You will continue. You will improve. It’ll hurt a lot, but there is no alternative, except slow death while your competition is looking over your soon-to-be corpse.
You got this far, and I can assume only two possible explanations. You are one of those who read technical books as a way to escape from reality, or you are applying at least some of the things we discussed thus far. I hope it’s the latter. If that’s the case, you do not fall into “yet another failure of mine.” I thank you for that. It makes me feel better.
If you do employ the lessons from this book, without faking, you are indeed doing something great. There is no way of pretending continuous delivery (CD). Every commit you make is ready for production if all the stages are green. The decision whether to deploy it to production is based on business or marketing needs, and it is not technical in any sense. You can even take a step forward and practice continuous deployment (CDP). It removes the only action performed by a human and deploys every green commit to production. Neither of the two can be faked. You cannot do CD or CDP partly. You cannot be almost there. If you are, you’re doing continuous integration, it-will-be-deployed-eventually process, or something else.
All in all, you are, hopefully, ready to do this. You will take a step towards continuous deployment inside a Kubernetes cluster. By the end of this book, the only thing left for you is to spend an unknown amount of time “modernizing” architecture of your applications or throwing them to thrash and starting over. You’ll be changing your tools, processes, and culture. This book will not help you with all of those. We’re focused on tools and processes. You’ll have to figure out what to do with your culture and architecture. The same holds true for the way you write your tests. I won’t teach you testing, and I won’t preach TDD. I’ll assume that you already know all that and that we can focus on continuous deployment pipeline only.
At this moment, you might feel desperate. You might not be ready. You might think that you don’t have a buyout from your management, that the people in your company will not accept this direction, or that you don’t have enough time and funds. Do not get depressed. Knowing the path is the most crucial part. Even if you cannot get there any time soon, you should still know what the destination is, so that your steps are at least moving you in the right direction.
The DevOps 2.5 Toolkit: Monitoring, Logging, and Auto-Scaling Kubernetes
The article you just read is an extract from The DevOps 2.5 Toolkit: Monitoring, Logging, and Auto-Scaling Kubernetes.
What do we do in Kubernetes after we master deployments and automate all the processes? We dive into monitoring, logging, auto-scaling, and other topics aimed at making our cluster resilient, self-sufficient, and self-adaptive.
Kubernetes is probably the biggest project we know. It is vast, and yet many think that after a few weeks or months of reading and practice they know all there is to know about it. It’s much bigger than that, and it is growing faster than most of us can follow. How far did you get in Kubernetes adoption?
From my experience, there are four main phases in Kubernetes adoption.
In the first phase, we create a cluster and learn intricacies of Kube API and different types of resources (e.g., Pods, Ingress, Deployments, StatefulSets, and so on). Once we are comfortable with the way Kubernetes works, we start deploying and managing our applications. By the end of this phase, we can shout “look at me, I have things running in my production Kubernetes cluster, and nothing blew up!” I explained most of this phase in The DevOps 2.3 Toolkit: Kubernetes.
The second phase is often automation. Once we become comfortable with how Kubernetes works and we are running production loads, we can move to automation. We often adopt some form of continuous delivery (CD) or continuous deployment (CDP). We create Pods with the tools we need, we build our software and container images, we run tests, and we deploy to production. When we’re finished, most of our processes are automated, and we do not perform manual deployments to Kubernetes anymore. We can say that things are working and I’m not even touching my keyboard. I did my best to provide some insights into CD and CDP with Kubernetes in The DevOps 2.4 Toolkit: Continuous Deployment To Kubernetes.
The third phase is in many cases related to monitoring, alerting, logging, and scaling. The fact that we can run (almost) anything in Kubernetes and that it will do its best to make it fault tolerant and highly available, does not mean that our applications and clusters are bulletproof. We need to monitor the cluster, and we need alerts that will notify us of potential issues. When we do discover that there is a problem, we need to be able to query metrics and logs of the whole system. We can fix an issue only once we know what the root cause is. In highly dynamic distributed systems like Kubernetes, that is not as easy as it looks.
Further on, we need to learn how to scale (and de-scale) everything. The number of Pods of an application should change over time to accommodate fluctuations in traffic and demand. Nodes should scale as well to fulfill the needs of our applications.
Kubernetes already has the tools that provide metrics and visibility into logs. It allows us to create auto-scaling rules. Yet, we might discover that Kuberentes alone is not enough and that we might need to extend our system with additional processes and tools. This phase is the subject of this book. By the time you finish reading it, you’ll be able to say that your clusters and applications are truly dynamic and resilient and that they require minimal manual involvement. We’ll try to make our system self-adaptive.
I mentioned the fourth phase. That, dear reader, is everything else. The last phase is mostly about keeping up with all the other goodies Kubernetes provides. It’s about following its roadmap and adapting our processes to get the benefits of each new release.