Let’s paint a high-level picture of the continuous delivery pipeline. To be more precise, we’ll draw a diagram instead of painting anything. But, before we dive into a continuous delivery diagram, we’ll refresh our memory with the one we used before for describing continuous deployment.

The continuous deployment pipeline contains all the steps from pushing a commit to deploying and testing a release in production.

Continuous delivery removes one of the stages from the continuous deployment pipeline. We do NOT want to deploy a new release automatically. Instead, we want humans to decide whether a release should be upgraded in production. If it should, we need to decide when will that happen. Those (human) decisions are, in our case, happening as Git operations. We’ll comment on them soon. For now, the important note is that the deploy stage is now removed from pipelines residing in application repositories.

The fact that our application pipeline (e.g., go-demo-5) does not perform deployment does not mean that it is not automated. The decisions which versions to use and when to initiate the upgrade process is manual, but everything else proceeding those actions is automated.

In our case, there is a separate repository (k8s-prod) that contains a full definition of what constitutes production environment. Whenever we make a decision to install a new application or to upgrade an existing one, we need to update files in k8s-prod and push them to the repository. Whether that push is performed directly to the master branch or to a separate branch, is of no importance to the process that relies solely on the master branch. If you choose to use separate branches (as you should), you can do pull requests, code reviews, and all the other things we usually do with code. But, as I already mentioned, those actions are irrelevant from the automation perspective. The master branch is the one that matters. Once a commit reaches it, it initiates a Webhook request that notifies Jenkins that there is a change and, from there on, we run a build that upgrades the production environment and executes light-weight tests with sanity checks. Except, that we did not set up GitHub Webhooks. I expect that you will have them once you create a “real” cluster with a “real” domain.

How does continuous delivery of applications combine with unified deployment to the production environment?

Let’s imagine that we have four applications in total. We’ll call them app 1, app 2, app 3, and app 4. Those applications are developed independently of each other. Whenever we push a commit to the master branch of one of those applications, corresponding continuous delivery pipeline is initiated and, if all the steps are successful, results in a new production-ready release. Pipelines are launched when code is pushed to other branches as well, but in those cases, production-ready releases are NOT created. So, we’ll ignore them in this story.

We are accumulating production-ready releases in those applications and, at some point, someone makes a decision to upgrade the production environment. Those upgrades might involve an update of a single application, or it might entail update of a few. It all depends on the architecture of our applications (sometimes they are not independent), business decisions, and quite a few other criteria. No matter how we made the decision which applications to update and which releases to use, we need to make appropriate changes in the repository that serves as the source of truth of the production environment.

Let’s say that we decided to upgrade app 1 to the release 2 and app 4 to release 4, to install the release 3 of app 2 for the first time, and to leave app 3 intact. In such a situation, we’d bump versions of app 1 and 4 in requirements.yaml. We’d add a new entry for app 2 since that’s the first time we’re installing that application. Finally, we’d leave app 3 in requirements.yaml as-is since we are not planning to upgrade it.

Once we’re finished with modifications to requirements.yaml, all that’s left is to bump the version in Chart.yaml and push the changes directly to master or to make a pull request and merge it after a review. No matter the route, once the change reaches the master branch, it fires a Webhook which, in turn, initiates a new build of the Jenkins job related to the repository. If all of the steps are successful, the Chart representing the production environment is upgraded and, with it, all the applications specified in requirements.yaml are upgraded as well. To be more precise, not all the dependencies are upgraded, but only those we modified. All in all, the production environment will converge to the desired stage after which we’ll execute the last round of tests. If something fails, we roll back. Otherwise, another iteration of production deployments is finished, until we repeat the same process.

To Continuously Deploy Or To Continuously Deliver?

Should we use the continuous deployment (CDP) or the continuous delivery (CD) process? That’s a hard question to answer which mainly depends on your internal processes. There are a few questions that might guide us.

1. Are your applications truly independent and can be deployed without changing anything else in your cluster?
2. Do you have such a high level of trust in your automated tests that you are confident that there’s no need for manual actions?
3. Are the teams working on applications authorized to make decisions on what to deploy to production and when?
4. Are those teams self-sufficient and do not depend on other teams?
5. Do you really want to upgrade production with every commit to the master branch?

If you answered with no to at least one of those questions, you cannot do continuous deployment. You should aim for continuous delivery, or not even that. Continuous delivery is almost as hard to practice as continuous deployment. The chances are that you cannot get there any time soon. If you can’t, that’s still not the end of the world. The lessons from this chapter can be easily modified to serve other processes.

If you answered with no to the second question (the one about tests), you cannot do either of those two processes. It’s not that one requires less confidence in tests than the other. The level of trust is the same. We do not use continuous delivery because we trust our tests less, but because we choose not to deploy every commit to production. Our business might not be ready to deploy every production-ready release. Or, maybe, we need to wait for a marketing campaign to start (I’m ignoring the fact that we’d solve that with feature toggles). There might be many reasons to use continuous delivery instead of deployment, but none of them is technical. Both processes produce production-ready releases, and only one of them deploys it to production automatically.

Now, if you do NOT trust your tests, you need to fall back to continuous integration. Fortunately, the pipeline can be very similar. The major difference is that you should create one more repository (call it k8s-test) and have a similar Jenkinsfile inside it. When you think you’re ready, you’ll bump the versions in that repo and let Jenkins upgrade the test environment. From there on, you can let the army of manual testers do their work. They will surely find more problems than you’re willing to fix but, once they stop finding those that impede you from upgrading the production, you can bump those versions in the repository that describes your production environment (k8s-prod). Apart from different Namespaces and, maybe, reduced number of replicas and Ingress hosts, the two repositories should contain the same Chart with similar dependencies, and changes to their master branch would result in very similar automated processes. You can even skip having the k8s-test repository and create a test-env branch in k8s-prod. That way, you can make changes to test-env, deploy them to the cluster, run manual testing, and, once you’re confident that the release is production-ready, merge the branch to master.

The DevOps 2.4 Toolkit: Continuous Deployment To Kubernetes

The article you just read is an extract from The DevOps 2.4 Toolkit: Continuous Deployment To Kubernetes.

This book explores continuous deployment to a Kubernetes cluster. It uses a wide range of Kubernetes platforms and provides instructions on how to develop a pipeline on few of the most commonly used CI/CD tools.

I am assuming that you are already proficient with Deployments, ReplicaSets, Pods, Ingress, Services, PersistentVolumes, PersistentVolumeClaims, Namespaces and a few other things. This book assumes that we do not need to go through the basic stuff. At least, not through all of it. The book assumes a certain level of Kubernetes knowledge and hands-on experience. If that’s not the case, what follows might be too confusing and advanced. Please read The DevOps 2.3 Toolkit: Kubernetes first, or consult the Kubernetes documentation. Come back once you’re done and once you think you can claim that you understand at least basic Kubernetes concepts and resource types.

Give it a try and let me know what you think.