I am continuously getting questions about blue-green releases inside a Docker Swarm cluster. Viktor, in your The DevOps 2.0 Toolkit book you told us to use blue-green deployment. How do we do it with services running inside a Swarm cluster? My answer is usually something along the following lines. With the old Swarm, blue-green releases were easier than rolling updates (neither were supported out of the box). Now we got rolling updates. Use them! The reaction to that is often that we still want blue-green releases.

This post is my brainstorming on this subject. I did not write it as a result of some deep thinking. There is no great wisdom in it. I just wrote what was passing through my mind while I was trying to answer another one of the emails containing blue-green deployment questions. What follows might not make much sense. Don’t be harsh on me.

Rolling updates through the docker service update command provide out of the box zero-downtime deployments. It works great, we like it, and we were even willing to ignore bugs it contained in early releases. Actually, we were not ignoring bugs. The first release with Swarm Mode (v1.12.0) had too many bugs. But, it was the first release of a completely new product, so we opened a lot of issues and hoped for the best. Indeed, Docker folks responded quickly and fixed many in the next release, most in the one after that, and so on. A few releases later rolling updates worked like a charm.

Going back to blue-green deployment… We want it. We want it. We want it. Why? We’re not sure, but we want it now. OK. That was enough from my side. I should not mock others! I’m being nasty and should express myself in a more politically correct way. Even though I live in Spain where political correctness still did not reach the mainstream, I should know better. Ignore what I said.

The only challenge with rolling updates (as far as I can think of) is how to test a new release before it becomes available to the public. The real question is not whether we should test deployment to production but what should be tested in production. Each environment should have its set of tests executed after deployment, and production should not be an exception.

How do others test their releases in production? Now that we have zero-downtime deployments through rolling updates, do we need blue-green as a way to test a new release in production?

We could deploy canary releases instead. It’s a practice used in quite a lot of companies. A short pause. This is one of those moments when I catch myself saying a lie. Canary releases are not used in quite a lot of companies. The majority of our industry lives around the year 1999 (especially big enterprise). Most are still struggling to understand that there is something called cloud computing, many are still using WebSphere, automated testing is an unreachable dream, and so on, and so forth. Heck, Cobol is still a thing and many did not reach further than “we will deploy it during the weekend because there will be downtime and we do not know whether it will work” approach to releasing software to production. So, I take it back. Not many companies are using canary releases but those that do know what they’re doing (and that’s a minority) adopted it. A short pause while my mind recuperates from another reset. I got off the topics, again.

In a nutshell, a canary release would start by updating some of the replicas, test whether the update works as expected, increase the number of replicas with the new release, test some more, and so on. The process is repeated until all the replicas run the new release. Throughout the process, we would observe logs, monitor some metrics, and what so not. Even though there are a few things left to be desired (see issue 26160), we can do canary releases with docker service update command.

My understanding is that the challenge might be in testing canary releases. Personally, I don’t worry whether a new feature is working as expected from the functional point of view. Since containers are immutable (at least when done right), I am very confident that what I tested in integration environment is the same as what I’ll deploy to production. Immutability vastly reduces the things we need to verify when running a new release in production. Performance should also not be an issue because that should also be tested before deploying a new release to production. That does not mean that we should not monitor production and make sure that each release is performing as expected. Quite the contrary. The thing is that blue-green deployment does not help us with that. It allows us to run our performance tests, but that’s what testing environments are for. What it does not allow us (and canary releases do) is to test (monitor) performance when “real” users use a new release and gradually increase the reach of the release.

So, we do not need to test new features in production, nor we should run performance tests in production (excluding monitoring which is testing in a way). The only thing I can think of as a missing validation is whether users find a new feature useful. For that, blue-green does not help much since users cannot see the release until it is available to the public. Canary releases handle that much better since we can monitor things gradually and, depending on metrics (e.g. do they click the new button) decide whether to continue increasing the number of replicas that are updated with the new release. If canary releases are not your thing, there are feature toggles. Or just use both…

What I’m trying to say is that I’m not sure how useful would be to have blue-green deployment considering that zero-downtime deployments are provided out-of-the-box through rolling updates. Through immutability that containers give us, we can be confident that what runs in production is exactly the same as what was tested. So, why do we want blue-green releases? I can come up with only a few possible answers to that question.

1. We want it because we are used to it. We are trying to apply logic we used before we adopted containers and started orchestrating our cluster with Docker Swarm (Mode).
2. We do not know that there is such a thing as rolling updates baked into Docker engine.
3. We are not confident in immutability. We do not believe that what was tested in other environments will be the same as what we’ll deploy to production.

None of those answers is valid. Our industry is changing very fast, and we cannot allow ourselves to do something only because we’re used to it. Docker Swarm (and rolling updates) were introduced half a year ago, so there’s no excuse for not knowing it exists. If our container adoption did not provide immutability, the chances are that there is something wrong with our system’s architecture. Containers require changes on many levels, and if we are not ready to implement those changes, maybe we should not adopt containers just yet.

I’m not trying to say that there is no use-case for blue-green deployment with Swarm Mode. I just have a hard time to see it myself. If it would come out of the box, sure, why not. We could use them in some cases and switch to rolling updates in other. But they are not included. We need to build a custom logic to support it, and I am failing to find good enough arguments for such an effort.

Quite a few of you are asking me for blue-green releases. Is it only because I advocated in their favor while we were using the old Swarm? If that’s the case, that Swarm is gone, and many of the assumptions we had are gone with it.

I would like to get your opinion. Do you want to use blue-green deployments with Swarm Mode or you switched to rolling updates? If you do, would it make sense to create a project that would enable blue-green releases in Swarm Mode or you’d prefer custom every-men-for-himself scripts?

Join us in DevOps20 Slack Channel and let’s discuss your use case.

The DevOps 2.1 Toolkit: Docker Swarm

If you liked this article, you might be interested in The DevOps 2.1 Toolkit: Docker Swarm book. Unlike the previous title in the series (The DevOps 2.0 Toolkit: Automating the Continuous Deployment Pipeline with Containerized Microservices) that provided a general overlook of some of the latest DevOps practices and tools, this book is dedicated entirely to Docker Swarm and the processes and tools we might need to build, test, deploy, and monitor services running inside a cluster.

You can get a copy from Amazon.com (and the other worldwide sites) or LeanPub. It is also available as The DevOps Toolkit Series bundle.

Give the book a try and let me know what you think.