Docker 1.13 introduced a set of features that allow us to centrally manage secrets and pass them only to services that need them. They provide a much-needed mechanism to provide information that should be hidden from anyone except designated services.
A secret (at least from Docker’s point of view) is a blog of data. A typical use case would be a certificate, SSH private keys, passwords, and so on. Secrets should stay secret meaning that they should not be stored unencrypted or transmitted over a network.
With all that being said, let’s see them in action and continue our discussion through practical examples.
Since a single node is more than enough to demonstrate Docker secrets, we’ll start by creating a one node Swarm cluster based on Docker Machines.
A note to Windows users
The recommendation is to run all the examples from Git Bash (installed through Docker Toolbox as well as Git). That way the commands you’ll see throughout the article will be same as those that should be executed on OS X or any Linux distribution.
docker-machine create \
-d virtualbox \
eval $(docker-machine env swarm)
docker swarm init \
–advertise-addr $(docker-machine ip swarm)
We created a Docker Machine node called
swarm and used it to initialize the cluster.
Now we can create a secret.
A note to Windows users
For mounts (a secret is a mount as well) used in the next command to work, you have to stop Git Bash from altering file system paths. Set this environment variable.
The format of the command that creates a secret is as follows (please do not run it).
docker secret create [OPTIONS] SECRET file|-
secret create command expects a file that contains a secret. However, creating a file with unencrypted secret defies the purpose of having secrets in the first place. Everyone can read that file. We could, delete the file after pushing it to Docker but that would only create unnecessary steps. Instead, we’ll use
- that will allow us to pipe standard output.
echo "I like candy" \
| docker secret create my_secret –
The command we just executed created a secret called
my_secret. That information was sent to the remote Docker Engine using TLS connection. If we had a bigger cluster with multiple managers, the secret would be replicated among all.
We can inspect the newly created secret.
docker secret inspect my_secret
The output is as follows.
The value of the secret is hidden. Even if a malicious person gains access to Docker Engine, the secret would still be unavailable. Truth be told, in such a case, our worries would be much greater that protection of a Docker secret but I’ll leave that discussion for some other time.
Now that we have encrypted the secret and stored in Swarm managers, we should explore ways to utilize it within our services.
A new argument
--secret was added to the
docker service create command. If a secret is attached, it will be available as a file in the
/run/secrets directory inside all the containers that form a service.
Let’s see it in action.
docker service create –name test \
–secret my_secret \
–restart-condition none \
alpine cat /run/secrets/my_secret
We created a service called
test and attached the secret called
my_secret. The service is based on
alpine and will output the content of the secret. Since it is a one-shot command that will terminate quickly, we set
none. Otherwise, the service would terminate a moment after it’s created, Swarm would reschedule it, only to see it terminate again, and so on. We would enter a never-ending loop.
Let’s take a look at the logs.
docker logs $(docker container ps -qa)
The output is as follows.
I like candy
The secret is available as the
/run/secrets/my_secret file inside the container.
Before we start discussing a more real-world example, let us remove the service and the secret we created.
docker service rm test
docker secret rm my_secret
A Real-World Example of Using Secrets
Docker Flow Proxy project exposes statistics that should be reserved for internal use only. Therefore, it needs to be protected with a username and password. Before Docker v1.13, situations like that one would be handled by allowing users to specify username and password through environment variables. Docker Flow Proxy is no exception and, indeed, has the environment variables
The command that would create the service with custom username and password would be as follows.
docker network create –driver overlay proxy
docker service create –name proxy \
-p 80:80 \
-p 443:443 \
-p 8080:8080 \
-e STATS_USER=my-user \
-e STATS_PASS=my-pass \
–network proxy \
-e MODE=swarm \
While that would protect the statistics page from ordinary users, it would still leave it exposed to anyone capable of inspecting the service. A simple example is as follows.
docker service inspect proxy –pretty
The relevant part of the output is as follows.
Env: STATS_USER=my-user STATS_PASS=my-pass MODE=swarm
The same result that does not reveal confidential information could be accomplished with the commands that follow.
echo "secret-user" \
| docker secret create dfp_stats_user –
echo "secret-pass" \
| docker secret create dfp_stats_pass –
docker service update \
–secret-add dfp_stats_user \
–secret-add dfp_stats_pass \
We created two secrets (
dfp_stats_pass) and updated our service. From now on, those secrets would be available inside service containers as files
/run/secrets/dfp_stats_pass. If a secret is named the same as the environment variable, is in lower case, and has the
dpf_ prefix, it will be used instead.
If you inspect the container one more time, you’ll notice that there is no trace of the secrets.
We could stop here. After all, there’s not much more to be said for Docker secrets. However, we got used to Docker stacks and it would be great if secrets would work in the new YAML Compose format.
Before we move on, let’s remove the
docker service rm proxy
Using Secrets With Docker Compose
True to the mission to have the same features available in all supported flavours, Docker introduced secrets in Compose YAML format version 3.1.
We’ll continue using Docker Flow Proxy to demonstrate how secrets work inside Compose files.
curl -o dfp.yml \
The relevant parts of the definition of the stack are as follows.
The version of the format is 3.1. The
proxy service has the two secrets attached. Finally, there is a separate
secrets section that defines the secrets as external entities. The alternative would be to specify secrets internally. An example would be as follows.
I prefer the first option that specifies secrets externally since that does not leave any trail. In some other cases, secrets might be used for non-secretive information (we’ll discuss it soon) and using internal secrets specified as files would probably be a better option.
Let’s run the stack and check whether it works.
docker stack deploy -c dfp.yml proxy
Statistics themselves are useless if there is no data so we’ll deploy another service that will be reconfigured in the proxy and start generating some stats.
curl -o go-demo.yml \
docker stack deploy -c go-demo.yml go-demo
Please wait a few moments until the services from the
go-demo stack are running. You can check their status by executing
docker stack ps go-demo. You might see
go-demo_main replicas in the failed status. Do not panic. They will continue failing only until the
go-demo_db is starts running.
Now we can, finally, confirm that the proxy is configured to use secrets for authentication.
curl -u secret-user:secret-pass \
"http://$(docker-machine ip swarm)/admin?stats;csv;norefresh"
It works! With only a single additional step (
docker service create), we made our system more secured.
Common Ways to Use Secrets
Until secrets were introduced, a common way to pass information to containers was through environment variables. While that will continue being the preferable way for non-confidential information, part of the setup should involve secrets as well. Both should be combined. The question is which method to choose and when.
The obvious use case for Docker secrets are secrets. That was obvious, wasn’t it. If there is a piece of information that should remain invisible to anyone but specific containers, it should be provided through Docker secrets. A commonly used pattern is to allow the same information to be specified as either environment variable and a secret. In case that both a set, secrets should take precedence. You already saw this pattern through Docker Flow Proxy. Every piece of information that can be specified through environment variables can be specified as a secret as well.
In some cases, you might not be able to modify code of your service and adapt it to use secrets. Maybe it’s not a question of ability but lack of desire to modify your code. If you fall into the latter case, I will, for now, restrain myself from explaining why code should be continuously refactored and imagine that you have a very good reason for it. In either case, the solution is usually to create a wrapper script that transforms secrets into whatever your service needs and then invoke the service. Put that script as
CMD instruction in Dockerfile and you’re done. Secrets stay secrets and you don’t get fired from refactoring your code. To some this last sentence sounds silly but it’s not uncommon for companies to consider refactoring a waste of time.
What should be a secret? No one can truly answer that question for you since it differs from one organization to another. Some of the examples would be usernames and passwords, SSH keys, SSL certificates, and so on. If you don’t want others to know about it, make it a secret.
We should strive for immutability and do our best to run containers that are exactly the same no matter where they run. True immutability means that even the configuration is always the same across all environments. However, that is not always easy and is sometimes even impossible to accomplish. Such a situation could be a good candidate for Docker secrets. They do not necessarily have to be used only as means of specifying confidential information. We can use secrets as a way to provide information that differs from one cluster to another. In such a case, pieces of configuration that should differ from one environment to another (e.g. staging and production clusters) can be stored as secrets.
I am certain that there are quite a few other use cases I didn’t even think about. After all, secrets are a new feature (a few weeks old from the day of this writing).
I would like to get your feedback. Do you think that secrets would be helpful and if you do, what would be your use case? Please join slack.devops20toolkit.com/ Slack channel and let me know.
Remove your Docker Machine VM and start applying secrets to your own Swarm cluster. There’s not much more to be said (for now).
docker-machine rm -f swarm
The DevOps 2.1 Toolkit: Docker Swarm
If you liked this article, you might be interested in The DevOps 2.1 Toolkit: Docker Swarm book. Unlike the previous title in the series (The DevOps 2.0 Toolkit: Automating the Continuous Deployment Pipeline with Containerized Microservices) that provided a general overlook of some of the latest DevOps practices and tools, this book is dedicated entirely to Docker Swarm and the processes and tools we might need to build, test, deploy, and monitor services running inside a cluster.
Give the book a try and let me know what you think.