In this article, we'll discuss a way to forward logs from containers created as Docker Swarm services inside our clusters. We'll use the ELK stack. They'll be forwarded from containers to LogStash and, from there, to ElasticSearch. Once in the database, they will be available through Kibana.
We'll start by creating a Docker Swarm cluster. I will assume you already have at least a basic knowledge how Docker Swarm Mode works and that you know how to create Docker services. If you don't, I suggest you read the Docker Swarm Introduction (Tour Around Docker 1.12 Series) article or fetch The DevOps 2.1 Toolkit: Docker Swarm book.
Some of the files will be shared between the host file system and Docker Machines we'll create soon. Docker Machine makes the whole directory that belongs to the current user available inside the VM. Therefore, please make sure that the code is cloned inside one of the user's sub-folders.
A note to Windows users
The recommendation is to run all the examples from Git Bash (installed through Docker Toolbox as well as Git). That way the commands you'll see throughout the book will be same as those that should be executed on OS X or any Linux distribution.
We cloned the
cloud-provisioning repository and executed the scripts/dm-swarm.sh script that created the production cluster.
Let's confirm that the cluster was indeed created correctly.
The output of the
node ls command is as follows (IDs are removed for brevity).
Now that the production cluster is up and running, we can create ELK services.
That's it. We should have a few services running inside the Swarm cluster. Let's double check it.
The output is as follows (IDs are removed for brevity).
We are, finally, ready to explore how to ship logs from our Swarm services to LogStash, and from there ElasticSearch.
Forwarding Logs To LogStash
How can we forward logs from all the containers no matter where they're running? One possible solution would be to configure logging drivers. We could use
--log-driver argument to specify a driver for each service. The driver could be
syslog or any other supported option. That would solve our log shipping problem. However, using the argument for each service is tedious and, more importantly, we could easily forget to specify it for a service or two and discover the omission only after we encounter a problem and are in need for logs. Let's see if there is another option to accomplish the same result.
We could specify a log driver as a configuration option of the Docker daemon on each node. That would certainly make the setup easier. After all, there are probably fewer servers than services. If we were to choose between setting a driver when creating a service or as the daemon configuration, I'd choose the later. However, we managed to get thus far without changing the default daemon configuration and I'd prefer if we can continue working without involving any special provisioning tools. Luckily, we still did not exhaust all our options.
We can ship logs from all our containers with the project called logspout.
LogSpout is a log router for Docker containers that runs inside Docker. It attaches to all containers on a host, then routes their logs wherever we want. It also has an extensible module system. It's a mostly stateless log appliance. It's not meant for managing log files or looking at history. It is just a tool to get your logs out to live somewhere else, where they belong.
If you go through the project documentation, you'll notice that there are no instructions how to run it as a Docker service. That should not matter since, by this time, you can consider yourself an expert in creating services.
What do we need from a service that should forward logs from all the containers running inside all the nodes that form a cluster? Since we want to forward them to LogStash that is already attached to the
elk network, we should attach LogSpout to it as well. We need it to ship logs from all the nodes so the service should be
global. It needs to know that the destination is the service called
logstash and that it listens on the port
51415. Finally, one of the LogSpout's requirements is that Docker socket from the host is mounted inside the service containers. It'll use it to monitor logs.
The command that creates the service that fulfills all those objectives and requirements is as follows.
We created a service called
logspout, attached it to the
elk network, set it to be
global, and mounted the Docker socket. The command that will be executed once containers are created is
syslog://logstash:51415. It tells LogSpout that we want to use
syslog protocol to send logs to
logstash running on port
This project is an example of a usefulness behind Docker Remote API. The
logspout containers will use it to retrieve the list of all currently running containers and stream their logs. This is already the second product inside our cluster that uses the API (the first being Docker Flow: Swarm Listener).
Let's see the status of the service we just created.
The output is as follows (IDs are removed for brevity).
The service is running in the global mode resulting in an instance inside each node.
Let's test whether the
logspout service is indeed sending all the logs to LogStash. All we have to do is create a service that generates some logs and observe them from LogStash' output. We'll use the
registry to test the setup we made so far.
Before we check the LogStash logs, we should wait until the
registry is running.
If the current state is still not running, please wait a few moments.
Now we can take a look at
logstash logs and confirm that
logspout sent it log entries generated by the
One of the entries from the output is as follows.
As before when we tested LogStash input with
logger, we have the
host, and a few other
syslog fields. We also got
logsource that holds the ID of the container that produced the log as well as
program that holds the container name. Both will be useful when debugging which service and container produced a bug.
If you go back to the command we used to create the
logstash service, you'll notice the environment variable
LOGSPOUT=ignore. It tells LogSpout that the service or, to be more precise, all containers that form the service, should be ignored. If we did not define it, LogSpout would forward all
logstash logs to
logstash thus creating an infinite loop. As we already discussed, in production we should not output LogStash entries to
stdout. We did it only to get a better understanding how it works. If
stdout output is removed from the
logstash configuration, there would be no need for the environment variable
LOGSPOUT=ignore. As a result
logstash logs would also be stored in ElasticSearch.
Now that we are shipping all the logs to LogStash and from there to ElasticSearch, we should explore the ways to consult them. You'll find them in the
Before leaving, please make sure to remove the machines we created and free your resources for some other tasks.
The article you just finished reading is an extract from the Defining Logging Strategy chapter of The DevOps 2.1 Toolkit: Docker Swarm book.
The DevOps 2.1 Toolkit: Docker Swarm
If you liked this article, you might be interested in The DevOps 2.1 Toolkit: Docker Swarm book. Unlike the previous title in the series (The DevOps 2.0 Toolkit: Automating the Continuous Deployment Pipeline with Containerized Microservices) that provided a general overlook of some of the latest DevOps practices and tools, this book is dedicated entirely to Docker Swarm and the processes and tools we might need to build, test, deploy, and monitor services running inside a cluster.
Give the book a try and let me know what you think.