Forwarding Logs From All Containers Running Anywhere Inside A Docker Swarm Cluster

In this article, we’ll discuss a way to forward logs from containers created as Docker Swarm services inside our clusters. We’ll use the ELK stack. They’ll be forwarded from containers to LogStash and, from there, to ElasticSearch. Once in the database, they will be available through Kibana.

Environment Setup

We’ll start by creating a Docker Swarm cluster. I will assume you already have at least a basic knowledge how Docker Swarm Mode works and that you know how to create Docker services. If you don’t, I suggest you read the Docker Swarm Introduction (Tour Around Docker 1.12 Series) article or fetch The DevOps 2.1 Toolkit: Docker Swarm book.

Some of the files will be shared between the host file system and Docker Machines we’ll create soon. Docker Machine makes the whole directory that belongs to the current user available inside the VM. Therefore, please make sure that the code is cloned inside one of the user’s sub-folders.

A note to Windows users

The recommendation is to run all the examples from Git Bash (installed through Docker Toolbox as well as Git). That way the commands you’ll see throughout the book will be same as those that should be executed on OS X or any Linux distribution.

git clone https://github.com/vfarcic/cloud-provisioning.git

cd cloud-provisioning

scripts/dm-swarm.sh

We cloned the cloud-provisioning repository and executed the scripts/dm-swarm.sh script that created the production cluster.

Let’s confirm that the cluster was indeed created correctly.

eval $(docker-machine env swarm-1)

docker node ls

The output of the node ls command is as follows (IDs are removed for brevity).

HOSTNAME  STATUS  AVAILABILITY  MANAGER STATUS
swarm-2   Ready   Active        Reachable
swarm-1   Ready   Active        Leader
swarm-3   Ready   Active        Reachable

Now that the production cluster is up and running, we can create ELK services.

scripts/dm-swarm-services-elk.sh

That’s it. We should have a few services running inside the Swarm cluster. Let’s double check it.

docker service ls

The output is as follows (IDs are removed for brevity).

NAME            REPLICAS  IMAGE                               COMMAND
swarm-listener  1/1       vfarcic/docker-flow-swarm-listener
logstash        1/1       logstash:2.4                        logstash -f /conf/logstash.conf
elasticsearch   1/1       elasticsearch:2.4
kibana          1/1       kibana:4.6
proxy           1/1       vfarcic/docker-flow-proxy

We are, finally, ready to explore how to ship logs from our Swarm services to LogStash, and from there ElasticSearch.

Forwarding Logs To LogStash

How can we forward logs from all the containers no matter where they’re running? One possible solution would be to configure logging drivers. We could use --log-driver argument to specify a driver for each service. The driver could be syslog or any other supported option. That would solve our log shipping problem. However, using the argument for each service is tedious and, more importantly, we could easily forget to specify it for a service or two and discover the omission only after we encounter a problem and are in need for logs. Let’s see if there is another option to accomplish the same result.

We could specify a log driver as a configuration option of the Docker daemon on each node. That would certainly make the setup easier. After all, there are probably fewer servers than services. If we were to choose between setting a driver when creating a service or as the daemon configuration, I’d choose the later. However, we managed to get thus far without changing the default daemon configuration and I’d prefer if we can continue working without involving any special provisioning tools. Luckily, we still did not exhaust all our options.

We can ship logs from all our containers with the project called logspout.

LogSpout is a log router for Docker containers that runs inside Docker. It attaches to all containers on a host, then routes their logs wherever we want. It also has an extensible module system. It’s a mostly stateless log appliance. It’s not meant for managing log files or looking at history. It is just a tool to get your logs out to live somewhere else, where they belong.

If you go through the project documentation, you’ll notice that there are no instructions how to run it as a Docker service. That should not matter since, by this time, you can consider yourself an expert in creating services.

What do we need from a service that should forward logs from all the containers running inside all the nodes that form a cluster? Since we want to forward them to LogStash that is already attached to the elk network, we should attach LogSpout to it as well. We need it to ship logs from all the nodes so the service should be global. It needs to know that the destination is the service called logstash and that it listens on the port 51415. Finally, one of the LogSpout’s requirements is that Docker socket from the host is mounted inside the service containers. It’ll use it to monitor logs.

The command that creates the service that fulfills all those objectives and requirements is as follows.

docker service create --name logspout \
    --network elk \
    --mode global \
    --mount "type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock" \
    -e SYSLOG_FORMAT=rfc3164 \
    gliderlabs/logspout syslog://logstash:51415

We created a service called logspout, attached it to the elk network, set it to be global, and mounted the Docker socket. The command that will be executed once containers are created is syslog://logstash:51415. It tells LogSpout that we want to use syslog protocol to send logs to logstash running on port 51415.

This project is an example of a usefulness behind Docker Remote API. The logspout containers will use it to retrieve the list of all currently running containers and stream their logs. This is already the second product inside our cluster that uses the API (the first being Docker Flow: Swarm Listener).

Let’s see the status of the service we just created.

docker service ps logspout

The output is as follows (IDs are removed for brevity).

NAME          IMAGE                NODE     DESIRED STATE  CURRENT STATE          ERROR
logspout      gliderlabs/logspout  swarm-3  Running        Running 9 seconds ago
 \_ logspout  gliderlabs/logspout  swarm-2  Running        Running 9 seconds ago
 \_ logspout  gliderlabs/logspout  swarm-1  Running        Running 9 seconds ago

The service is running in the global mode resulting in an instance inside each node.

Let’s test whether the logspout service is indeed sending all the logs to LogStash. All we have to do is create a service that generates some logs and observe them from LogStash’ output. We’ll use the registry to test the setup we made so far.

docker service create --name registry \
    -p 5000:5000 \
    --mount "type=bind,source=$PWD,target=/var/lib/registry" \
    --reserve-memory 100m \
    registry

Before we check the LogStash logs, we should wait until the registry is running.

docker service ps registry

If the current state is still not running, please wait a few moments.

Now we can take a look at logstash logs and confirm that logspout sent it log entries generated by the registry.

LOGSTASH_NODE=$(docker service ps logstash | tail +2 | awk '{print $4}')

eval $(docker-machine env $LOGSTASH_NODE)

LOGSTASH_ID=$(docker ps -q --filter "ancestor=logstash:2.4")

docker logs $LOGSTASH_ID

One of the entries from the output is as follows.

{
           "message" => "time=\"2016-10-19T23:14:19Z\" level=info msg=\"listening on [::]:5000\" go.version=go1.6.3 instance.id=87c31e30-a747-4f70-b7c2-396dd80eb47b version=v2.5.1 \n",
          "@version" => "1",
        "@timestamp" => "2016-10-19T23:14:19.000Z",
              "host" => "10.0.0.7",
          "priority" => 14,
     "timestamp8601" => "2016-10-19T23:14:19Z",
         "logsource" => "c51c177bd308",
           "program" => "registry.1.abszmuwq8k3d7comu504lz2mc",
               "pid" => "4833",
          "severity" => 6,
          "facility" => 1,
         "timestamp" => "2016-10-19T23:14:19Z",
    "facility_label" => "user-level",
    "severity_label" => "Informational"
}

As before when we tested LogStash input with logger, we have the message, timestamp, host, and a few other syslog fields. We also got logsource that holds the ID of the container that produced the log as well as program that holds the container name. Both will be useful when debugging which service and container produced a bug.

If you go back to the command we used to create the logstash service, you’ll notice the environment variable LOGSPOUT=ignore. It tells LogSpout that the service or, to be more precise, all containers that form the service, should be ignored. If we did not define it, LogSpout would forward all logstash logs to logstash thus creating an infinite loop. As we already discussed, in production we should not output LogStash entries to stdout. We did it only to get a better understanding how it works. If stdout output is removed from the logstash configuration, there would be no need for the environment variable LOGSPOUT=ignore. As a result logstash logs would also be stored in ElasticSearch.

Now that we are shipping all the logs to LogStash and from there to ElasticSearch, we should explore the ways to consult them. You’ll find them in the kibana service.

open http://$(docker-machine ip swarm-1)/app/kibana

Before leaving, please make sure to remove the machines we created and free your resources for some other tasks.

docker-machine rm -f swarm-1 swarm-2 swarm-3

The article you just finished reading is an extract from the Defining Logging Strategy chapter of The DevOps 2.1 Toolkit: Docker Swarm book.

The DevOps 2.1 Toolkit: Docker Swarm

The DevOps 2.1 Toolkit: Docker SwarmIf you liked this article, you might be interested in The DevOps 2.1 Toolkit: Docker Swarm book. Unlike the previous title in the series (The DevOps 2.0 Toolkit: Automating the Continuous Deployment Pipeline with Containerized Microservices) that provided a general overlook of some of the latest DevOps practices and tools, this book is dedicated entirely to Docker Swarm and the processes and tools we might need to build, test, deploy, and monitor services running inside a cluster.

You can get a copy from Amazon.com (and the other worldwide sites) or LeanPub. It is also available as The DevOps Toolkit Series bundle.

Give the book a try and let me know what you think.

Advertisements

6 thoughts on “Forwarding Logs From All Containers Running Anywhere Inside A Docker Swarm Cluster

  1. Thomas

    Thanks a bunch for this very helpful article Victor! I was wondering if you have any experience w. a plugin or add-on that can provide an authentication layer on top of the Kibana dashboard in Docker Swarm Mode? Thanks in advance

    Reply
  2. Viktor Farcic Post author

    Kibana itself does not support authentication. From the top of my head, I can imagine two possible solutions.

    1) Use x-pack (also from Elastic). It consists of different modules that allow you to create alerts, monitoring, reporting, graphs, and security. The security module (formerly known as Shield), you can, among other things, secure Kibana as well as ElasticSearch. The downside is that it is not OSS so you’ll have to purchase Elastic’s subscription model.

    2) Implement authentication inside the proxy. If Kibana does not expose any ports and all communication goes through a proxy, it should not be hard to add user/pass. It’s not ideal since authentication through a proxy usually means that you can have only plain credentials and cannot use things like Active Directory or OpenID.

    Reply
  3. Pingback: Collecting Metrics and Monitoring The Cluster | Technology Conversations

  4. Leo

    Hi,

    How you can get the physical hostname from log on logstash and send it elasticsearch?
    Can you share your logstash pipeline configuration?

    Regards

    Reply
    1. Viktor Farcic Post author

      That depends on the type of networking you’re using. With Overlay network, IP of a node and host name are lost. Instead, you have IP assigned to each replica. If you want to preserve original IP and//or hostname, you’d need to use host network instead. However, in my opinion, the advantages of overlay network overweight the disadvantages (e.g. loss of the original IP and/or hostname). The logic behind schedulers like Swarm is that everything is dynamic. It decides where something runs and moves replicas around the cluster depending on resource allocations, failures, and so on. In such a setting, hostname becomes irrelevant since the entity we operate is a cluster, not individual servers.

      Reply
  5. Pingback: Why Docker makes sense for startups – Bit Treat

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s