Using Docker Stack And Compose YAML Files To Deploy Swarm Services

Bells are ringing! Docker v1.13 is out!

The most common question I receive during my Docker-related talks and workshops is usually related to Swarm and Compose.

Someone: How can I use Docker Compose with Docker Swarm?

Me: You can’t! You can convert your Compose files into a Bundle that does not support all Swarm features. If you want to use Swarm to its fullest, be prepared for docker service create commands that contain a never ending list of arguments.

Such an answer was usually followed with disappointment. Docker Compose showed us the advantages of specifying everything in a YAML file as opposed to trying to remember all the arguments we have to pass to docker commands. It allowed us to store service definitions in a repository thus providing a reproducible and well-documented process for managing them. Docker Compose replaced bash scripts, and we loved it. Then, Docker v1.12 came along and put a difficult choice in front of us. Should we adopt Swarm and discard Compose? Since summer 2016, Swarm and Compose were not in love anymore. It was a painful divorce.

But, after almost half a year of separation, they are back together, and we can witness their second honeymoon. Kind of… We do not need Docker Compose binary for Swarm services, but we can use its YAML files.

Docker Engine v1.13 introduced support for Compose YAML files within the stack command. At the same time, Docker Compose v1.10 introduced a new version 3 of its format. Together, they allow us to manage our Swarm services using already familiar Docker Compose YAML format.

I will assume you are already familiar with Docker Compose and won’t go into details of everything we can do with it. Instead, we’ll go through an example of creating a few Swarm services.

We’ll explore how to create Docker Flow Proxy service through Docker Compose files and the docker stack deploy command.

Requirements

The examples that follow assume that you are using Docker v1.13+, Docker Compose v1.10+, and Docker Machine v0.9+.

If you are a Windows user, please run all the examples from Git Bash (installed through Docker Toolbox). Also, make sure that your Git client is configured to check out the code AS-IS. Otherwise, Windows might change carriage returns to the Windows format.

Swarm Cluster Setup

To setup an example Swarm cluster using Docker Machine, please run the commands that follow.

Feel free to skip this section if you already have a working Swarm cluster.

curl -o swarm-cluster.sh \
    https://raw.githubusercontent.com/vfarcic/docker-flow-proxy/master/scripts/swarm-cluster.sh

chmod +x swarm-cluster.sh

./swarm-cluster.sh

docker-machine ssh node-1

Now we’re ready to deploy the docker-flow-proxy service.

Creating Swarm Services Through Docker Stack Commands

We’ll start by creating a network.

docker network create --driver overlay proxy

The proxy network will be dedicated to the proxy container and services that will be attached to it.

We’ll use docker-compose-stack.yml from the vfarcic/docker-flow-proxy repository to create docker-flow-proxy and docker-flow-swarm-listener services.

The content of the docker-compose-stack.yml file is as follows.

version: "3"

services:

  proxy:
    image: vfarcic/docker-flow-proxy
    ports:
      - 80:80
      - 443:443
    networks:
      - proxy
    environment:
      - LISTENER_ADDRESS=swarm-listener
      - MODE=swarm
    deploy:
      replicas: 2

  swarm-listener:
    image: vfarcic/docker-flow-swarm-listener
    networks:
      - proxy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - DF_NOTIFY_CREATE_SERVICE_URL=http://proxy:8080/v1/docker-flow-proxy/reconfigure
      - DF_NOTIFY_REMOVE_SERVICE_URL=http://proxy:8080/v1/docker-flow-proxy/remove
    deploy:
      placement:
        constraints: [node.role == manager]

networks:
  proxy:
    external: true

The format is written in version 3 (mandatory for docker stack deploy).

It contains two services; proxy and swarm-listener. Since this article is not meant to teach you how to use the proxy, I won’t go into the meaning of each argument.

When compared with previous Compose versions, most of the new arguments are defined within deploy. You can think of that section as a placeholder for Swarm-specific arguments. In this case, we are specifying that the proxy service should have two replicas while the swarm-listener service should be constrained to manager roles. Everything else defined for those two services is using the same format as in earlier Compose versions.

At the bottom of the YAML file is the list of networks which are referenced within services. If a service does not specify any, the default network will be created automatically. In this case, we opted for manual creation of a network since services from other stacks should be able to communicate with the proxy. Therefore, we created a network manually and defined it as external in the YAML file.

Let’s create the stack based on the YAML file we explored.

curl -o docker-compose-stack.yml \
    https://raw.githubusercontent.com/vfarcic/docker-flow-proxy/master/docker-compose-stack.yml

docker stack deploy -c docker-compose-stack.yml proxy

The first command downloaded the Compose file docker-compose-stack.yml from the vfarcic/docker-flow-proxy repository. The second command created the services that form the stack.

The tasks of the stack can be seen through the stack ps command.

docker stack ps proxy

The output is as follows (IDs are removed for brevity).

NAME                   IMAGE                                     NODE   DESIRED STATE CURRENT STATE         ERROR  PORTS
proxy_proxy.1          vfarcic/docker-flow-proxy:latest          node-2 Running       Running 2 minutes ago
proxy_swarm-listener.1 vfarcic/docker-flow-swarm-listener:latest node-1 Running       Running 2 minutes ago
proxy_proxy.2          vfarcic/docker-flow-proxy:latest          node-3 Running       Running 2 minutes ago

We are running two replicas of the proxy (for high-availability in the case of a failure) and one of the swarm-listener.

Deploying More Stacks

Let’s deploy another stack.

This time we’ll use Docker stack defined in the Compose file docker-compose-stack.yml located in the vfarcic/go-demo repository. It is as follows.

version: '3'

services:

  main:
    image: vfarcic/go-demo
    environment:
      - DB=db
    networks:
      - proxy
      - default
    deploy:
      replicas: 3
      labels:
        - com.df.notify=true
        - com.df.distribute=true
        - com.df.servicePath=/demo
        - com.df.port=8080

  db:
    image: mongo
    networks:
      - default

networks:
  default:
    external: false
  proxy:
    external: true

The stack defines two services (main and db). They will communicate with each other through the default network that will be created automatically by the stack (no need for docker network create command). Since the main service is an API, it should be accessible through the proxy, so we’re attaching proxy network as well.

The important thing to note is that we used the deploy section to define Swarm-specific arguments. In this case, the main service defines that there should be three replicas and a few labels. As with the previous stack, we won’t go into details of each service. If you’d like to go into more depth of the labels used with the main service, please visit the Running Docker Flow Proxy In Swarm Mode With Automatic Reconfiguration tutorial.

Let’s deploy the stack.

curl -o docker-compose-go-demo.yml \
    https://raw.githubusercontent.com/vfarcic/go-demo/master/docker-compose-stack.yml

docker stack deploy \
    -c docker-compose-go-demo.yml go-demo

docker stack ps go-demo

We downloaded the stack definition, executed stack deploy command that created the services and run the stack ps command that lists the tasks that belong to the go-demo stack. The output is as follows (IDs are removed for brevity).

NAME           IMAGE                  NODE    DESIRED STATE CURRENT STATE          ERROR PORTS
go-demo_main.1 vfarcic/go-demo:latest node-2 Running        Running 7 seconds ago
...
go-demo_db.1   mongo:latest           node-2 Running        Running 21 seconds ago
go-demo_main.2 vfarcic/go-demo:latest node-2 Running        Running 19 seconds ago
...
go-demo_main.3 vfarcic/go-demo:latest node-2 Running        Running 20 seconds ago
...

Since Mongo database is much bigger than the main service, it takes more time to pull it, resulting in a few failures. The go-demo service is designed to fail if it cannot connect to its database. Once the db service is running, the main service should stop failing, and we’ll see three replicas with the current state Running.

After a few moments, the swarm-listener service will detect the main service from the go-demo stack and send the proxy a request to reconfigure itself. We can see the result by sending an HTTP request to the proxy.

curl -i "localhost/demo/hello"

The output is as follows.

HTTP/1.1 200 OK
Date: Thu, 19 Jan 2017 23:57:05 GMT
Content-Length: 14
Content-Type: text/plain; charset=utf-8

hello, world!

The proxy was reconfigured and forwards all requests with the base path /demo to the main service from the go-demo stack.

For more advanced usage of the proxy, please see the examples from Running Docker Flow Proxy In Swarm Mode With Automatic Reconfiguration tutorial or consult the configuration and usage documentation.

To Stack Or Not To Stack

Docker stack is a great addition to the Swarm Mode. We do not have to deal with docker service create commands that tend to have a never ending list of arguments. With services specified in Compose YAML files, we can replace those long commands with a simple docker stack deploy. If those YAML files are stored in code repositories, we can apply the same practices to service deployments as to any other area of software engineering. We can track changes, do code reviews, share with others, and so on.

The addition of the Docker stack command and its ability to use Compose files is a very welcome addition to the Docker ecosystem.

Cleanup

Please remove Docker Machine VMs we created. You might need those resources for some other tasks.

exit

docker-machine rm -f node-1 node-2 node-3

The DevOps 2.1 Toolkit: Docker Swarm

The DevOps 2.1 Toolkit: Docker SwarmIf you liked this article, you might be interested in The DevOps 2.1 Toolkit: Docker Swarm book. Unlike the previous title in the series (The DevOps 2.0 Toolkit: Automating the Continuous Deployment Pipeline with Containerized Microservices) that provided a general overlook of some of the latest DevOps practices and tools, this book is dedicated entirely to Docker Swarm and the processes and tools we might need to build, test, deploy, and monitor services running inside a cluster.

You can get a copy from Amazon.com (and the other worldwide sites) or LeanPub. It is also available as The DevOps Toolkit Series bundle.

Give the book a try and let me know what you think.

Advertisements

19 thoughts on “Using Docker Stack And Compose YAML Files To Deploy Swarm Services

  1. katopz

    This is for single host am I right? What it’s look like for multiple DigitalOcean’s droplets? I just can’t find any document for that.

    Reply
      1. katopz

        Hey Vikor, This example work quite well at Jan but after Docker 1.13.1 I got

        $ docker stack deploy -c docker-compose-go-demo.yml go-demo
        Error response from daemon: network go-demo_default not found

        $ docker stack ps proxy
        ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
        lk2oyaqbddhg proxy_proxy.1 vfarcic/docker-flow-proxy:latest node-2 Running Running 51 minutes ago
        s9r68fuhlpxr proxy_swarm-listener.1 vfarcic/docker-flow-swarm-listener:latest node-1 Running Running 52 minutes ago
        l99v7d23i0pt proxy_proxy.2 vfarcic/docker-flow-proxy:latest node-3 Running Running 51 minutes ago

        $ docker stack ps go-demo
        Nothing found in stack: go-demo

        And I also got “Docker Flow Proxy: 503 Service Unavailable” when curl -i “localhost/demo/hello”

        Can you confirm this example is still work?

        Reply
        1. Viktor Farcic Post author

          I just rerun the commands from this article and everything work.

          I did notice a similar error in the past. I think it was somehow related to service discovery not being fast enough to pick it up. Those few times I just rerun it and it worked. Cannot guarantee that the problem you’re experiencing is the same 😦

          If creating the stack did not work (you can see it with docker stack ps go-demo), the services are not created and sending a request to them (curl) will fail as well.

          Can you try it one more time (maybe with a fresh Swarm cluster)?

          Reply
          1. katopz (@katopz)

            Thanks for confirmation, I can make it work with another Mac now by just remove and clean everything. For fresh swarm, the first attempt will fail (like you said) but second attempt will succeed.

            I’ll try again with other Mac to see the problems and will let you know. Thanks!

            Reply
  2. Damien Lust

    Thanks for the article , I applied it on my prototype based on docker 1.12 , quite easy to use this docker stack command. Some remarks:
    – a service name is automatically created based on the stack name and the service defined in the yml, it lets the possibility to get the status of individual service via the command docker service ps go-demo_main. Some governance about stack name must be defined : if it’s a new stack name but with the same service description it will create separated services, if it’s the same stack name with same service , an update will done for each service.
    – it simplify the deployment , I see now 3 main steps :
    – network creation (perhaps it will be part of the yml compose in the future?)
    – deploy of the stack
    – curl command to update the proxy

    Reply
  3. Viktor Farcic Post author

    a service name is automatically created based on the stack name and the service defined in the yml, it lets the possibility to get the status of individual service via the command docker service ps go-demo_main.

    Once you get used to working with stacks, you’ll probably use “docker service stack ps [STACK_NAME]” commands much more than “docker service ps…”. Going back to your question, you’re right. Services are names with a combination of the name of the stack and the name of the service separated with underscore.

    network creation (perhaps it will be part of the yml compose in the future?)

    I tend to create networks manually only when they span multiple stacks. I do that only because it’s more convenient. You could always, use network from one stack inside the other. For example, https://github.com/vfarcic/docker-flow-stacks/blob/master/logging/logging-df-proxy.yml stack will automatically create a network _default. If the stack name is logging, the network will be called logging_default. Now, if you take a look at https://github.com/vfarcic/docker-flow-stacks/blob/master/metrics/prometheus-grafana-df-proxy.yml, you’ll see that it uses the logging_default network created with the first stack.

    The naming convention is the same as in the previous Compose versions. It’s _. In case of stacks, PROJECT_NAME is STACK_NAME and SOMETHING is the name of a service, network, volumes, and so on. In other words, whatever you specify is created with the name prefixed with the name of the stack and underscore.

    curl command to update the proxy

    There is no need for curl command. Docker Flow Swarm Listener will update the proxy for you. Just make sure that services that should be configured in the proxy have the labels.

    Reply
  4. Andy

    I agree with katopz. The examples & documentation around real production use don’t exist, e.g.
    1. Managing different environments, and configuration-based compose.yml through environment variable substitution.
    2. Real deployment scenarios in a modern microservices architecture: I want to deploy and update 1 thing in my stack while everything continues running happily.
    3. In relation to #2, strategies and rollout policies to minimise or negate disruption.
    4. Server roles and container placement.
    5. Recommendations for configuration management, secret storage.
    etc etc etc.

    The individual pieces for these points do exist, I guess what I’m asking if for leaders of the community to provide more complete examples based upon real & battle hardened use.

    Reply
    1. Viktor Farcic Post author
      1. I think that in most cases we do not need to have different environment variables on different systems. Exception would be release number, CPU, and memory. The more we differ from one environment to another, the less benefits we’re getting from immutability containers give us.
      2. I’m not sure I understand the challenge with that. Deploying one thing (one service) means that all you have to do it modify that service. Docker will make sure not to touch the rest if their specification is the same as what’s currently running.

      3. I think that depends from one case to another. Not everyone shares the same architecture. I do not believe in rules that everyone should follow. Our job is to understand the tools we can use and make decisions based on individual use cases and circumstances. For example, strategy to deploy a microservice written in Go and using MongoDB is not the same as dealing with applications that use shared database.

      4. I’m not sure what you mean by that.

      5. What type of configuration management you have in mind? It depends on whether you fully adopted Docker or not. Whether you’re running in cloud or on-premise. And so on and so forth. Secrets storage was released but still not widely announced. You’ll hear from Docker about it soon. Please note that it is in its infancy. If you’d like something mature now, I’d recommend HashiCorp Vault.

      The individual pieces for these points do exist, I guess what I’m asking if for leaders of the community to provide more complete examples based upon real & battle hardened use.

      If this sentence refers to Docker Stack, that would be very hard to do only a few days after it was released. Some of use played with it while in beta. Many of us will start using it now. If you are an early adopter, you are part of the community that is trying new things and, hopefully, contributing back. If you’re expecting battle hardened examples, I think you’ll have to give us (community) a bit of time. That comes after a few months of usage (if not more). Doesn’t it?

      Reply
  5. Krys

    Can you use docker stack deploy‘s ‘–compose-file’ option multiple times like you could with docker-compose -f ?

    Reply
    1. Viktor Farcic Post author

      Yes. In that aspect, it works in a similar way as Docker Compose. It’s idempotent. You can run it as many times as you like and the result will always be the same.

      Reply
  6. Evan

    How would you achieve volumes synchronization between swarm nodes, so that the service switch from one now to another are consistently working?

    Reply
    1. Viktor Farcic Post author

      You should combine network drives with one of Docker volume plugins. My favorite is REX-Ray. If you don’t want to use plugins, a simple NFS directory mounted on all nodes should do.

      Reply
      1. Evan

        Without a network drive, is there any recommended way?

        I have tried the glusterfs, but fail to probe the nodes.

        Sample: two VMs with two virtual disk each – one for OS and one for synchronization.

        Reply
  7. Pingback: Docker: How To Get Started With Containers in Ubuntu – Sajjan's Blog

  8. Pingback: Docker: How To Get Started With Containers in Ubuntu - Sajjan's Blog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s