The DevOps 2.0 Toolkit

Today is an exciting day for me. I just decided that the book I spent the last eight months writing is ready for general public.

What made me write the book? Certainly not the promise of wealth since, as any author of technical books will confirm, there is no money that can compensate the number of hours involved in writing a technical book. The reasons behind this endeavor are of a different nature. I realized that this blog is a great way for me to explore different subjects and share my experience with the community. However, due to the format, blog posts do not give enough space to explore, in more details, subjects I’m passionate about so, around eight months ago, I decided to start working on The DevOps 2.0 Toolkit: Automating the Continuous Deployment Pipeline with Containerized Microservices book. It treats similar subjects as those I write about in this blog, but with much more details. More importantly, the book allowed me to organize my experience into a much more coherent story.

The book is about different techniques that help us architect software in a better and more efficient way with microservices packed as immutable containers, tested and deployed continuously to servers that are automatically provisioned with configuration management tools. It’s about fast, reliable and continuous deployments with zero-downtime and ability to roll-back. It’s about scaling to any number of servers, designing self-healing systems capable of recuperation from both hardware and software failures and about centralized logging and monitoring of the cluster.

In other words, this book envelops the whole microservices development and deployment lifecycle using some of the latest and greatest practices and tools. We’ll use Docker, Kubernetes, Ansible, Ubuntu, Docker Swarm and Docker Compose, Consul, etcd, Registrator, confd, Jenkins, and so on. We’ll go through many practices and, even more, tools.

At this moment, around 70% is finished and you’ll receive regular updates if you decide to purchase the book. The truth is that my motivation for writing the book is the same as with this blog. I like sharing my experience and this book is one more way to accomplish that. You can set your own price and if you feel that the minimum amount is still too high, please send me a private message and I’ll get back to you with a free copy.

Please give The DevOps 2.0 Toolkit: Automating the Continuous Deployment Pipeline with Containerized Microservices a try and let me know what you think. Any feedback is welcome and appreciated.

9 thoughts on “The DevOps 2.0 Toolkit

  1. Michael Matskevich

    Hi! I’m trying to follow the book tutorial and really stuck with Page 60-61 where I should for a first time to execute tests. Here is my output:

    vagrant@vagrant-ubuntu-trusty-64:/vagrant$ sudo docker run -it –rm -v $PWD/client/components:/source/client/components -v $PWD/client/test:/source/client/test -v $PWD/src:/source/src -v $PWD/target:/source/target -p 8080:8080 –env TEST_TYPE=watch-front vfarcic/books-ms-tests

    NPM is broken (again) so front-end tests are temporarily disabled.
    mongod –help for help and startup options
    mongod –help for help and startup options
    Wed Jul 13 19:48:46.596 [initandlisten] MongoDB starting : pid=7 port=27017 dbpath=/data/db/ 64-bit host=b0c7caea7c9e
    Wed Jul 13 19:48:46.599 [initandlisten] db version v2.4.10
    Wed Jul 13 19:48:46.599 [initandlisten] git version: nogitversion
    Wed Jul 13 19:48:46.599 [initandlisten] build info: Linux julia 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt2-1 (2014-12-08) x86_64 BOOST_LIB_VERSION=1_55
    Wed Jul 13 19:48:46.599 [initandlisten] allocator: tcmalloc
    Wed Jul 13 19:48:46.601 [initandlisten] options: {}
    Wed Jul 13 19:48:46.603 [initandlisten] journal dir=/data/db/journal
    Wed Jul 13 19:48:46.603 [initandlisten] recover : no journal files present, no recovery needed
    Wed Jul 13 19:48:46.632 [initandlisten] MongoDB starting : pid=5 port=27017 dbpath=/data/db/ 64-bit host=b0c7caea7c9e
    Wed Jul 13 19:48:46.635 [initandlisten] db version v2.4.10
    Wed Jul 13 19:48:46.636 [initandlisten] git version: nogitversion
    Wed Jul 13 19:48:46.637 [initandlisten] build info: Linux julia 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt2-1 (2014-12-08) x86_64 BOOST_LIB_VERSION=1_55
    Wed Jul 13 19:48:46.637 [initandlisten] allocator: tcmalloc
    Wed Jul 13 19:48:46.639 [initandlisten] options: {}
    Wed Jul 13 19:48:46.643 [initandlisten] exception in initAndListen: 10310 Unable to lock file: /data/db/mongod.lock. Is a mongod instance already running?, terminating
    Wed Jul 13 19:48:46.644 dbexit:
    Wed Jul 13 19:48:46.645 [initandlisten] shutdown: going to close listening sockets…
    Wed Jul 13 19:48:46.646 [initandlisten] shutdown: going to flush diaglog…
    Wed Jul 13 19:48:46.646 [initandlisten] shutdown: going to close sockets…
    Wed Jul 13 19:48:46.647 [initandlisten] shutdown: waiting for fs preallocator…
    Wed Jul 13 19:48:46.648 [initandlisten] shutdown: lock for final commit…
    Wed Jul 13 19:48:46.648 [initandlisten] shutdown: final commit…
    Wed Jul 13 19:48:46.649 [initandlisten] shutdown: closing all files…
    Wed Jul 13 19:48:46.650 [initandlisten] closeAllFiles() finished
    Wed Jul 13 19:48:46.651 dbexit: really exiting now
    Wed Jul 13 19:48:46.659 [FileAllocator] allocating new datafile /data/db/local.ns, filling with zeroes…
    Wed Jul 13 19:48:46.660 [FileAllocator] creating directory /data/db/_tmp
    Wed Jul 13 19:48:46.662 [FileAllocator] done allocating datafile /data/db/local.ns, size: 16MB, took 0 secs
    Wed Jul 13 19:48:46.670 [FileAllocator] allocating new datafile /data/db/local.0, filling with zeroes…
    Wed Jul 13 19:48:46.672 [FileAllocator] done allocating datafile /data/db/local.0, size: 64MB, took 0 secs
    Wed Jul 13 19:48:46.673 [initandlisten] waiting for connections on port 27017
    Wed Jul 13 19:48:46.674 [websvr] admin web console waiting for connections on port 28017
    [info] Loading project definition from /source/project
    [info] Set current project to books-ms (in build file:/source/)
    [info] Compiling 2 Scala sources to /source/target/scala-2.10/classes…
    [info] ‘compiler-interface’ not yet compiled for Scala 2.10.6. Compiling…
    [info] Compilation completed in 22.087 s
    [info] Running com.technologyconversations.api.Service
    19:49:48.804 [routingSystem-akka.actor.default-dispatcher-4] DEBUG c.m.c.c.c.s.RegisterConversionHelpers$ – Registering Scala Conversions.
    19:49:48.815 [routingSystem-akka.actor.default-dispatcher-4] DEBUG c.m.c.c.c.s.RegisterConversionHelpers$ – Deserializers for Scala Conversions registering
    19:49:48.815 [routingSystem-akka.actor.default-dispatcher-4] DEBUG c.m.c.c.c.s.RegisterConversionHelpers$ – Serializers for Scala Conversions registering
    19:49:48.817 [routingSystem-akka.actor.default-dispatcher-4] DEBUG c.m.c.c.c.s.RegisterConversionHelpers$ – Setting up OptionSerializer
    19:49:48.824 [routingSystem-akka.actor.default-dispatcher-4] DEBUG c.m.c.c.c.s.RegisterConversionHelpers$ – Setting up ScalaProductSerializer
    19:49:48.826 [routingSystem-akka.actor.default-dispatcher-4] DEBUG c.m.c.c.c.s.RegisterConversionHelpers$ – Setting up ScalaCollectionSerializer
    19:49:48.829 [routingSystem-akka.actor.default-dispatcher-4] DEBUG c.m.c.c.c.s.RegisterConversionHelpers$ – Setting up ScalaRegexSerializers
    19:49:48.831 [routingSystem-akka.actor.default-dispatcher-4] DEBUG c.m.c.c.c.s.RegisterConversionHelpers$ – Hooking up scala.util.matching.Regex serializer
    19:49:48.835 [routingSystem-akka.actor.default-dispatcher-4] DEBUG c.m.c.c.c.s.RegisterConversionHelpers$ – Reached base registration method on MongoConversionHelper.
    Wed Jul 13 19:49:49.695 [initandlisten] connection accepted from 127.0.0.1:38988 #1 (1 connection now open)
    [INFO] [07/13/2016 19:49:50.045] [routingSystem-akka.actor.default-dispatcher-3] [akka://routingSystem/user/IO-HTTP/listener-0] Bound to /0.0.0.0:8080
    [INFO] [07/13/2016 19:49:50.051] [routingSystem-akka.actor.default-dispatcher-4] [akka://routingSystem/deadLetters] Message [akka.io.Tcp$Bound] from Actor[akka://routingSystem/user/IO-HTTP/listener-0#1397891951] to Actor[akka://routingSystem/deadLetters] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings ‘akka.log-dead-letters’ and ‘akka.log-dead-letters-during-shutdown’.

    There is no firefox processes and there is deadletter heppen, so everything is stuck ;(

    What can it be?

    Reply
  2. Viktor Farcic Post author

    One of the NPM dependencies got broken some time ago so I disabled tests and updated the book (electronic version) explaining what happened. For the time being, it does not run tests. During this weekend, I’ll try it out and, if it’s fixed, put it back as it was. Until then, it should not prevent you from continuing with the rest of the book.

    I assume you have paperback version. If that’s so, can you please send me an email to viktor@farcic.com and I’ll send you a coupon for a free eBook that gets updated every time there is a change in content or the code.

    Reply
  3. mihadeth

    So as I follow the book (with latest from github):

    FAIL with deadletter:
    – sudo docker run -it –rm \
    -v $PWD/client/components:/source/client/components \
    -v $PWD/client/test:/source/client/test \
    -v $PWD/src:/source/src \
    -v $PWD/target:/source/target \
    -p 8080:8080 \
    –env TEST_TYPE=watch-front \
    vfarcic/books-ms-tests

    FAIL with deadletter:
    sudo docker-compose -f docker-compose-dev.yml run feTestsLocal

    Runs till completion – SUCCESS:
    sudo docker-compose -f docker-compose-dev.yml run testsLocal

    Reply
  4. mihadeth

    Thank you for reply!
    I purchased your work as eBook, sorry if I’m using outdated copy – will refresh it now.

    You did a great one – it is one of my favorite books right now. Thanks again for support

    Reply
    1. Viktor Farcic Post author

      There’s no need to apologise. It was my mistake for using that code as an example. Front-end tests are too flaky and I should have known better and do something different. However, now it is distributed among too many readers to make drastic changes to the code. The good news is that there are only three types of complaints I got from readers: front-end tests (NPM failing every once in a while), Akka warning messages that look like errors, and problems to download things if you live in China (many addresses are blocked). You already hit two out of three. Assuming that you don’t live in China, the rest should be smooth sailing.

      Reply
      1. mihadeth

        I don’t reside in China, but still had a lot of troubles with proxy server at work. Managed to set a proxy for Vagrant in Vagrantfile, so I could continue following the book. Unfortunately, when behind the proxy, scala compiler cannot build the code due to unsatisfied dependencies, it seems like Docker cannot pass proxy configuration further. To be sure it is a proxy thing, I made the same at home and managed to pass scala compilation and get to the point I wrote you in the first reply.
        Yes I understand that deadletters are not failure, but in my case, the command is stuck. Deadletter message is the last one printed, even after 20 minutes nothing happens.

        I’m curious to ask you regarding your machine and network setup at work. Do you work behind proxy? And what are the best practices to overcome the problems with it in development of microservices?

        Thanks again,
        Misha

        Reply
        1. Viktor Farcic Post author

          Hi,

          I’m sorry to hear about the problems with company’s firewall. Those situations tend to be very frustrating and time-consuming.

          It’s OK that the command is stuck. Even though I removed tests from execution, that command is still in watch mode (or a simulation of it). Originally, the idea was to enter an endless loop and run front-end tests every time the source code changes. That way, you don’t need to re-run the command all the time (especially if you’re practicing TDD). Later on in the book (after that command), there is the instruction to press CTRL+c to stop the loop.

          I work with many different clients and the situation can be anything from no firewall/proxy to you’ll-never-do-anything-useful-here type of setup. In some situations, I would work partly from customer’s office and partly from home just to avoid to much hassle with their proxy.

          I don’t think that corporate proxy problems are related to microservices development. It would affect you (more or less) equally no matter whether you’re working on monoliths or microservices. In both cases, it leads to too much time wasted on tasks that do not bring much value. I’m very much against any company intention to limit creativity and freedom and that’s what proxies do. I don’t see a problem securing production environment but see no reason for doing that to employees. On the other hand, we don’t tend to have a say in those matters so we have to live with such policies.

          Reply
          1. mihadeth

            Agree with you 100%. I asked for separate ADSL line, so hopefully I will be free to code without banging my head against the firewall.

            I will continue reading this great book and try to create something equally great. Thank you so much for your help!

            Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s