Centralized System and Docker Logging with ELK Stack

With Docker there was not supposed to be a need to store logs in files. We should output information to stdout/stderr and the rest will be taken care by Docker itself. When we need to inspect logs all we are supposed to do is run docker logs [CONTAINER_NAME].

With Docker and ever more popular usage of micro services, number of deployed containers is increasing rapidly. Monitoring logs for each container separately quickly becomes a nightmare. Monitoring few or even ten containers individually is not hard. When that number starts moving towards tens or hundreds, individual logging is unpractical at best. If we add distributed services the situation gets even worst. Not only that we have many containers but they are distributed across many servers.

The solution is to use some kind of centralized logging. Our favourite combination is ELK stack (ElasticSearch, LogStash and Kibana). However, centralized logging with Docker on large-scale was not a trivial thing to do (until version 1.6 was released). We had a couple of solutions but none of them seemed good enough.
Continue reading

Build Automation Panel

Last Tuesday, I participated in an online panel on the subject of Build Automation as part of Continuous Discussions (#c9d9), a series of community panels about Agile, Continuous Delivery and DevOps. Automating the build pipeline has many challenges, including third-party dependencies, build version management and especially culture, and panelists discussed real-life experiences addressing these challenges.
Continue reading

Continuous Integration, Delivery or Deployment with Jenkins, Docker and Ansible

This article tries to provide one possible way to set up the Continuous Integration, Delivery or Deployment pipeline. We’ll use Jenkins, Docker, Ansible and Vagrant to set up two servers. One will be used as a Jenkins server and the other one as an imitation of production servers. First one will checkout, test and build applications while perform deployment and post-deployment tests.
Continue reading

Microservices Development with Scala, Spray, MongoDB, Docker and Ansible

This article tries to provide one possible approach to building microservices. We’ll use Scala as programming language. API will be RESTful JSON provided by Spray and Akka. MongoDB will be used as database. Once everything is done we’ll pack it all into a Docker container. Vagrant with Ansible will take care of our environment and configuration management needs.

We’ll do the books service. It should be able to do following:

  • List all books
  • Retrieve all the information related to a book
  • Update an existing book
  • Delete an existing book
    Continue reading

Monolithic Servers vs Microservices

Introduction

At the beginning applications were simple and small due to simple requirements. With time requirements and needs grew and with them our applications became bigger and more complex. That resulted in monolithic servers developed and deployed as a single unit. Microservices are, in a way, return to basics with simple applications that are fulfilling today’s needs for complexity by working together through utilization of each others APIs.
Continue reading

Continuous Deployment: Implementation with Ansible and Docker

This article is part of the Continuous Integration, Delivery and Deployment series.

The previous article described several ways to implement Continuous Deployment. Specifically, it described, among other things, how to implement it using Docker to deploy applications as containers and nginx for reverse proxy necessary for successful utilization of blue-green deployment technique. All that was running on top of CoreOS, operating system specifically designed for running Docker containers.

In this article we’ll try to do the same process using Ansible (an open-source platform for configuring and managing computers). Instead of CoreOS, we’ll be using Ubuntu.

Source code used in this article can be found in the GitHub repo vfarcic/provisioning (directory ansible).
Continue reading

Continuous Deployment: Implementation

This article is part of the Continuous Integration, Delivery and Deployment series.

The previous article described several Continuous Deployment strategies. In this one we will attempt to provide one possible solution for reliable, fast and automatic continuous deployment with ability to test new releases before they become available to general users. If something goes wrong we should be able to rollback back easily. On top of that, we’ll try to accomplish zero-downtime. No matter how many times we deploy our applications, there should never be a single moment when they are not operational.
Continue reading