Jenkins X itself is serverless. That helps with many things, with better resource utilization and scalability being only a few of the benefits. Can we do something similar with our applications? Can we scale them to zero when no one is using them? Can we scale them up when the number of concurrent requests increases? Can we make our applications serverless?
If you'd like to follow the examples, I will assume that you already have a cluster with serverless (Tekton-based) Jenkins X up-and-running.
Before we start exploring how to override different components in serverless Jenkins X pipelines, we'll create a new quickstart project so that we have a sample application to play with.
jx create quickstart \
--language go \
--project-name jx-go-loops \
--batch-modeHopefully, this is not the first time you created a quick start project, and you are already familiar with the out-of-the-box pipeline our new application inherited from a build pack. Also, I will assume that you do understand that
buildPack: goinstruction injenkins-x.ymlmeans that the pipeline inherits all the steps defined in the corresponding build pack.
Our pipeline is currently building a Linux binary of our application before adding it to a container image. But what if we'd like to distribute the application also as executables for different operating systems? We could provide that same binary, but that would work only for Linux users since that is the architecture it is currently built for. We might want to extend the reach to Windows and MacOS users as well, and that would mean that we'd need to build two additional binaries. How could we do that?
Jenkins X main logic is based on applying GitOps principles. Every change must be recorded in Git, and only Git is allowed to initiate events that result in changes in our clusters. That logic is the cornerstone of Jenkins X, and it served us well so far. However, there are actions we might need to perform that do not result in changes to the source code or configurations. Hence the emergence of ChatOps.
The serverless flavor of Jenkins X or, as some call it, Jenkins X Next Generation, is an attempt to redefine how we do continuous delivery and GitOps inside Kubernetes clusters. It does that by combining quite a few tools into a single easy-to-use bundle. As a result, most people will not have a need to understand intricacies of how the pieces work independently, nor how they are all integrated. Instead, many will merely push a change to Git and let the system do the rest. But, there are always those who would like to know what's happening behind the hood. To satisfy those craving for insight, we'll explore the processes and the components involved in the serverless Jenkins X platform. Understanding the flow of an event initiated by a Git webhook will give us insight into how the solution works and help us later on when we go deeper into each of the new components.
Software development is hard. It takes years to become a proficient developer, and the tech and the processes change every so often. What was effective yesterday, is not necessarily effective today. The number of languages we code in is increasing. While in the past, most developers would work in the same language throughout their whole carrier, today it is not uncommon for a developer to work on multiple projects written in different languages. We might, for example, work on a new project and code in Go, while we still need to maintain some other project written in Java. For us to be efficient, we need to install compilers, helper libraries, and quite a few other things.
I stand by my claim that "you do not need to understand Kubernetes to use Jenkins X." To be more precise, those who do not want to know Kubernetes and its ecosystem in detail can benefit from Jenkins X ability to simplify the processes around software development lifecycle. That's the promise or, at least, one of the driving ideas behind the project. Nevertheless, for that goal to reach as wide of an audience as possible, we need a variety of build packs. The more we have, the more use cases can be covered with a single jx import or jx quickstart command. The problem is that there is an infinite number of types of applications and combinations we might have. Not all can be covered with community-based packs. No matter how much effort the community puts into creating build packs, they will always be a fraction of what we might need. That's where you come in.
If you're reading this, the chances are that you do not want to use jx cluster create to create a new cluster that will host Jenkins X. That is OK, or even welcome. That likely means that you are already experienced with Kubernetes and that you already have applications running in Kubernetes. That's a sign of maturity and your desire to add Jenkins X to the mix of whichever applications you are already running there. After all, it would be silly to create a new cluster for each set of applications.
However, using an existing Kubernetes cluster is risky. Many people think that they are so smart that they will assemble their Kubernetes cluster from scratch. "We're so awesome that we don't need tools like Rancher to create a cluster for us." "We'll do it with kubeadm." Then, after a lot of sweat, we announce that the cluster is operational, only to discover that there is no StorageClass or that networking does not work. So, if you assembled your own cluster and you want to use Jenkins X inside it, you need to ask yourself whether that cluster is set up correctly. Does it have everything we need? Does it comply with standards, or did you tweak it to meet your corporate restrictions? Did you choose to remove StorageClass because all your applications are stateless? Were you forced by your security department to restrict communication between Namespaces? Is the Kubernetes version too old? We can answer those and many other questions by running compliance tests.
Continue reading
When I finished the last book (The DevOps 2.5 Toolkit: Monitoring, Logging, and Auto-Scaling Kubernetes), I wanted to take a break from writing for a month or two. I thought that would clear my mind and help me decide which subject to tackle next. Those days were horrible. I could not make up my mind. So many cool and useful tech is emerging and being adopted. I was never as undecided as those weeks. Which should be my next step?
I could explore serverless. That's definitely useful, and it might be considered the next big thing. Or I could dive into Istio. It is probably the biggest and the most important project sitting on top of Kubernetes. Or I could tackle some smaller subjects. Kaniko is the missing link in continuous delivery. Building containers might be the only thing we still do on the host level, and Kaniko allows us to move that process inside containers. How about security scanning? It is one of the things that are mandatory in most organizations, and yet I did not include it in "The DevOps 2.4 Toolkit: Continuous Deployment To Kubernetes". Then there is skaffold, prow, KNative, and quite a few other tools that are becoming stable and very useful.
The DevOps 2.5 Toolkit: Monitoring, Logging, and Auto-Scaling Kubernetes is finally finished!!!
What do we do in Kubernetes after we master deployments and automate all the processes? We dive into monitoring, logging, auto-scaling, and other topics aimed at making our cluster resilient, self-sufficient, and self-adaptive.
Continue reading
There are quite a few candidates for your need for centralized logging. Which one should you choose? Will it be Papertrail, Elasticsearch-Fluentd-Kibana stack (EFK), AWS CloudWatch, GCP Stackdriver, Azure Log Analytics, or something else?
When possible and practical, I prefer a centralized logging solution provided as a service, instead of running it inside my clusters. Many things are easier when others are making sure that everything works. If we use Helm to install EFK, it might seem like an easy setup. However, maintenance is far from trivial. Elasticsearch requires a lot of resources. For smaller clusters, compute required to run Elasticsearch alone is likely higher than the price of Papertrail or similar solutions. If I can get a service managed by others for the same price as running the alternative inside my own cluster, service wins most of the time. But, there are a few exceptions.
Continue reading
