Automating Jenkins Docker Setup

37 Replies

Jenkins is, by far, the most used CI/CD tool in the market. That comes as no surprise since it’s been around for a while, it has one of the biggest open source communities, it has enterprise version for those who need it, and it is straightforward to extend it to suit (almost) anyone’s needs.

Products that dominate the market for years tend to be stable and very feature rich. Jenkins is no exception. However, with age come some downsides as well. In the case of Jenkins, automation setup is one of the things that has a lot to be desired. If you need Jenkins to serve as an orchestrator of your automation and tasks, you’ll find it to be effortless to use. But, if you need to automate Jenkins itself, you’ll realize that it is not as smooth as you’d expect from modern tools. Never the less, Jenkins setup can be automated, and we’ll go through one possible solution.

I will not go through everything we can do as part of Jenkins setup. A whole book could be written on that subject. Instead, I’ll try to show you how to do two of the most common configuration steps. We’ll automate the installation of Jenkins plugins and setup of an administrator user. If you have a need to add more to the mix, please drop me an email (you’ll find my info on the About page).

Let us quickly go through the objectives we’ll set in front of us.

The Objectives

Automation is the key to success of any software company. Since I believe that every company is a software company (even though some are still not aware of that), we can say that automation is the key for everyone.

I expect that you are using Docker to run your services. If you’re not, please reconsider your strategy. It provides so many benefits that ignoring it will mean that competition will overrun you. I won’t go into details why Docker is great. You probably already know all that.

The short version of the objectives behind automated Jenkins setup is that we do not want to do anything but run a single command. The result should be a fully configured Jenkins master that we can multiply to as many instances as we need. Docker, through Swarm, will provide fault tolerance, (kind of) high availability, and what so not.

A bit longer version is that we want to set up all the plugins we’ll need and create at least one administrative user. That does not mean that you will not add more plugins later. You probably will. However, that should not prevent us from pre-installing those that are most commonly used within your organization. Without at least one user, anyone could access your Jenkins master and, potentially, get access to your whole system. There is probably no need to discuss why we should avoid that.

Before we start working on automation, we’ll run a Jenkins master manually and fetch some information that will be useful later on.

Setting Up Jenkins Manually

We’ll start by creating a Jenkins service inside a Swarm cluster. If you do not have one, you can easily convert your Docker for Windows/Mac/Linux to a single node cluster by executing docker swarm init.

W> If you are a Windows user, please run all the commands from Git Bash (installed through Git) or any other Bash you might have.

docker service create --name jenkins \
    -p 8080:8080 jenkins/jenkins:lts-alpine

After a while, jenkins image will be pulled, and the service will be running. Feel free to check the status by executing docker service ps jenkins.

Once Jenkins is up and running, we can open its UI in a browser.

If you’re not using your local Docker for Windows/Mac/Linux engine as the only node in the cluster, please replace localhost with the IP of one of your Swarm nodes.

If you’re a Windows user, Git Bash might not be able to use the open command. If that’s the case, replace open with echo. As a result, you’ll get the full address that should be opened directly in your browser of choice. 

open "http://localhost:8080"

You should see the first screen of the setup where you should enter the Administrator password. It is available inside the /var/jenkins_home/secrets/initialAdminPassword file inside the container that hosts the service.

If you’re not using your local Docker for Windows/Mac/Linux engine as the only node in the cluster, please find the node where the service is running and SSH into it before executing the commands that follow. 

ID=$(docker container ls -q \
    -f "label=com.docker.swarm.service.name=jenkins")

docker container exec -it $ID \
    cat /var/jenkins_home/secrets/initialAdminPassword

We used docker container ls command with a filter to find the ID of the container that hosts the service. Next, we executed a command that displayed content of the /var/jenkins_home/secrets/initialAdminPassword file. The output will vary from one execution to another. In my case, it is as follows.

ecd46df9ec1b420dadacdb56de9492c8

Please copy the password and paste it into the Administrator password field in Jenkins UI and follow the instructions to setup your Jenkins master. Choose the plugins that you’ll need. When you reach the last setup screen called Create First Admin User, please use admin as both the username and the password.

Once the setup is finished, we can send a request that will retrieve the list of all the plugins we installed.

Before we proceed, please check whether you have jq installed. If you don’t, you can find on in its official site. 

curl -s -k "http://admin:admin@localhost:8080/pluginManager/api/json?depth=1" \
  | jq -r '.plugins[].shortName' | tee plugins.txt

We sent a request to the Jenkins plugin manager API and retrieved all the plugins we installed during the manual setup. We piped the result to jq and filtered it in a way that only short names are output. The result is stored in the plugins.txt file.

We don’t need Jenkins service anymore. It has only a temporary function that allowed us to retrieve the list of plugins we’ll need. Let’s destroy it.

docker service rm jenkins

Creating Jenkins Image With Automated Setup

Before we define Dockerfile that we’ll use to create Jenkins image with automatic setup, we need to figure out how to create at least one administrative user. Unfortunately, there is no (decent) API we can invoke. Our best bet is to execute a Groovy script that will do the job. The script is as follows.

#!groovy

import jenkins.model.*
import hudson.security.*
import jenkins.security.s2m.AdminWhitelistRule

def instance = Jenkins.getInstance()

def hudsonRealm = new HudsonPrivateSecurityRealm(false)
hudsonRealm.createAccount("admin", "admin")
instance.setSecurityRealm(hudsonRealm)

def strategy = new FullControlOnceLoggedInAuthorizationStrategy()
instance.setAuthorizationStrategy(strategy)
instance.save()

Jenkins.instance.getInjector().getInstance(AdminWhitelistRule.class).setMasterKillSwitch(false)

The key is in the hudsonRealm.createAccount("admin", "admin") line. It creates an account with both username and password set to admin.

The major problem with that script is that it has the credentials hard-coded. We could convert them to environment variables, but that would be very insecure. Instead, we’ll leverage Docker secrets. When attached to a container, secrets are stored in /run/secrets in-memory directory.

More secure (and flexible) version of the script is as follows.

#!groovy

import jenkins.model.*
import hudson.security.*
import jenkins.security.s2m.AdminWhitelistRule

def instance = Jenkins.getInstance()

def user = new File("/run/secrets/jenkins-user").text.trim()
def pass = new File("/run/secrets/jenkins-pass").text.trim()

def hudsonRealm = new HudsonPrivateSecurityRealm(false)
hudsonRealm.createAccount(user, pass)
instance.setSecurityRealm(hudsonRealm)

def strategy = new FullControlOnceLoggedInAuthorizationStrategy()
instance.setAuthorizationStrategy(strategy)
instance.save()

Jenkins.instance.getInjector().getInstance(AdminWhitelistRule.class).setMasterKillSwitch(false)

This time, we placed contents of jenkins-user and jenkins-pass files into variables and used them to create an account.

Please save the script as security.groovy file.

Equipped with the list of plugins stored in plugins.txt and the script that will create a user from Docker secrets, we can proceed and create a Dockerfile.

Please create Dockerfile with the content that follows.

FROM jenkins/jenkins:lts-alpine

ENV JAVA_OPTS="-Djenkins.install.runSetupWizard=false"

COPY security.groovy /usr/share/jenkins/ref/init.groovy.d/security.groovy

COPY plugins.txt /usr/share/jenkins/ref/plugins.txt
RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/ref/plugins.txt

The image will be based on alpine version of Jenkins which is much smaller and more secure than others.

We used JAVA_OPTS to disable the setup wizard. We won’t need it since our setup will be fully automated.

Next, we’re copying the security.groovy file into /usr/share/jenkins/ref/init.groovy.d/ directory. Jenkins, when initialized, will execute all the Groovy scripts located in that directory. If you create additional setup scripts, that is the place you should place them.

Finally, we are copying the plugins.txt file that contains the list of all the plugins we need and passing that list to the install-plugins.sh script which is part of the standard distribution.

With the Dockerfile defined, we can build our image and push it to Docker registry.

Please replace vfarcic with your Docker Hub user. 

docker image build -t vfarcic/jenkins .

docker login # Answer the questions with your credentials

docker image push vfarcic/jenkins

With the image built and pushed to the Registry, we can, finally, create a Jenkins service.

Creating Jenkins Service With Automated Setup

We should create a Compose file that will hold the definition of the service. It is as follows.

Please replace vfarcic with your Docker Hub user. 

version: '3.1'

services:

  main:
    image: vfarcic/jenkins
    ports:
      - 8080:8080
      - 50000:50000
    secrets:
      - jenkins-user
      - jenkins-pass

secrets:
  jenkins-user:
    external: true
  jenkins-pass:
    external: true

Save the definition as jenkins.yml file.

The Compose file is very straightforward. If defines only one service (main) and uses the newly built image. It is exposing ports 8080 for the UI and 50000 for the agents. Finally, it defines jenkins-user and jenkins-pass secrets.

We should create the secrets before deploying the stack. The commands that follow will use admin as both the username and the password. Feel free to change it to something less obvious.

echo "admin" | docker secret create jenkins-user -

echo "admin" | docker secret create jenkins-pass -

Now we are ready to deploy the stack.

docker stack deploy -c jenkins.yml jenkins

A few moments later, the service will be up and running. Please confirm the status by executing docker stack ps jenkins.

Let’s confirm that everything works as expected.

open "http://localhost:8080"

You’ll notice that this time, we did not have to go through the Setup Wizard steps. Everything is automated and the service is ready for use. Feel free to log in and confirm that the user you specified is indeed created.

If you create this service in a demo environment (e.g. your laptop) now is the time to remove it and free your resources for something else. The image you just built should be ready for production.

docker stack rm jenkins

docker secret rm jenkins-user

docker secret rm jenkins-pass

37 thoughts on “Automating Jenkins Docker Setup”

    1. cuky

      the source image needs an update. nasty message found on usage.

      Manage Jenkins
      New version of Jenkins (2.89.2) is available for download (changelog).
      Warnings have been published for the following currently installed components:

      Jenkins 2.60.3 core and libraries:
    Multiple security vulnerabilities in Jenkins 2.88 and earlier, and LTS 2.73.2 and earlier
    Multiple security vulnerabilities in Jenkins 2.83 and earlier, and LTS 2.73.1 and earlier
    Multiple security vulnerabilities in Jenkins 2.94 and earlier, and LTS 2.89.1 and earlier
      Reply ↓
      1. Viktor Farcic Post author

        jenkins image became deprecated since I wrote this article and a few weeks ago it stopped receiving updates. I updated the article to use the new image jenkins/jenkins:lts-alpine. Can you please try it out and let me know if it works as expected.

        Reply ↓
  3. Ankit Bhalla

    Hi Viktor,
    while setting up jenkins blueocean official image in docker swarm mode
    from a service
    docker service create –name jenkins \
    -p 8080:8080 \
    -p 50000:50000 \
    –mount “type=bind,source=$PWD/docker/jenkins,target=/var/jenkins_home” \
    –mount “type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock” \
    –mount “type=bind,source=/var/lib/docker,target=/var/lib/docker” \
    –label com.df.notify=true \
    –label com.df.distribute=true \
    –label com.df.servicePath=/blue \
    –label com.df.port=8080 \
    –network proxy \
    –reserve-memory 1500m \
    jenkinsci/blueocean

    ##Getting this error while creating pipeline, as pipeline is failing while calling docker client from jenkins container itself
    docker pull maven:3.3.9-jdk-8

    ##CLI error inside container
    bash-4.3$ docker info
    Cannot connect to the Docker daemon. Is the docker daemon running on this host?

    ##
    root@ip-192-168-1-237:/home/ubuntu# ls -lrt /var/run/docker.sock
    srw-rw—- 1 root docker 0 Jun 25 09:51 /var/run/docker.sock

    Also privileged mode is also disabled in docker swarm service

    Thanks in Advance
    Looking forward for your advise on this

    Warm Regards
    Ankit

    Reply ↓
    1. Viktor Farcic Post author

      Judging by the command you used to create a master, you’re running jobs inside it, not through agents? If that’s the case, you need to extend the official image and add Docker to it. Otherwise, you’re sending Docker commands to the container that doesn’t have Docker.

      As a side note, I would suggest attaching agents to that master. It is not recommended to run jobs inside a master.

      Reply ↓
    2. Mikko Koskinen

      Jenkins runs as ‘jenkins’ inside a container. Adding ‘jenkins’ user to ‘docker’ group should do it. You need to use the same gid.

      Reply ↓
      1. Viktor Farcic Post author

        In most cases, you should not run jobs on the master server. Instead, all the jobs should run through Jenkins agents. Those agents should mount Docker socket as a volume. Otherwise, you’ll run Docker inside Docker (DinD) and that not a very good idea. If the socket is mounted, it does not matter whether you run as a user that is in the docker group or not.

        Reply ↓
  4. Guillem CANAL

    Very insightful, many thanks.

    It should be interesting to explain how to add Jenkins slave programmatically using the Docker swarm mode scale feature.

    Reply ↓
    1. Viktor Farcic Post author

      When working with stateful services (Jenkins is one of them) that do not support data replication across replicas, you should always store the state on a network drive. That way it will be persisted across failures. Depending on your hosting provider, that network drive can be EFS/EBS (for AWS), NFS (mostly for on-prem), and so on. I’d recommend REXRay for that.

      Assuming that Jenkins home is persisted, you can periodically copy it to a tape (or whatever you’re using for backups) but, in that case, you risk corruption. A better option would be to use one of the backup plugins and store the results somewhere safe. An alternative is to commit Jenkins home to Git.

      Keep in mind that jobs should reside in Jenkinsfile stored/application in service repositories. That means that your jobs are not in Jenkins and are backup with whatever mechanism you’re using to back up your code. That leaves us with general Jenkins configs that should be backed-up.

      Reply ↓
      1. Viktor Farcic Post author

        I’m not sure what you meant when you said that you’re “downloading the jar file”. If you’re building your Jenkins image from scratch, there will be no install-plugins.sh script unless you create it or download it. If, like in this article, you create your image based (FROM) the official or community version of Jenkins, the script is there. You can see that from https://github.com/jenkinsci/docker/blob/master/Dockerfile#L74

        Reply ↓
  7. taragurung (@Taragrg6)

    COPY security.groovy /usr/share/jenkins/ref/init.groovy.d/security.groovy . Regarding this in the grovy script when does this get executed . is just copying it enough . when will these get executed and created users .

    Reply ↓
  8. raketemensch

    I’m getting this:
    Step 1/5 : FROM jenkins/jenkins:lts-alpine
    —> 3e789cbee0e5
    Step 2/5 : ENV JAVA_OPTS “-Djenkins.install.runSetupWizard=false”
    —> Using cache
    —> ddfc6e1a6007
    Step 3/5 : COPY security.groovy /usr/share/jenkins/ref/init.groovy.d/security.groovy
    —> Using cache
    —> ac31dcf4f1cd
    Step 4/5 : COPY plugins.txt /usr/share/jenkins/ref/plugins.txt
    —> Using cache
    —> bff92d50450a
    Step 5/5 : RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/ref/plugin
    —> Running in fed849637519
    /bin/sh: can’t open /usr/share/jenkins/ref/plugin: no such file

    Reply ↓
    1. Viktor Farcic Post author

      Seems that you did not copy the complete line to Dockerfile. It should be:

      RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/ref/plugins.txt

      You seem to be missing s.txt.

      Reply ↓
      1. raketemensch

        Wow, thanks. I was stuck in the house all day with a fever and thought I’d learn something rather than waste the time. Thanks for the post, and the help, next time I’ll try to use a more coherent brain.

        Reply ↓
  9. Dan

    This is the only way of setting up jenkins within docker properly. out of tons of garbage your guide worked. very well done!

    Reply ↓
  10. Fraser Goffin

    Using the YADP Plugin is a great way of running ephemeral Jenkins slave containers. All you need is a simple docker host which you configure the YADP Cloud part of the Plugin to point to the running docker daemon on your slave host. Then you just create a Docker image containing whatever build tools you need (you might create several different variants for different types of build to make it easier to upgrade them independently) and finish off the YADP Plugin (adding a template) by selecting your image. When you run a job, the Plugin will spin up a container on your slave host using the image you configured, run your job, harvest any result then kill off the container. The job you run can be defined as a Groovy script Pipeline so all just SCM managed code too. All very straight forwards yet powerful. You slave hosts are just ‘cattle’ docker instance running on whatever platform you need and all of the slave install and config is nicely encapsulated in your slave build container image(s).

    You can run as many YADP clouds as you want and within each cloud, as many templates (slave image configs) as you need. You can operate your slave hosts as a swarm cluster if you need more scalability for any cloud config.

    We’ve been using this approach for a year or so very successfully. It allows us to provide a tailored Jenkins set up for any team that wants there own master/slave setup rather than the hassle of trying to operate a shared mixed team service. It’s very easy to cookie-cut this for anyone who needs it in a matter of minutes. Master config persistence can easily be handled using network storage.

    HTHs

    Fraser.

    Reply ↓
    1. Viktor Farcic Post author

      I have not (yet) done that. I’d suggest first checking which XML file contains the proxy config. In many cases it’s easier to add a file than to script it.

      Reply ↓
  12. borgified

    great post, im following along and so far so good

    in order to upload your built docker image to dockerhub
    (docker image push vfarcic/jenkins), first you have to login with this:

    docker login –username=yourhubusername

    otherwise you get

    denied: requested access to the resource is denied

    Reply ↓
  13. Arran Bartish

    Great Post!

    When generating the plugins.txt it only includes the plugin names and not plugin versions. This may produce some unexpected surprises the next time you need to build the image.

    curl -s -k "http://username:password@localhost:8080/pluginManager/api/json?depth=1" | jq -r '.plugins[] | .shortName +":"+.version' | tee plugins.txt

    This minor update to the original will create a plugins.txt in the format

    pluginShortName:pluginVersion

    If you want to use the latest plugin versions for any new builds then the original command does the trick.

    Reply ↓
  15. Matt

    Very helpful!

    Question about security.groovy: is it possible to read “user” and “pass” from a credentials file located in an AWS S3 bucket? If so, what would that look like?

    Reply ↓
  16. RigbyReardon

    Hi Viktor,
    Excellent post, it really helped me get my Jenkins docker Setup.
    I’m quite new to docker and even newer to swarms/services:

    How can I remove the secret files in the docker after I use them to set up credentials in Jenkins at (first) startup? Secrets rm does not work with a running container. And not removing them keeps the credentials openly visible in the containers file system (which might be bad in something as powerful as Jenkins).

    Or am I missing something?

    Thanks again for the post.

    Cheers
    Hendrik

    Reply ↓

  17. Pingback: Basic Jenkins Setup Part 1 – daydreamingtech

  18. August

    Hello there,
    Having a tough time, deploying, kept looping thru ‘ready’ and ‘running’ state, but could not check at localhost:8080 for jenkins, here’s my logs:
    [root@localhost var]# docker stack ps jenkins
    ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
    dggcg8x8cv9b jenkins_main.1 atrillanes72/jenkins:latest localhost.localdomain Ready Ready 2 seconds ago
    t9bk39s6v89s _ jenkins_main.1 atrillanes72/jenkins:latest localhost.localdomain Shutdown Failed 2 seconds ago “starting container failed: Re…”
    0lm6yjiu5l1w _ jenkins_main.1 atrillanes72/jenkins:latest localhost.localdomain Shutdown Failed 9 seconds ago “starting container failed: Re…”

    Any ideas, what I did wrong? (some ideas on my setup: I’m running this test on virtualbox VM Centos 7, 3GB RAM 2vCpu)

    Reply ↓
    1. Viktor Farcic Post author

      Can you add --no-trunc so that we see full messages? Also, take a look at the logs and the events. Once we find an indication of the issue, we can try to figure out what’s causing it.

      Reply ↓
      1. August

        thanks for the reply Viktor, here’s what I got when i tried the –no-trunc flag:

        [root@localhost auto-jenkins]# docker stack ps jenkins –no-trunc
        ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
        fypj0vqvpf8z499y1s28g06an jenkins_main.1 atrillanes72/jenkins:latest@sha256:29b3768054cabbc6683cdde228b6e6247959c51426abc073953f07d0a6846ed1 localhost.localdomain Ready Ready 1 second ago
        rsmqyh7ymhdwnja8l3x6vwvbl _ jenkins_main.1 atrillanes72/jenkins:latest@sha256:29b3768054cabbc6683cdde228b6e6247959c51426abc073953f07d0a6846ed1 localhost.localdomain Shutdown Failed 2 seconds ago “starting container failed: RemoveSecretsPath failed: remove /var/lib/docker/containers/160e12916382266d79d148e5f29bab14779586a36b64470598a60c6cc2a44049/secrets/jenkins-pass: read-only file system”
        m17a1k801krfipngs8khj97zw _ jenkins_main.1 atrillanes72/jenkins:latest@sha256:29b3768054cabbc6683cdde228b6e6247959c51426abc073953f07d0a6846ed1 localhost.localdomain Shutdown Failed 7 seconds ago “starting container failed: RemoveSecretsPath failed: remove /var/lib/docker/containers/443fd08444ce58203c425098ad1319374bda22a9217e1d50c3dd18aa31fa6b0d/secrets/jenkins-pass: read-only file system”
        0a0v2sptgc513hwhidocoo25p _ jenkins_main.1 atrillanes72/jenkins:latest@sha256:29b3768054cabbc6683cdde228b6e6247959c51426abc073953f07d0a6846ed1 localhost.localdomain Shutdown Failed 14 seconds ago “starting container failed: RemoveSecretsPath failed: remove /var/lib/docker/containers/54ebe1500dc7c6680c860ed1f92da05adac3f1abb10e3018ca51be763030f07f/secrets/jenkins-pass: read-only file system”
        naq70uvv2x7oka9liv2fzy2ce _ jenkins_main.1 atrillanes72/jenkins:latest@sha256:29b3768054cabbc6683cdde228b6e6247959c51426abc073953f07d0a6846ed1 localhost.localdomain Shutdown Failed 20 seconds ago “starting container failed: RemoveSecretsPath failed: remove /var/lib/docker/containers/cfc479fb7978c52a5defa191703e4759289ff924decb9e15c0deda2bbde3e634/secrets/jenkins-pass: read-only file system”

        Reply ↓

Leave a Reply