Setting Up Cluster Autoscaler In EKS

Unlike GKE, EKS does not come with Cluster Autoscaler. We’ll have to configure it ourselves. We’ll need to add a few tags to the Autoscaling Group dedicated to worker nodes, to put additional permissions to the Role we’re using, and to install Cluster Autoscaler.

Let’s get going.

The commands that follow assume that you created an EKS cluster using etsctl and that you have the environment variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_DEFAULT_REGION defined. If you do not have such a cluster, you can create one using the
eks-ca.sh Gist. Alternatively, you can create the cluster without eksctl, but the commands that follow might need to be slightly modified. You’ll also need to clone the code of the vfarcic/k8s-specs repository (e.g., git clone https://github.com/vfarcic/k8s-specs) and enter inside the cloned folder (e.g.,cd k8s-specs`). The repository contains the definitions we’ll use in this article.

We’ll add a few tags to the Autoscaling Group dedicated to worker nodes. To do that, we need to discover the name of the group. Since we created the cluster using eksctl, names follow a pattern which we can use to filter the results. If, on the other hand, you created your EKS cluster without eksctl, the logic should still be the same as the one that follows, even though the commands might differ slightly.

First, we’ll retrieve the list of the AWS Autoscaling Groups, and filter the result with jq so that only the name of the matching group is returned.

export NAME=devops25

ASG_NAME=$(aws autoscaling \
    describe-auto-scaling-groups \
    | jq -r ".AutoScalingGroups[] \
    | select(.AutoScalingGroupName \
    | startswith(\"eksctl-$NAME-nodegroup\")) \
    .AutoScalingGroupName")

echo $ASG_NAME

The output of the latter command should be similar to the one that follows.

eksctl-devops25-nodegroup-0-NodeGroup-1KWSL5SEH9L1Y

We stored the name of the cluster in the environment variable NAME. Further on, we retrieved the list of all the groups and filtered the output with jq so that only those with names that start with eksctl-$NAME-nodegroup are returned. Finally, that same jq command retrieved the AutoScalingGroupName field and we stored it in the environment variable ASG_NAME. The last command output the group name so that we can confirm (visually) that it looks correct.

Next, we’ll add a few tags to the group. Kubernetes Cluster Autoscaler will work with the one that has the k8s.io/cluster-autoscaler/enabled and kubernetes.io/cluster/[NAME_OF_THE_CLUSTER] tags. So, all we have to do to let Kubernetes know which group to use is to add those tags.

aws autoscaling \
    create-or-update-tags \
    --tags \
    ResourceId=$ASG_NAME,ResourceType=auto-scaling-group,Key=k8s.io/cluster-autoscaler/enabled,Value=true,PropagateAtLaunch=true \
    ResourceId=$ASG_NAME,ResourceType=auto-scaling-group,Key=kubernetes.io/cluster/$NAME,Value=true,PropagateAtLaunch=true

The last change we’ll have to do in AWS is to add a few additional permissions to the role created through eksctl. Just as with the Autoscaling Group, we do not know the name of the role, but we do know the pattern used to create it. Therefore, we’ll retrieve the name of the role, before we add a new policy to it.

IAM_ROLE=$(aws iam list-roles \
    | jq -r ".Roles[] \
    | select(.RoleName \
    | startswith(\"eksctl-$NAME-nodegroup-0-NodeInstanceRole\")) \
    .RoleName")

echo $IAM_ROLE

The output of the latter command should be similar to the one that follows.

eksctl-devops25-nodegroup-0-NodeInstanceRole-UU6CKXYESUES

We listed all the roles, and we used jq to filter the output so that only the one with the name that starts with eksctl-$NAME-nodegroup-0-NodeInstanceRole is returned. Once we filtered the roles, we retrieved the RoleName and stored it in the environment variable IAM_ROLE.

Next, we need JSON that describes the new policy. I already prepared one, so let’s take a quick look at it.

cat scaling/eks-autoscaling-policy.json

The output is as follows.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "autoscaling:DescribeAutoScalingGroups",
        "autoscaling:DescribeAutoScalingInstances",
        "autoscaling:DescribeLaunchConfigurations",
        "autoscaling:DescribeTags",
        "autoscaling:SetDesiredCapacity",
        "autoscaling:TerminateInstanceInAutoScalingGroup"
      ],
      "Resource": "*"
    }
  ]
}

If you’re familiar with AWS (I hope you are), that policy should be straightforward. It allows a few additional actions related to autoscaling.

Finally, we can put the new policy to the role.

aws iam put-role-policy \
    --role-name $IAM_ROLE \
    --policy-name $NAME-AutoScaling \
    --policy-document file://cluster/eks-autoscaling-policy.json

Now that we added the required tags to the Autoscaling Group and that we created the additional permissions that will allow Kubernetes to interact with the group, we can install Cluster Autoscaler Helm Chart.

helm install stable/cluster-autoscaler \
    --name aws-cluster-autoscaler \
    --namespace kube-system \
    --set autoDiscovery.clusterName=$NAME \
    --set awsRegion=$AWS_DEFAULT_REGION \
    --set sslCertPath=/etc/kubernetes/pki/ca.crt \
    --set rbac.create=true

kubectl -n kube-system \
    rollout status \
    deployment aws-cluster-autoscaler

Once the Deployment is rolled out, the autoscaler should be fully operational. Try it out by deploying more Pods than your cluster can handle and observe that a new node was added a few minutes later. Similarly, nodes will be removed when the cluster is underutilized.

The DevOps 2.5 Toolkit: Monitoring, Logging, and Auto-Scaling Kubernetes

The article you just read is an extract from The DevOps 2.5 Toolkit: Monitoring, Logging, and Auto-Scaling Kubernetes.

What do we do in Kubernetes after we master deployments and automate all the processes? We dive into monitoring, logging, auto-scaling, and other topics aimed at making our cluster resilient, self-sufficient, and self-adaptive.

The book is still in progress and it is currently available only from LeanPub.com. I’d love to hear your thoughts and comments.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s