AWS EKS / Node Allocatable Settings

Details Link to heading

If you’re viewing this page, you’ve likely found that your node is transitioning to “Unknown” status because you cannot apply resource limits to a workload for some reason. You may be seeking a way to protect the kubelet process from being overwhelmed by other workloads in the same cgroup.

Protecting the Kubelet Link to heading

Using Kubelet Arguments Link to heading

You can achieve this with minimal configuration by simply adding the following two arguments:

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
  kubelet:
    flags:
      - >-
        --enforce-node-allocatable=pods,kube-reserved
      - >-
        --kube-reserved-cgroup=/runtime

The --kube-reserved-cgroup argument is necessary for this configuration to work correctly. The AWS EKS AMI creates the /runtime cgroup and it is available for use for this purpose.

The AWS EKS AMI automatically calculates the kube-reserved setting for you during its bootstrap process. Optionally, you can add the following to adjust your reservation, but use this at your own risk:

--kube-reserved=cpu=200m,memory=500Mi,ephemeral-storage=1Gi

Using Kubelet Configuration Link to heading

The modern method of configuration is through a kubelet configuration file. I haven’t configured this yet, so I’ll leave this section as “to-do.” You can likely infer the necessary configurations based on the arguments mentioned above:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration

kubeReserved:
  cpu: "200m"
  memory: "500Mi"
  ephemeral-storage: "1Gi"
kubeReservedCgroup: "/runtime" # Example cgroup path
enforceNodeAllocatable: ["pods", "kube-reserved"]

Testing

The best way to test this setup is to create an EC2 instance with user data configured in one of the ways mentioned above. Then, use EC2 Connect to log in to the instance and run the following command to follow the logs for the kubelet. You should be able to identify any issues related to your cgroup:

sudo journalctl -u kubelet -f

Protecting the Container Runtime Link to heading

Additionally, you have options to separate the container runtime process into its own cgroup. I have not found this configuration particularly helpful, but if you want to try it, you will need to create your cgroup (note: the AWS EKS AMI uses cgroupv2):

--enforce-node-allocatable=pods,kube-reserved,system-reserved
--system-reserved-cgroup=/<your_cgroup_name>
--system-reserved=cpu=200m,memory=500Mi,ephemeral-storage=2Gi

Feedback Link to heading

Although I have a long history of experience with AWS EKS, my expertise in this particular topic is limited. There may be some inaccuracies or details that I have overlooked. If you notice any issues, please feel free to reach out.

While I cannot guarantee changes will be made to my blog, I will certainly consider any feedback.