Baremetal Kubernetes on Ubuntu unresponsive

Categories: Kubernetes

EDIT 24-11-2017

The solution below did not help me fix freezing cluster. I have identified since then, though, another problem with the setup. Prometheus was eating up all memory of a machine. The link to the article:

https://cwienczek.com/lesson-1-always-set-limits-to-containers-running-in-your-cluster/

Summary

If you’ve:

  • installed Kubernetes on your baremetal machine
  • that machine is running Ubuntu
  • and your machine is suddenly unresponsive, then:
  1. Make sure you don’t have docker 1.9.1 as per this topic: [kubelet high CPU][https://github.com/kubernetes/kubernetes/issues/18515]
  2. (This one was my issue) Make sure you have disabled SWAP properly:
sudo swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

# I was missing the command below
# Make sure you don't have this line already in /etc/sysctl.conf, otherwise just edit the file manually
echo "vm.swappiness=0" | sudo tee --append /etc/sysctl.conf

You need to restart the machine afterwards.

Long Story ;)

I’ve noticed today that my laptop that is running kubernetes is totally unresponsive. Laptop has had Ubuntu installed. My machine was last rebooted 3 days ago, which means that the issue has appeared after 3 days of uninterrupted running. I could not SSH into machine anymore. Fortunately the machine is on a desk next room, so I could physically access the terminal. Despite the fact that the machine was highly unresponsive, I was able to run top from terminal. It turned out that both kubelet and kube-apiserver were using somewhere around 200% of CPU time (it’s a 4-core machine). What was surprising, there was another process taking up much of CPU, which was kswapd0. It was strange as I have disabled swap before installing Kubernetes on my machine (it is actually a requirement as kubelets cannot work on a machine with swap enabled at this point). It immediately struck me as I remember that kubelet cannot work with swap enabled.

The problem? It looks like after disabling swap I forgot about setting swappinness in /etc/sysctl.conf. After adding a line

vm.swappiness=0

It looks like the problem is solved for now, I will let you know in a few days in case that doesn’t solve it.

See also

Share this post with your friends

comments powered by Disqus