EDIT 24-11-2017
The solution below did not help me fix freezing cluster. I have identified since then, though, another problem with the setup. Prometheus was eating up all memory of a machine. The link to the article:
https://cwienczek.com/lesson-1-always-set-limits-to-containers-running-in-your-cluster/
Summary
If you’ve:
- installed Kubernetes on your baremetal machine
- that machine is running Ubuntu
- and your machine is suddenly unresponsive, then:
- Make sure you don’t have docker 1.9.1 as per this topic: [kubelet high CPU][https://github.com/kubernetes/kubernetes/issues/18515]
- (This one was my issue) Make sure you have disabled SWAP properly:
sudo swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
# I was missing the command below
# Make sure you don't have this line already in /etc/sysctl.conf, otherwise just edit the file manually
echo "vm.swappiness=0" | sudo tee --append /etc/sysctl.conf
You need to restart the machine afterwards.
Long Story ;)
I’ve noticed today that my laptop that is running kubernetes is totally unresponsive. Laptop has had Ubuntu installed. My machine was last rebooted 3 days ago, which means that the issue has appeared after 3 days of uninterrupted running. I could not SSH into machine anymore. Fortunately the machine is on a desk next room, so I could physically access the terminal. Despite the fact that the machine was highly unresponsive, I was able to run top from terminal. It turned out that both kubelet and kube-apiserver were using somewhere around 200% of CPU time (it’s a 4-core machine). What was surprising, there was another process taking up much of CPU, which was kswapd0. It was strange as I have disabled swap before installing Kubernetes on my machine (it is actually a requirement as kubelets cannot work on a machine with swap enabled at this point). It immediately struck me as I remember that kubelet cannot work with swap enabled.
The problem? It looks like after disabling swap I forgot about setting swappinness in /etc/sysctl.conf. After adding a line
vm.swappiness=0
It looks like the problem is solved for now, I will let you know in a few days in case that doesn’t solve it.