Always set limits to containers running in your cluster

History about a Prometheus eating up 20GB of RAM 2017-11-30 UPDATE As those few days have passed I have not had any problems with Kubernetes cluster being unresponsive. Therefore this article concludes a few weeks of investigation why machine could entirely freeze. Set limits to your pods so that they won't kill your node. The History WeaveWorks Cloud DaemonSet deploys by default Prometheus to the cluster. Prometheus scrapes metrics out of your cluster and stores them and creates time-series data out of them ( this might not be the accurate description of what Prometheus does, but it's good enough for what just hapened ). The prometheus (version 1.7.1 used by my WeaveWorks setup) has apparently been very eager in scraping the data and pushing it to WeaveWorks servers, but never actually got rid of the data. This resulted in over 20GB of data over 3 days of a lifespan in a single-node cluster. I have upgraded to Prometheus 1.8.1 since then, we'll see if that helps. I have also put a strict limit of 2GB on Prometheus so that it won't kill my cluster ever again. Lesson to learn Always, ALWAYS set container limits. Kubernetes was not smart enough to…

Continue Reading

Baremetal Kubernetes on Ubuntu unresponsive

EDIT 24-11-2017 The solution below did not help me fix freezing cluster. I have identified since then, though, another problem with the setup. Prometheus was eating up all memory of a machine. The link to the article: https://cwienczek.com/lesson-1-always-set-limits-to-containers-running-in-your-cluster/ Summary If you've: - installed Kubernetes on your baremetal machine - that machine is running Ubuntu - and your machine is suddenly unresponsive, then: Make sure you don't have docker 1.9.1 as per this topic: kubelet high CPU (This one was my issue) Make sure you have disabled SWAP properly: [crayon-5b7c5a1e50d3b915342254/] You need to restart the machine afterwards. Long Story ;) I've noticed today that my laptop that is running kubernetes is totally unresponsive. Laptop has had Ubuntu installed. My machine was last rebooted 3 days ago, which means that the issue has appeared after 3 days of uninterrupted running. I could not SSH into machine anymore. Fortunately the machine is on a desk next room, so I could physically access the terminal. Despite the fact that the machine was highly unresponsive, I was able to run top from terminal. It turned out that both kubelet and kube-apiserver were using somewhere around 200% of CPU time (it's a 4-core machine). What was…

Continue Reading

Setting up secure helm chart repository on Azure Blob Storage

Kubernetes helm repository supports only basic authentication at the time of writing this article. There is, though, another and perhaps simpler way as of helm 2.7.0. Using Azure Blob Storage you can easily make your helm repository private. Requirements Time: ~10 minutes Helm Package Manager 2.7.0-rc1 or later Microsoft Azure account, at least with permissions to create azure storage account Azure CLI, tested on 2.0.19 Darwin Helm chart that you can upload to the cloud Summary Create azure storage account in one of your resource groups Add blob storage container to Azure Storage Account and set access to private Go to Storage account -> Shared Access Signature and generate read-only credentials for helm users The url to your repository will be: [crayon-5b7c5a1e50f66424138510-i/] Step by step guide (optional) Create new resource group [crayon-5b7c5a1e50f6b332192909/] Create storage account and get account key [crayon-5b7c5a1e50f6d929301020/] Create blob container [crayon-5b7c5a1e50f6f316943171/] Upload a helm chart to the repository and index.yaml In this tutorial I will download chart from stable repository and re-upload it to private azure repository. However, you can easily use [crayon-5b7c5a1e50f70814310289-i/]to package your helm chart into [crayon-5b7c5a1e50f72842128350-i/]file. [crayon-5b7c5a1e50f73718903560/] The last command should create [crayon-5b7c5a1e50f74967421499-i/] file inside [crayon-5b7c5a1e50f76598186872-i/] directory. Now, provided that environment variables [crayon-5b7c5a1e50f77798737458-i/] and [crayon-5b7c5a1e50f78740904168-i/] are still set, you can upload those files to our…

Continue Reading
  • 1
  • 2
Close Menu