Kubernetes vs Azure Service Fabric

Recently I have investigated container orchestration solutions on Azure. The main competitors in this area are Azure Kubernetes Service and Azure Service Fabric. Both have their place, advantages and downsides. This is a small smackdown of those two based on my research and experience with Kubernetes. Unfortunately I have not had practical experience with Service Fabric so far. Please let me know your thoughts in the comments section. Let’s start then. Kubernetes Although I am focusing on the upstream Kubernetes, most of these points are also valid for Azure Kubernetes Service (AKS). The good parts Everything is customizable You would like to use Haproxy instead of Nginx as your reverse proxy and load balancer? No problem. Just grab an appropiate helm chart and install it into your cluster. The list of Kubernetes addons and extensions is endless. They run directly on the cluster and are usually managed by Kubernetes as any other service. Stability Kubernetes is very stable. I’ve been running production workloads with low-to-average load for almost half a year now without any problems. Some services are running for all this time without even restarts. The Kubernetes itself is very stable and does its job very well in keeping…

Continue Reading

Jupyter on Kubernetes – the easy way

Introduction I have been playing a bit more with Python recently. I wanted to test out some algorithms using this awesome tool called Project Jupyter. I also did not want to drain battery of my laptop while doing this, as I have a powerful machine at home running single node Kubernetes cluster. If you are interested in learning how to install such single-node-cluster on your old desktop, you can find it here. UPDATE 08-05-2018 Minimal notebook doesn't have much inside. If you want to use your jupyter notebook for data science or tensorflow experiments, I recommend switching the minimal jupyter image jupyter/minimal-notebook with jupyter/tensorflow-notebook First iteration First we need to know what are we running and whether we can run it locally. I have tried different images on Docker Hub, but unfortunately most of them require non-trivial amount of setup. What I wanted instead, was something quick and dirty, just to make a single notebook running on my home baremetal cluster. I ended up running minimal-notebook in a container. Let's first create namespace for our experiments: [crayon-5b7c5a352b0d7851515384/] After that we need to create simple jupyter deployment: [crayon-5b7c5a352b0dd545213464/] And apply it [crayon-5b7c5a352b0e0597894983/] Let it download all the layers, it will take…

Continue Reading

Creating Grafana dashboards for Kubernetes

In this blog post I will show you how to get started with grafana custom dashboard for Kubernetes. We will also be learning basics of Prometheus query language. Both of those tools are very useful in everyday of cluster admin's and user's life. Why you should learn Prometheus query language Grafana and Prometheus are very powerful tools that enable you to monitor almost anything about your Kubernetes cluster. They are, though, difficult to master. I have recently installed Kubernetes 1.9.2 and deployed Prometheus with Grafana. After having installed a few dashboards it turns out that most of the graphs do not show any data. After digging a bit deeper, I have learned that Dashboards were setup with metrics that were either deprecated, removed or renamed. Therefore I have started to customize dashboards in order to get metrics from newest Kubernetes. I have also noticed that dashboards do not provide very valuable metrics out-of-the-box. Therefore you still need to customize grafana in order to get most of your setup. Getting started Before we dive in, you need to deploy prometheus and grafana to your cluster. You can find my setup on github here. I have deployed grafana using kubectl-compatible yaml files…

Continue Reading

Custom vNet on Kubernetes on Azure with acs-engine

These are a few quirks you might encounter when you deploy Kubernetes using acs-engine onto Azure cloud with pre-defined custom vNet. Deploying custom vNet is a common scenario in most of the use-cases. You usually want to run Kubernetes along other services on your Azure cloud, such as legacy applications. This article is intended to provide you with some guidance on how to avoid the same issues I have experienced during acs-engine deployments. First thing - necessary parameters for Kubernetes deployment According to the official acs-engine documentation the only parameter you need to deploy custom vNet is firstConsecutiveStaticIP. However, with Kubernetes you need to specify also vnetCidr. If you forget to do that your cluster will fail to start. This is because of a script on master machine that does the setup of iptables. This script is run before any kubelet is started and it requires vnetCidr parameter. Unfortunately acs-engine does not validate whether this parameter is provided or not. Second thing - Azure CNI plugin needs different configuration than the one in the acs-engine docs The previous step uses Swarm as an exemplary orchestrator. However, according to the documentation of custom vNet, there is a route table that needs…

Continue Reading

Some Kubernetes notes on Azure acs-engine

Summary Recently I've worked on deploying Kubernetes clusters using Azure acs-engine as a part of our automation effort at the firm. It is a very useful tool that generates ARM (Azure Resource Manager) templates with ready to-use deployments of Virtual Machines and Kubernetes components on them. It also supports other container orchestrators, such as DC/OS or Swar. However, I will focus only on Kubernetes in this article as this is the only ochestrator I use. Acs-engine saves a lot of time compared to other custom solutions used to deploy Kubernetes cluster onto Azure cloud. It also gives a lot of flexibility with the configuration of the cluster. It is, however, open source tool with support being provided only by the community. This is a list of some of the noteworthy things that will give you more overview of what an acs-engine is and whether this tool can be of use for you or not. Some of them will also be valid for Azure ACS service as the deployment process is almost the same. Acs-engine does not have an SLA Azure can guarantee only SLA for the underlying virtual machines, but not for the cluster created by acs-engine. You need to…

Continue Reading

Prometheus problem with container metrics (cAdvisor)

Summary I've been fighting for the whole day with Prometheus and Grafana. It turns out the documentation about Prometheus and Grafa get outdated very quickly. Let me shed some light on both the problem and the solution. Intro I've been trying very hard to setup perfect grafana/prometheus configuration that I can apply to all of my clusters. This requires me to configure both of those services from YAML files, so that both prometheus and grafana start with prope configuration in place that are ready to be used. I've even been able to setup dashboards upon grafana pod start. However, there were few problems along the way. Let's start with my target setup: Kubernetes 1.9 single-node using kubeadm Prometheus 2.1.0 Grafana 5.0.0-beta1 - the freshest you can get The problem Everysthing started with prometheus. I have installed Prometheus using official helm chart. Unfortunately the default installation (I have customized only the Prometheus image to 2.1.0) has had problem with one scraping job. It was the cAdvisor job that was failing. cAdvisor is a Kubelet component that exposes containers' metrics as an API endpoint. The metrics include CPU usage, memory usage and many more. Basically those are the metrics you usually want…

Continue Reading

I have broken my Kubernetes cluster

My Kubernetes cluster is down My single-node baremetal cluster using kubeadm and on calico networking plugin is down (link to the article here). The machine won't startup anymore and there is very mysterius error from cephlib reporting that it was unable to start. Few days ago I was playing with rook (https://github.com/rook/rook). Rook uses ceph to give you easy to use persistent volume experience. Everything was fine at this point until I started doing research on advanced Prometheus configuration. After having installed Prometheus helm chart, I have noticed that Prometheus is not scraping my docker metrics (cAdvisor metrics to be more precise). I found out later that I might have installed old docker version. It was version way before 17.xx. As every smart person would do (not really...) I decided to try to upgrade docker "on the fly", while all Kubernetes components are still running on my machine. I've decided that I will perform upgrade on the fly by uninstalling and installing new version of docker while kubelet is still running. Everything was pretty fine until I restarted the machine. Then I began having error from libceph failing to start. The laptop also refused to boot and was stuck at…

Continue Reading

Watch out for faulty ingress definitions

Summary Ingress objects can interfere between each other in the cluster. If you deploy incorrect ingress definition (and kubernetes won't detect the error), Nginx Ingress Controller will fall into restart loop and won't accept any new configuration. This will lead to broken deployments from that time onwards. Description I have been trying out how wildcards in Nginx Ingress Controller work. This is required when you want to have an URL prefixed with the application's language or a country. Let say you want to host an application on a URL [crayon-5b7c5a352c47b210491702-i/], for example [crayon-5b7c5a352c480170782601-i/]. If you want to do this for all of the countries in the world, you would have quite a few ingress defitions. The easier way is to use wildcard URLs. NOTE Please remember that ingresses and especially ingress wildcards are very badly documented and there is still not a clear direction from a community how wildcards with ingress should work. This post is about Ingresses used in conjunction with Nginx Ingress Controller version 0.9-beta15. In order for this tutorial to work with newer versions of ingress you need to change annotatios from [crayon-5b7c5a352c482474725096-i/] to [crayon-5b7c5a352c484561233996-i/]. NOTE 2 Despite poorly documented, you need to make use of URL rewrite annotation in order…

Continue Reading

Kubernetes automatic SSL certificate provisioning

There is an open source project from JetStack called kube-lego. It allows you to automatically request SSL certificates for your Kubernetes cluster using Let's Encrypt free service. Working with Let's Encrypt using Kube-lego is quite straightforward. Nginx Ingress Controller has built-in support for kube-lego. Having RBAC might seem like a complication but in fact it doesn't add much of a complexity to the solution.

Continue Reading
  • 1
  • 2
Close Menu