Kubernetes Cluster in a Weekend

The Cluster - K3s

K3s is a light-weight Kubernetes compliant distribution which can run well on low powered devices and can be used on a single node cluster or it can scale up to a multi-node cluster.

This project starts with a 4 node cluster, 1 control and 3 workers. Nodes are virtual machines with 8GB of RAM and 2 disks. The OS disk is 32GB with Debian installed while the second, data disk, is used for cluster storage.

K3s can easily be installed on the control node with the following command, as described in the documentation

control-01:~$ curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644 --disable servicelb --token super_secret_password --node-taint CriticalAddonsOnly=true:NoExecute --bind-address 10.33.0.100

A couple of additional flags are passed to the install script then described in the K3s Quick start guide. Detail on the configuration options are briefly:

--write-kubeconfig-mode 644 will allow non-root users to read /etc/rancher/k3s/k3s.yaml, the k3s configuration file.

--disable servicelb is used to disable the ServiceLB package. Later on, a different load balance manager, MetalLB will be installed.

--token super_secret_password defines the password that will be used to connect agent nodes to the server.

--node-taint CriticalAddonsOnly=true:NoExecute creates a node taint that prevents pods to running on the control-plane node unless a tolerance value of "CriticalAddonsOnly" exits. A further explanation would be,

By default, server nodes will be schedulable and thus your workloads can get launched on them. If you wish to have a dedicated control plane where no user workloads will run, you can use taints. https://docs.k3s.io/datastore/ha#2-launch-server-nodes

Setup an env variable on control node to simplify using kubectl.

control-01:~$ echo "KUBECONFIG=/etc/rancher/k3s/k3s.yaml" >> /etc/environment

Install K3s as an Agent on the worker nodes, worker01-03. K3S_URL is the IP of the control node.

worker-01:~$ curl -sfL https://get.k3s.io | K3S_URL=https://10.33.0.100:6443 K3S_TOKEN=super_secret_password sh -

Label each worker to organize the nodes in the cluster.

control-01:~$ kubectl label nodes k3s-worker-01 kubernetes.io/role=worker

At this point, the cluster should be operational. We can check the status of the system from the control node. ```

control-01:~$ kubectl get nodes --show-labels

Load Balancing - MetalLB

K3s ships with, ServiceLB, a no-frills, limited configuration LoadBalancer implementation. A more feature rich replacement can be dropped in place. MetalLB is used to handle load balancing services for bare metal installation of Kubernetes, providing network connectivity into the cluster through deditcated virtual IP address (VIP).

Start with disabling ServiceLB by adding a statement to config.yaml

echo disable:
  - \"servicelb\" >> /etc/rancher/k3s/config.yaml

Using Helm, install MetalLB.

helm repo add metallb https://metallb.github.io/metallb
helm install metallb metallb/metallb

IPAddressPool configures the addresses available for MetalLB to assign to Services deployed in K3s. The IP address selected from the pool effectively act as the "external" address for the service in the cluster.

---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: metallb-address-pool
  namespace: metallb-system
spec:
  addresses:
  - 10.33.0.200-10.33.0.254

---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: metallb-l2-adver
  namespace: metallb-system
spec:
  ipAddressPools:
  - default

Now, MetalLB will automatically assign the next available IP address to any Service with configuration type: LoadBalancer

Storage Layer - Longhorn

Longhorn provides a distributed block storage system for Kubernetes clusters and can be integrated with K3s. The K3s worker nodes have a secondary disk attached which will be used by Longhorn to provision storage and must be prepped.

First, we will wipe the second disk attached to the node.

worker-01:~# wipefs -a /dev/<data_disk>

Now, format the disk as ext4.

worker-01:~# mkfs.ext4 /dev/<data_disk>

Create a directory to mount the disk. Longhorn will use this directory for data storage.

worker-01:~# mkdir /longhorn-storage

Mount the disk, then add it to /etc/fstab to persist reboot.

worker-01:~# mount /dev/<data_disk> /longhorn-storage/
worker-01:~# echo UUID\=$(findmnt -n -o UUID,TARGET,FSTYPE,OPTIONS /longhorn-storage) 0 0 | tee -a /etc/fstab

open-iscsi and a NFSv4 client is required for Longhorn.

worker-01:~# apt install -y nfs-common open-iscsi

Now, we are ready to install Longhorn using Helm. Additional configuration flags, such as, defaultDataPath can be passed to Helm as described in customize default settings.

helm repo add longhorn https://charts.longhorn.io; helm repo update

helm install longhorn longhorn/longhorn --namespace longhorn-system --create-namespace --set defaultSettings.defaultDataPath="/longhorn-storage"

We can access the Longhorn UI by creating a service from the control node.

control-01:~$ kubectl apply -f longhorn-svc-lb.yaml

longhorn-svc-lb.yaml

---
apiVersion: v1
kind: Service
metadata:
  name: longhorn-svc-lb
  namespace: longhorn-system
spec:
  selector:
    app: longhorn-ui
  type: LoadBalancer
  loadBalancerIP: 10.33.0.210
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: http

Architecture Diagram

< back