NOTE: This is a rough record of my journey to set up my Kubernetes cluster. I'm going to migrate this to an article or better documentation when I get the chance.
The following is a loose record of exactly how I managed to set up my own home Kubernetes cluster. It is useful for others following this path or for myself in 6 months when I can't remember all of this.
I used Raspberry Pi 5 devices, specifically the 16gb version (4 CPU cores each, so 16 total CPU cores + 64gb memory). I decided to run the cluster initially with 4 of them.
I plan to add a full PC with a 5090 to the cluster to allow for scheduling model training in the cluster.
Here's the list of what I bought for each of the main control/worker nodes:
After putting together your hardware, you'll need to go ahead and flash the Pi OS to an SD card. I used micro SD cards for the boot drive (the NVMe drives are for all other data used in the kube cluster).
Download Raspberry Pi Imager and connect your micro SD card.
When going through the flow, select the following options:
- Choose Raspberry Pi 5 for the device
Raspberry Pi OS 64 bit- Choose your SD card as the storage to write to
- Edit settings under
Generaland set the hostname, a username/password for SSH (although we won't use it), configure the LAN, and set your locale - Also, go to
Servicesand enable SSH. I'd allow public-key authentication only (and set an authorized_keys key for your account)
Go ahead and write to the device. Once it's done, remove your micro SD card.
Insert it into the Pi and power it up. Your Pi will initiate the headless boot process, but you should be able to ssh into it eventually via:
ssh <your username>@<Pi IP address>.localGreat job. You're ssh-ed into your Pi!
You'll want to setup static IP addresses for your Pi devices. You should assign these on your router itself (whatever router setup you have).
This will be unique to you, so I'll leave you to research this out and do it your yourself.
After doing that, you should be able to ssh into Pis via <your username>@<Pi IP address>. To be able to reference all of your Pis by hostname on your main machine (such as your computer you're reading this on now), update your /etc/hosts file to list all of your Pis:
# Kubernetes cluster nodes
192.168.0.11 node1 node1.local # Control node
192.168.0.12 node2 node2.local
192.168.0.13 node3 node3.local
192.168.0.14 node4 node4.localNext (while ssh-ed into each of your Pis), install the DHCP server for each device via this command:
sudo apt upgrade
sudo apt install dhcpcd5 # The 5 here is correct
sudo systemctl enable dhcpcd
sudo systemctl start dhcpcdThen, add a static IP to the dchpcd.conf file:
sudo nano /etc/dhcpcd.conf
# Add something like this to the bottom. It will be different for you depending on your router IP ranges.
interface eth0
static ip_address=192.168.0.18/24
static routers=192.168.0.1
static domain_name_servers=1.1.1.1 8.8.8.8Flush your DHCP leases and restart your Pi afterwards:
sudo ip addr flush dev wlan0
sudo systemctl restart dhcpcd
sudo reboot
ip addr show wlan0 # Check for correct IP assignmentIf you used the Raspberry Pi OS instead of Debian (or some Ubuntu flavor), you'll need to instead run these commands:
# Check status of wired connection - should show something like "Wired connection 1"
nmcli con show
# Set static IP
sudo nmcli con mod "Wired connection 1" ipv4.addresses <your initial IP for IP range here>/24
sudo nmcli con mod "Wired connection 1" ipv4.gateway <your gateway IP>
sudo nmcli con mod "Wired connection 1" ipv4.dns "<your gateway IP> 8.8.8.8"
sudo nmcli con mod "Wired connection 1" ipv4.method manual
sudo nmcli con up "Wired connection 1"
# Turn off Wifi
sudo nmcli radio wifi off
sudo systemctl disable wpa_supplicant
# As an aside, you can turn wifi on at any time you want with this
sudo nmcli radio wifi on
sudo systemctl enable wpa_supplicant
sudo systemctl start wpa_supplicantThe above + setting static IP at your router level should be sufficient.
Next, turn off wifi:
sudo nmcli radio wifi off
sudo systemctl disable wpa_supplicant
sudo ip link set wlan0 down
# Check that WIFI is disabled
nmcli radioNext, set a static hostname on the network. Example:
sudo hostnamectl set-hostname some-cool-pi-name-hereIf you're using ethernet connections like me, you'll also want to disable Wi-Fi for cluster stability:
echo "dtoverlay=disable-wifi" | sudo tee -a /boot/config.txtAlso, set the CPU governer to schedutil so that you can control CPU frequency scaling for balanced performance and power usage:
# This will make it persistent across reboots which is what you want
sudo apt install -y cpufrequtils
echo 'GOVERNOR="schedutil"' | sudo tee /etc/default/cpufrequtilsDo this on every single one of your Pis after booting them up. Next, check the network from your control node via:
#switch to root
sudo -s
#install nmap
apt install nmap
#scan local network range to see who is up
nmap -sP 192.168.0.1-254Then, edit your /etc/hosts file on the control node (whichever one you choose). Here's an example of mine:
127.0.1.1 node1 # There's already a localhost one, but add this as well
192.168.0.11 node1 node1.local # Control node
192.168.0.12 node2 node2.local
192.168.0.13 node3 node3.local
192.168.0.14 node4 node4.localNext, we need to generate an RSA key and distribute it to our worker nodes in order for us to issue ssh commands from inside our control node. I'd advise NOT reusing your own SSH key you're using right now already to connect to each of the nodes. Ideally, we'll be able to tell later if the intra-node SSH key was being used.
Run this on your control node Pi:
ssh-keygen -t ed25519 -C "ansible_key" -f ~/.ssh/anbsible_id_ed25519Then, copy it to your computer and then to other Pis in the cluster. This will allow the control node to ssh into them:
# Copy from control node Pi to computer
scp <your ssh username>@<your control node pi name>.local:~/.ssh/anbsible_id_ed25519.pub ~/ansible_id_ed25519.pub
scp <your ssh username>@<your control node pi name>.local:~/.ssh/anbsible_id_ed25519 ~/ansible_id_ed25519
# Copy from your computer to Pis
ssh-copy-id -i ~/ansible_id_ed25519.pub -f <your username>@node2Next, we're going to use a tool called Ansible to set up remote control over all our nodes. It will effectively allow us to issue install commands or customize all our nodes at once via single commands.
Run the following commands:
# You may need to run this as root
sudo -i
# Install ansible
apt install ansibleNext, you'll want to create a file called /etc/ansible/hosts and add all our hosts to it. We're defining hosts and groups of hosts that Ansible will try to manage for us:
# To edit the file (you make have to create this file)
sudo nano /etc/ansible/hosts
# File /etc/ansible/hosts
[control]
node1 ansible_connection=local
[workers]
node2 ansible_connection=ssh
node3 ansible_connection=ssh
node4 ansible_connection=ssh
[cube:children]
control
workersAbove, you can see I have added 3 groups: control, workers and cube. Name of the group is the one in between [ ]. This was split so that if I want to execute some actions only on control server, I use the “control” group. Group “cube” has children. This basically means that it’s a group of groups, and when I’m using cube I’m targeting every single node from the listed groups.
Variable: ansible_connection: we are telling Ansible how to connect to that host. The primary method is ssh, but I specified “local” for node1, because this is the node that we are running Ansible from. This way, it won’t try to ssh to itself.
Lastly, we are going to make it so that user root will be able to log in to other nodes from node1 without the password using an ssh key. This step is optional, but after this you won’t need to type the password every time you run Ansible.
Once this is set up, you can do a test run of Ansible with this command:
# Test pinging all control + worker nodes to verify they are setup correctly
ansible cube -m ping
## Response
node1 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python3"
},
"changed": false,
"ping": "pong"
}
node2 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python3"
},
"changed": false,
"ping": "pong"
}
node3 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python3"
},
"changed": false,
"ping": "pong"
}
node4 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python3"
},
"changed": false,
"ping": "pong"
}Also, install iptables on all of the nodes via this process (needed for k3s/Kubernetes):
# Install
ansible cube -m apt -a "name=iptables state=present" --become
# Reboot
ansible workers -b -m shell -a "reboot"
# Alternately, manually install on each node
apt -y install iptablesFinally, logout of the control node and get ready for the next section:
logoutBy the way, you can get CPUs temperature, RP1 I/O controller temperature, SSD temperature, and fan speed:
ansible all -m shell -a "sudo apt update -y && sudo apt install -y lm-sensors && yes | sudo sensors-detect && echo CPU: \$((\$(cat /sys/class/thermal/thermal_zone0/temp)/1000))°C && sensors 2>/dev/null || true"If you're just interested in core temps, use this command after installing lm-sensors on every node:
ansible cube -m shell -a "sensors"TODO: Add additional useful ansible commands (shoudl work out of the box if all other steps in this guide are followed) including:
# Get sensor temp readings (CPU, NVMe, memory)
ansible cube -m shell -a "sensors"
# Get sensor temp readings (CPU, NVMe, memory) + Nvidia GPU temp readings
ansible cube -m shell -a "sensors && nvidia-smi --query-gpu=temperature.gpu --format=csv,noheader"At any time during this guide, you can run the following commands to start/stop k3s:
# Stop k3s
sudo systemctl stop k3s
# Start k3s
sudo systemctl restart k3s
# Check status
sudo systemctl status k3sFirst, set memory constraints for every node (including control) via appending this to the end of /boot/firmware/cmdline.txt (you will need to open with sudo):
cgroup_enable=memory cgroup_memory=1 cgroup_enable=cpuset swapaccount=1
# Reboot when you're done
sudo rebootInstall k3s via this command:
# Install
curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644 --disable servicelb --token <some random password> --node-taint CriticalAddonsOnly=true:NoExecute --bind-address <control node ip address> --disable-cloud-controller --disable local-storage
# Verify the node taint with this
kubectl describe node node1 | grep -i taintIf you want to remove the taint (which prevents non-critical resources, like your apps, from being installed and having pods run on the control node), run this:
kubectl taint nodes <your control node name> CriticalAddonsOnly=true:NoExecute-Be sure to replace "some_random_password" with a password you save and preserve. You'll need this to connect to the main k3s master node. Replace the bind address flag IP with your control node's IP that you set earlier.
Once you run and install this via the curl command, check the installation with kubectl:
# Check nodes
kubectl get nodes
# Listing out
NAME STATUS ROLES AGE VERSION
node1 Ready control-plane,master 10m v1.33.3+k3s1After this, you'll need to install k3s onto your worker nodes which you can do with Ansible:
ansible workers -b -m shell -a "curl -sfL https://get.k3s.io | K3S_URL=https://<your control node IP>:6443 K3S_TOKEN=some_random_password sh -"Once you've done this, verify this worked via:
# Checking if it worked
kubectl get nodes
# Response with nodes in cluster
NAME STATUS ROLES AGE VERSION
node1 Ready control-plane,master 10m v1.33.3+k3s1
node2 Ready <none> 3m32s v1.33.3+k3s1
node3 Ready <none> 3m32s v1.33.3+k3s1
node3 Ready <none> 3m32s v1.33.3+k3s1If you're interested in total resources now in the cluster, you can check with this command:
# Long
kubectl get nodes -o custom-columns=NAME:.metadata.name,CPU:.status.capacity.cpu,MEM:.status.capacity.memoryYou should also probably label the other nodes, so do something like this:
# Labels for cosmetic reasons
kubectl label nodes node2 kubernetes.io/role=worker
kubectl label nodes node3 kubernetes.io/role=worker
kubectl label nodes node4 kubernetes.io/role=worker
kubectl label nodes node5 kubernetes.io/role=worker
kubectl label nodes node6 kubernetes.io/role=worker
# Labels used for directing deployments to prefer certain nodes
kubectl label nodes node2 node-type=worker
kubectl label nodes node3 node-type=worker
kubectl label nodes node4 node-type=worker
kubectl label nodes node3 node-type=worker
kubectl label nodes node6 node-type=workerVerify via showing all labels:
kubectl get nodes --show-labelsShow any taints with this command:
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints --no-headersLastly, change the source of the kubeconfig like so via Ansible:
ansible cube -b -K -m lineinfile -a "path='/etc/environment' line='KUBECONFIG=/etc/rancher/k3s/k3s.yaml'"This is the source of truth for each of the kube deployments (client and servers for control and workers).
From this point forward, you can get all your pods and states by using this:
kubectl get pods --all-namespaces -o wideIf you ever want to upgrade the version on each of the nodes, you can just run these commands:
# Upgrading control node
sudo k3s-uninstall.sh
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=<version-such-as-v1.33.5+k3s1> sh -
# Upgrading agents (workers) nodes
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=<version-such-as-v1.33.5+k3s1> K3S_URL=https://<server>:6443 K3S_TOKEN=<token> sh -Next, we need to install Helm in order to make use of Helm charts. Run this on the control node Pi:
curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bashThat's it! Onwards!
K3s comes with Traefik which is pretty great. However, we want to be able to assign an external IP to service (like our dashboards), and it's just not as customizable as we'd like.
Instead, let's move to using metallb as our load balancer for the cluster. Documentation
Run these commands on your control node to install it:
# First add metallb repository to your helm
helm repo add metallb https://metallb.github.io/metallb
# Check if it was found
helm search repo metallb
# Install metallb
helm upgrade --install metallb metallb/metallb --create-namespace \
--namespace metallb-system --waitThe command may take a second. You can check in another terminal tab while ssh-ed into your control node to check the installation process:
kubectl get pods -n metallb-systemOnce install is done, the command from above will finish. If it hands longer than ~5 minutes, you have networking issues you'll need to resolve and god help you.
Finally, apply this custum resource definition:
cat << 'EOF' | kubectl apply -f -
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: default-pool
namespace: metallb-system
spec:
addresses:
- 192.168.0.200-192.168.0.250 # replace this with your own IP range
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
name: default
namespace: metallb-system
spec:
ipAddressPools:
- default-pool
EOFNext, we're going to boostrap Longhorn for our file storage. This will enable us to use the NVMe drives on our Pis. We'll later move to using ArgoCD and Helm charts as the source of truth for these, but that will come later.
Run these on the control node:
ansible cube -b -K -m apt -a "name=nfs-common state=present"
ansible cube -b -K -m apt -a "name=open-iscsi state=present"
ansible cube -b -K -m apt -a "name=util-linux state=present"We'll be using Ansible a ton for this setup which will make all the pain there worth it.
Go ahead and run this command to see all disk labels:
ansible cube -b -m shell -a "lsblk -f"If you decided to go the route of NVMEs like I did, you'll actually need to format them.
First, set the below in your /etc/ansible/hosts:
sudo nano /etc/ansible/hosts
# File /etc/ansible/hosts
[control]
node1 ansible_connection=local var_hostname=node1 var_disk=<your nvme drive name here>
[workers]
node2 ansible_connection=ssh var_hostname=node2 var_disk=<your nvme drive name here>
node3 ansible_connection=ssh var_hostname=node3 var_disk=<your nvme drive name here>
node4 ansible_connection=ssh var_hostname=node4 var_disk=<your nvme drive name here>
[cube:children]
control
workersThen, run these commands (but triple check you set the right drives above beforehand):
# Wipe
ansible cube -b -K -m shell -a "wipefs -a /dev/{{ var_disk }}"
# Format to ext4
ansible cube -b -K -m filesystem -a "fstype=ext4 dev=/dev/{{ var_disk }}"Afterwards, get all drives and their available sizes with this command:
ansible cube -b -K-m shell -a "lsblk -f"
# Response
node1 | CHANGED | rc=0 >>
f1f2c384-4619-4a93-be82-42bbfee0269c
node2 | CHANGED | rc=0 >>
5002bfe6-dcd1-4814-85ab-54b9c0fe710e
node3 | CHANGED | rc=0 >>
4f92985d-5d8f-4429-ab2c-b10c650a5d0b
node4 | CHANGED | rc=0 >>
b114d056-c935-4410-b490-02a3302b38d2We'll get unique UUIDs for the drives in case the paths change. Let's go ahead and update our Ansible config to use these:
[control]
node1 ansible_connection=local var_hostname=node1 var_disk=<your nvme drive name here> var_uuid=<your drive UUID here>
[workers]
node2 ansible_connection=ssh var_hostname=node2 var_disk=<your nvme drive name here> var_uuid=<your drive UUID here>
node3 ansible_connection=ssh var_hostname=node3 var_disk=<your nvme drive name here> var_uuid=<your drive UUID here>
node4 ansible_connection=ssh var_hostname=node4 var_disk=<your nvme drive name here> var_uuid=<your drive UUID here>
[cube:children]
control
workersNext, we'll go ahead and mount the storage disks via this command:
ansible cube -b -K -m ansible.posix.mount -a "path=/storage01 src=UUID={{ var_uuid }} fstype=ext4 state=mounted"We'll now install longhorn to be able to interact with these drives. Fortunately, we can just use Helm again for this. We'll make ArgoCD pages to deploy it later:
# Run these on the control node
cd
helm repo add longhorn https://charts.longhorn.io
helm repo update
helm install longhorn longhorn/longhorn --namespace longhorn-system --create-namespace --set defaultSettings.defaultDataPath="/storage01" --version 1.9.1I had a lot of trouble getting this to work correctly and had to fully delete all the resources and reinstall several times.
You can check the pods for the deployment and the CRDs with these commands:
# Pods
kubectl -n longhorn-system get pod
# Response
NAME READY STATUS RESTARTS AGE
discover-proc-kubelet-cmdline 0/1 Completed 0 31s
engine-image-ei-b4bcf0a5-4c2tt 1/1 Running 0 96s
engine-image-ei-b4bcf0a5-fplzs 1/1 Running 0 96s
engine-image-ei-b4bcf0a5-nptbl 1/1 Running 0 96s
engine-image-ei-b4bcf0a5-ttmhg 1/1 Running 0 96s
instance-manager-4a529424618fd898f5b182591ed252d1 1/1 Running 0 63s
instance-manager-8acb062def932e1d509146c2b7564a73 1/1 Running 0 57s
instance-manager-f6f0478e3af0dd0843f02cb22a7f591b 1/1 Running 0 66s
instance-manager-f7413a220c4171cc02cca50a7900a7f1 1/1 Running 0 51s
longhorn-driver-deployer-5647c54d4f-mdhzv 1/1 Running 0 2m2s
longhorn-manager-96glw 2/2 Running 0 2m2s
longhorn-manager-dt58x 2/2 Running 0 2m2s
longhorn-manager-vclq5 2/2 Running 0 2m2s
longhorn-manager-vfm6f 2/2 Running 0 2m2s
longhorn-ui-8666455ff7-vnkm5 1/1 Running 0 2m2s
longhorn-ui-8666455ff7-zmlwv 1/1 Running 0 2m2s
# CRDs
kubectl get crds | grep engineimages
# Response
engineimages.longhorn.io 2025-08-19T20:30:35ZNext, apply a config for it here:
# On your control node
touch longhorn.yaml
# File contents
apiVersion: v1
kind: Service
metadata:
name: longhorn-ingress-lb
namespace: longhorn-system
spec:
selector:
app: longhorn-ui
type: LoadBalancer
loadBalancerIP: <one of your IPs from the metallb range here>
ports:
- name: http
protocol: TCP
port: 80
targetPort: http
# Then apply
kubectl apply -f longhorn.yamlBefore we finish, verify that Longhorn is now the default storage class:
# Run this command
kubectl get storageclass
# You'll get a response lik
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
longhorn (default) driver.longhorn.io Delete Immediate true 2m30s
longhorn-static driver.longhorn.io Delete Immediate true 2m27sFinally, I personally wanted my control node to be able to participate in distributed storage with its NVMe drive. I used this patch for the config to allow this even though the control node has a taint:
kubectl -n longhorn-system patch daemonset longhorn-manager \
-p '{"spec":{"template":{"spec":{"tolerations":[{"key":"CriticalAddonsOnly","operator":"Equal","value":"true","effect":"NoExecute"}]}}}}'This repo has a variety of services and architecture patterns. Initially on the Kube cluster, we'll want to bootstrap a ArgoCD installation which will then point to this repo and allow Argo to start deploying itself as well as other services. This repo has an app-of-apps architecture pattern for deployments.
First, create a namespace for ArgoCD (we'll later set up Terraform to handle this for us, but we're bootstrapping at the moment):
# Create the namespace
kubectl create namespace argocd
# Install ArgoCD
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
# Validate the install worked
kubectl get pods -n argocd
kubectl get svc -n argocdPatch this to be a LoadBalancer type deployment so you can have an external IP here as well for the dashboard:
kubectl patch service argocd-server -n argocd --patch '{ "spec": { "type": "LoadBalancer", "loadBalancerIP": "<your IP here>" } }'Once applied, you'll want to login. But you'll need to get the initial admin password (change this later):
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 --decode
echoNext, create an argocd_boostrap.yaml file in the control Pi. We're going to manually apply it to point to your gitops repository (realistically, a fork of this one):
touch argocd_bootstrap.yamlCopy in the content from this file
Then, apply it to have the rocket takeoff and bootstrap Argo:
kubectl apply -f argocd_boostrap.yamlTODO: Show setting up the external IP, logging in, and changing password + deploying more apps
Now that we have ArgoCD setup and you bootstrapped the application cluster, getting sealed secrets working should be as simple as deploying the kube-system namespace app and then deploying the sealed-secrets-app.
After you do that, you'll also need to install kubeseal, a CLI tool for encrypting secrets. You'll be using this locally to make your secret before putting it into Git to be committed.
Get the most recent version from here and then install via:
# Install via these commands
curl -sSL https://github.com/bitnami-labs/sealed-secrets/releases/download/v<version here>/kubeseal-<version here>-linux-arm64.tar.gz | tar -xz
sudo mv kubeseal /usr/local/bin/kubeseal
chmod +x /usr/local/bin/kubeseal
# Check it worked correctly
kubeseal --versionTo seal a secret, here's an example (outputting to YAML and then encoding) of creating one:
# Create the secret for one key-value
echo -n bar | kubectl create secret generic mysecret \
--from-file=foo=/dev/stdin \
--type=Opaque \
--namespace=<service namespace here> \
--dry-run=client \
-o yaml | yq '.metadata.name = "<secret name here>"' > mysecret.yaml
# Create the secret for multiple key-values
kubectl create secret generic <secret name> \
--from-literal=KEY1='value' \
--from-literal=KEY2='value' \
--type=Opaque \
--namespace=<service namespace here>\
--dry-run=client \
-o yaml | yq '.metadata.name = "<secret name here>"' > mysecret.yaml
# View the sealed secret
cat mysecret.yaml
# Encode the secret
kubeseal --controller-name=sealed-secrets \
--controller-namespace=kube-system \
--format yaml \
< mysecret.yaml \
> mysealedsecret.yaml
# Get the secret hash if that's all you need (as in my setups in this repo)
cat mysealedsecret.yaml | grep "<secret name here>:" | awk '{print $2}'The full sealed secret will end up looking something much like this:
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
creationTimestamp: null
name: mysecret
namespace: default
spec:
encryptedData:
foo: AgDEPWfG7iH8p2DBSqRGe+hVpRa1+d06hWffB1krTyF2iBpxTPY/rZw6Ba26dA+txlWYZN5uw/CxLyk+zs1WqU64qskHptC5dcbEuCPwXnZQbUL6x/HBzkr4sXwAcYGFKPXtCSG98o5E5F/Mx7PtFQAMcZ0Jo1e2OZt4vH07QMaDdTLwwPFrWGOiIcyOGJX/XFOeW/s7wGj31loIHi50uljGxCGns4l2DiU29mo7VSq4aHAOEWAM8jiyGPC8eapdrmYU2NpEBJJMAWwwsO6WkF6jAMIiDvKXMC1alYIYxIFJB7OcEyuLvddLbmn0fvh9hcQJe2bRSe/yT7AVvYoCWsSX1zji3IIPCzLTOHDzf74gsi2Gbt+FCyTYJNu2dzQpl5jIBctMwVOTF1H1154RFHyRoAGl5R3jgwbQ2kn1pK4O1w23FuLKHfLr8ExllPeoiSDykYetrvRuKcV2BUeAztbDs+aKXn4yfVHNpvryhyzEbm7804CyjDpSRkjC2tKcnrv1mEoCD5EyAqWvfIbFVnoj7mp8cOhLlDWz0cp32u1KAHxg21dK3K0XQfUDaqOiXa1TiBmGFedjyo/MpMKMbZTqs6TPpiEPiP6Eso9fr5u/9LQQY2V1eYpTChI8e824U3YUmo2ooC+GOGarUk5en1VQLC9yGf5XcppeZh23NAp9Egzlo3j+B25P2IEuLqiiqfyLLHc=
template:
data: null
metadata:
creationTimestamp: null
name: mysecret
namespace: defaultThis secret is encrypted by your sealed secret controller on your cluster and can be safely committed. No one except your controller can decrypt it. You can also apply this to your cluster and also commit the file (even publicly like in this repo).
If you want to just manually apply a secret in your cluster, you can do that like this:
kubectl create -f mysealedsecret.yamlRunning the above command creates a regular secret in your cluster (inside whatever namepsace you defined earlier) which is merely base64 encoded. The sealed secrets controller in your cluster decrypts your encryptedData.
You can also view these secrets once applied via:
kubectl get secret <secret-name> -n <namespace>You should also get your master key and store it wherever you also store major secrets:
kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml >master.keyIf you need to redeploy your cluster, you'll need this in order to use the same secrets. If that happens, you'll also need to apply the old master key which you can do via these commands:
# Apply the master key
kubectl apply -f master.key
# Delete the old pod so that it will restart and read the new key
kubectl delete pod -n kube-system -l name=sealed-secrets-controllerTODO: Describe setting up Terraform-based applying of secrets from files in the repo (will use Atlantis)
To temporarily turn off a Kube node, use this command:
kubectl cordon <node-name>To turn it back on, use:
kubectl uncordon <node-name>On the control node, run this:
# Uninstall K3s
sudo /usr/local/bin/k3s-uninstall.sh
# Remove K3s binaries
sudo rm -rf /usr/local/bin/k3s
sudo rm -rf /usr/local/bin/k3s-agent
# Remove K3s data
sudo rm -rf /etc/rancher/k3s
sudo rm -rf /var/lib/rancher/k3s
sudo rm -rf /var/lib/kubelet
sudo rm -rf /var/lib/cni
sudo rm -rf /run/k3s
sudo rm -rf /var/run/k3sOn the worker nodes, run this:
# Uninstall K3s
sudo /usr/local/bin/k3s-agent-uninstall.sh
# Remove K3s binaries
sudo rm -rf /usr/local/bin/k3s
sudo rm -rf /usr/local/bin/k3s-agent
# Remove K3s data
sudo rm -rf /etc/rancher/k3s
sudo rm -rf /var/lib/rancher/k3s
sudo rm -rf /var/lib/kubelet
sudo rm -rf /var/lib/cni
sudo rm -rf /run/k3s
sudo rm -rf /var/run/k3sGood news! The hardest parts are behind us. Or rather, I should say that the hardest parts are behind you. You don't want to know how much time I burned making the charts in this repository work correctly.
However, you can now utilize ArgoCD to do all the hard parts for you. Merely go to your ArgoCD dashboard, sync the monitoring namespace app, and then sync all the sub-apps. Boom. Now you have monitoring in your cluster.
Until I get sealed secrets validated in how it works with non-environment variable secrets (e.g. for port numbers and such in Argo/Helm), I'm manually applying LoadBalancer configurations for Prometheus and Grafana to access them.
Adapt these with your own IP addresses and then access them at the IP + port you've set them to:
# For Prometheus
apiVersion: v1
kind: Service
metadata:
name: prometheus-external
namespace: monitoring
spec:
selector:
prometheus: prometheus-persistent
type: LoadBalancer
ports:
- name: web
protocol: TCP
port: 9090
targetPort: web
loadBalancerIP: <IP address that falls within the metallb range you previously set>
# For Grafana
apiVersion: v1
kind: Service
metadata:
name: grafana
namespace: monitoring
spec:
selector:
app: grafana
type: LoadBalancer
ports:
- name: http
port: 3000
targetPort: http
loadBalancerIP: <IP address that falls within the metallb range you previously set>
Make files for each of those and then use kubectl apply -f <file name> to get a working URL you can use (e.g. http://<the IP you set>"<the port you set>).
The default username/password for Grafana is admin and admin, but you should change the password immediately on accessing the dashboard.
Once logged in, go to add a new data source here:
Get the IP and port for your Promethues instance using this command:
root@control01:~/monitoring/grafana# kubectl get services -n monitoring
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
.
.
prometheus-external LoadBalancer 10.43.167.23 192.168.0.205 9090:32545/TCP 12d
prometheus ClusterIP 10.43.108.243 <none> 9090/TCP 12d
.
.In the above example with demo data, you'd use 10.43.108.24:9090 as your IP and port for the data source.
From there, select Prometheus as your new data source:
Add the IP and port we just got to this input:
Scroll to the bottom and click "Save and Test." From here, you should be goop to add dashboards. You can do this by going to this location:
You can then choose to import a pre-built dashboard (although you can definitely build your own later if you want):
From here, use numbers from the Prometheus website to import pre-built websites:
I like these pre-built dashboards:
6417 - Kubernetes Cluster (Prometheus)
7249 - Kubernetes Cluster
8171 - Kubernetes Nodes
13032 - LonghornYou can search through all the community premade dashboards on the Prometheus website here.
Once you enter the number and click "Load", finalize with this screen:
Great job! Let's move on to metrics!
As it turns out, we're not quite done with our monitoring configuration. K3s doesn't quite work as easily with our monitoring setup as we'd like given it's a lighter weight setup than full K8s.
To fix this, we need to add Grafana Loki and Alloy.
TODO: Describe setting up Loki and Alloy.
This repository already has a full configuration for n8n. Once you have ArgoCD functional, you should be able to just go to the n8n application and deploy it. The only caveat is that you'll need to manually set a LoadBalancer deployment type in order to view it in the browser. You can kubectl apply a file with this in it:
apiVersion: v1
kind: Service
metadata:
name: n8n-app-loadbalancer
namespace: applications
spec:
selector:
app.kubernetes.io/name: n8n
app.kubernetes.io/instance: n8n-app
type: LoadBalancer
loadBalancerIP: <your-ip-address-here>
ports:
- name: http
protocol: TCP
port: 80
targetPort: httpTODO: Update with information on sealed secret for IP load balancer address once that's functional.
The last thing that you should do is hop into the ArgoCD dashboard (via whatever IP you set it to for external) and deploy all the root/namespaces/apps there.
You'll also end up redeploying ArgoCD there, so it's now fully in control of its own deployment as well.
First, you need to set up the right settings in your device's OS (the nest steps commands are for Ubuntu).
To do this, begin by running these commands locally on the node to set up the OS WOL:
# All steps pulled from https://pimylifeup.com/ubuntu-enable-wake-on-lan/
# Get the network interface name
nmcli connection showThat should output someting like this:
NAME UUID TYPE DEVICE
Wired connection 1 fd179af5-6b4d-35aa-97c4-3e14bfe9ee81 ethernet enp1s0 Also, get the network adapter MAC address for later use:
nmcli device show "<NETWORK DEVICE NAME>" | grep "GENERAL.HWADDR"These should give you the interface name and MAC address which we'll be using shortly. Also, you'll want the broadcast address for your subnet (e.g. take the IPs from your nodes and append 255 at the end, such as 192.168.1.255).
Next, let's check out existing WOL setting:
# CONNECTION NAME = something like "Wired connection 1" from earlier
nmcli connection show "<CONNECTION NAME>" | grep 802-3-ethernet.wake-on-lanThat should output something like this:
802-3-ethernet.wake-on-lan: default
802-3-ethernet.wake-on-lan-password: --Let's update it to be enabled with this command:
nmcli connection modify "<CONNECTION NAME>" 802-3-ethernet.wake-on-lan magicIf you check the WOL setting again, it should say "magic" which means it's enabled.
Next, go into your BIOS and disable the settings for ErP Ready and enable "Resume by PCI-e/Networking Devices". This is different for every motherboard, so I'll leave it up to you to google through enabling BIOS WOL for your motherboard. There's usually a wealth of resources out there.
Once all of this is complete, you can add this alias to your terminal config on any machine on the same network that you want to use WOL from:
alias wol<node-name-here>="wakeonlan -i <broadcase-ip-address-for-subnet> <device-mac-address>"If you're using MacOS, you can install the wakeonlan package (from the alias above) to run it:
brew install wakeonlanWe're done! If everything works right, we should just be able to run this in our terminal any time we want to start our node up (with the node name replaced here from your alias):
wol<node-name-here>As an aside, you can also add an alias on the node in question to shut it down when you're ssh-ed into it:
alias shutdown="systemctl poweroff"Once you're done with all of the above, you'll need to go into your BIOS and ensure the following settings are setup:
- ErP is disabled
- Wake on PCIe is enabled
Follow Nvidia's setup steps here for nvidia-container-toolkit:
https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html
TODO
Test with containerd that the Nvidia GPU is available:
sudo ctr image pull docker.io/nvidia/cuda:12.3.2-base-ubuntu22.04
sudo ctr run --rm --gpus 0 -t docker.io/nvidia/cuda:12.3.2-base-ubuntu22.04 cuda-12.3.2-base-ubuntu22.04 nvidia-smiIf there are any issues with containerd, make sure that the following is set up for the /etc/containerd/config.d/99-nvidia.toml file:
version = 2
[plugins]
[plugins."io.containerd.grpc.v1.cri"]
[plugins."io.containerd.grpc.v1.cri".containerd]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia]
privileged_without_host_devices = false
runtime_root = ""
runtime_type = "io.containerd.runc.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia.options]
BinaryName = "/usr/bin/nvidia-container-runtime"