The configuration of this project is done via files in the ./vars directory. The inventory is composed of .ini files in the ./vars directory. Each .ini file represents an inventory and can be used with the -i flag when running playbooks.

The variables for the hosts and groups are defined in the ./vars/group_vars directory. The structure of this directory is as follows:

vars/
├── group_vars/
│   ├── all/
│   │   ├── secrets.yml
│   │   └── vars.yml
│   ├── <group_name>/
│   │   ├── *.yml
├── docker.ini
├── k3s.ini
├── kubernetes.ini
├── proxmox.ini
└── vps.ini

The all group contains variables that are common to all hosts. Each other directory in group_vars corresponds to a group defined in the inventory files and contains variables specific to that group.

Run Playbook

To run a playbook, you need to specify the inventory file and the playbook file. For example, to run the k3s-servers.yml playbook with the k3s.ini inventory, you can use the following command:

ansible-playbook -i vars/k3s.ini playbooks/k3s-servers.yml

After successful k3s installation

To access our Kubernetes cluster from our host machine to work on it via flux and such we need to manually copy a k3s config from one of our server nodes to our host machine. Then we need to install kubectl on our host machine and optionally kubectx if we're already managing other Kubernetes instances. Then we replace the localhost address inside of the config with the IP of our load balancer. Finally we'll need to set the KUBECONFIG variable.

mkdir ~/.kube/
scp k3s-server00:/etc/rancher/k3s/k3s.yaml ~/.kube/config
chown $USER ~/.kube/config
sed -i "s/127.0.0.1/192.168.20.22/" ~/.kube/config
export KUBECONFIG=~/.kube/config

Install flux and continue in the flux repository.

Longhorn Nodes

To create longhorn nodes from existing kubernetes nodes we want to increase their storage capacity. Since we're using VMs for our k3s nodes we can resize the root-disk of the VMs in the proxmox GUI.

Then we have to resize the partitions inside of the VM so the root partition uses the newly available space. When we have LVM-based root partition we can do the following:

# Create a new partition from the free space.
sudo fdisk /dev/sda
# echo "n\n\n\n\n\nw\n"
# n > 5x\n > w > \n
# Create a LVM volume on the new partition
sudo pvcreate /dev/sda3
sudo vgextend  k3s-vg /dev/sda3
# Use the newly available storage in the root volume
sudo lvresize -l +100%FREE -r /dev/k3s-vg/root

Cloud Init VMs

# On Hypervisor Host
qm resize <vmid> scsi0 +32G
# On VM
sudo fdisk -l /dev/sda # To check
echo 1 | sudo tee /sys/class/block/sda/device/rescan
sudo fdisk -l /dev/sda # To check
# sudo apt-get install cloud-guest-utils
sudo growpart /dev/sda 1