These are my personal notes while performing the installation. Instructions might be incomplete, dangerous or wrong.
Cleanest solution: Mount NFS on host and use hostpath on pod
Also: Would be good to export all k3s state, maybe backups are better than full nfs export
dom0$ cat libvirt-net-private.xml <network> <name>private</name> <bridge name='virbr1' stp='on' delay='0'/> <ip address='169.254.169.1' netmask='255.255.255.0'> <dhcp> <range start='169.254.169.10' end='169.254.169.254'/> </dhcp> </ip> </network>
virsh net-define ./libvirt-net-private.xml virsh net-autostart private virsh net-start private
Add to vbox
virsh edit k3s <interface type='bridge'> <source bridge='virbr1'/> <model type='virtio'/> </interface>
virsh shutdown k3s virsh start k3s # shows eth1 ip l # configure network auto eth1 iface eth1 inet static address 169.254.169.3 netmask 255.255.255.0 gateway 169.254.169.1
Ping 169.254.169.1 does not work from k3s Check arp on dom0 (works):
$ arp -i virbr1 169.254.169.3 ether 52:54:00:f0:60:24 C virbr1
My guess is that firewall is preventing traffic.
dom0$ iptables -P INPUT ACCEPT
Ping works -> add iptable rules
Add to firewall definition:
IVMS="virbr1" iptables -A INPUT -i $IVMS -j ACCEPT
UUID=7a8730cf-4185-4d56-b1e9-d053a7b17a50 /mnt/red1_vms btrfs rw,noatime,nodiratime,compress=lzo,space_cache,subvol=@vms 0 0 /mnt/red1_vms/storage/k3s /export/k3s_store none bind 0 0
# Reload nfs exportfs -ra
k3s$ mount 169.254.169.1:/export/k3s_store /mnt/nfs # Does not work connection refused k3s$ apk add nfs-utils k3s$ mount 169.254.169.1:/export/k3s_store /mnt/nfs # works now
Add correct fstab automount: /etc/fstab
169.254.169.1:/export/k3s_store /mnt/k3s_store nfs auto,rw,_netdev 0 0
$ rc-update add nfsmount $ rc-service nfsmount start $ reboot
Checklist - Mounted with rw - Exported with rw - dom0 can write on it
Problem with mount directory is root owned and client-root is trying to write at it. With NFSs default setting root_squash (man exports), the root user is mapped to another uid.
We could disable the root uid remapping:
This handles the k3s root user the same as the dom0 root user. Another approach is to chmod a+rwx the desired directory and face the situation, that uid/gid 0 from the guest is mapped to nobody:nobody on the filesystem. Since many containers are running as root and security is not a real concern here, we disable root squashing with no_root_squash for now.
<filesystem type='mount' accessmode='mapped'> <source dir='/mnt/red1_vms/storage/k3s'/> <target dir='k3s_store'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </filesystem>
k3s_store /mnt/k3s_store 9p trans=virtio 0 0
Does not work as expected, because file permissions are mapped. Only virtualized filesystem. Problems with k3s setup (I/O errors).