Well life catches up with you fast I guess. I haven’t posted any updates because I was posting as I was building and I hit a roadblock. I’ll first talk about that in case Google gets you here while you’re running into the same problem and then give the pieces I did figure out.
Docker Swarm in Proxmox LXC Containers
Docker swarm can run real easily in VM’s. Install docker, create a swarm, add nodes to the swarm, toss some stuff in it, you’re basically done. Within an LXC container, some restrictions are going to give you problems. For the pct.conf file, I ended up with this configuration that works.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 arch: amd64 cores: 2 hostname: swarm-wrkr3-lxc memory: 2048 mp0: /lib/modules/4.15.18-11-pve,mp=/lib/modules/4.15.18-11-pve,ro=1 net0: name=eth0,bridge=vmbr0,gw=192.168.1.1,hwaddr=C6:1F:1A:5A:A2:83,ip=192.168.1.4/22,type=veth onboot: 1 ostype: ubuntu rootfs: local-lvm:vm-111-disk-0,size=25G swap: 512 unprivileged: 0 lxc.apparmor.profile: unconfined lxc.cgroup.devices.allow: a lxc.cap.drop:
The important bits are the ones at the end. Proxmox provides a handful of app armor profiles that allow you to run certain applications that would be blocked from running inside a container. Unconfined isn’t great from a security perspective, but since docker will still isolate processes from the LXC container, you do preserve some level of isolation. This profile allows you to get a container running to begin with.
After running my docker-ce and swarm install playbooks, things would look up and running, but containers would always fail to launch on lxc worker nodes. My docker-compose file spun up traefik and another network to allow things to talk to each other without being exposed publicly. A
docker network ls on my worker showed:
1 2 3 4 5 6 root@swarm-wrkr3-lxc:~# docker network ls NETWORK ID NAME DRIVER SCOPE 3bca5c1794b6 bridge bridge local 6ec1975c097b host host local f6c66b4943e1 none null local root@swarm-wrkr3-lxc:~#
But listing the networks on the manager, running in a virtual machine, showed:
1 2 3 4 5 6 7 8 9 10 root@swarm-mgr-vm1:~# docker network ls NETWORK ID NAME DRIVER SCOPE 6cee66aec471 bridge bridge local a7cd74d3722a docker_gwbridge bridge local mq2gpjfvhgz0 exposed overlay swarm b4ef8f4e3714 host host local 1mkakpah4inc ingress overlay swarm fa853d2b4f5e none null local 7ozgltfb5q9j sample overlay swarm xwyi8zbxy488 traefik overlay swarm
A significant number of networks were not being created. A
docker info on the worker showed why.
1 2 3 4 5 6 7 8 9 <SNIP> Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false Product License: Community Engine WARNING: bridge-nf-call-iptables is disabled WARNING: bridge-nf-call-ip6tables is disabled
I have ipv6 disabled on my proxmox hosts, so the ipv4 bit is the one causing problems. Docker needs iptables to route traffic appropriately and manage your network traffic. The problem is with the linux kernel, not anything I was doing (which was frustrating to find out now). The problem is out of my area of expertise, but the br_netfilter kernel module doesn’t handle namespaces correctly.
This issue in lxd is where I finally found someone describing it and the fix. That leads you here and this discussion in lkml. The short of it is theres a proposed solution, some back and forth on it, and more work to do.
So what now?
I have my swarm running VMs now. It works, traefik serves the traffic for all my hosts, I watch stuff with portainer, and just have a few services in there to do random things. I will continue to run it in my VM’s until the changes are merged into the kernl and migrate to LXC from there. Fortunately, I can add nodes and remove them as I scale services up and down. I’ll update this when the feature is available.