Kind cluster creation failing because of rootless provider and systemd property Delegate=yes

Question

As a newbie to Podman and Kind, I'm having a very hard time running Kind in a rootless way on RHEL. It would be very helpful if someone had a clue.

When I run

$ kind create cluster
enabling experimental podman provider
ERROR: failed to create cluster: running kind with rootless provider requires setting systemd property "Delegate=yes", see https://kind.sigs.k8s.io/docs/user/rootless/
the cluster creation fails.

I have to mention that my home directory is mounted on a NFS. But I've modified the path to podman container storage.conf runroot & graphroot

I have followed the documentation here: https://kind.sigs.k8s.io/docs/user/rootless/

1 - I enabled cgroup2 by editing and applying changes in /etc/default/grub

$ cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto resume=UUID=c6b70b51-8dca-4e70-aa69-5e4fd2265281 rhgb quiet systemd.unified_cgroup_hierarchy=1"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true

2 - I added the Delegate=yes

$ cat /etc/systemd/system/[email protected]/delegate.conf
[Service]
Delegate=yes

But I'm still having the error.

When running sudo kind create cluster, it works fine.

System Information

OS: RHEL 8.6 (linux/amd64)

$ cat /etc/redhat-release
Red Hat Enterprise Linux release 8.6 (Ootpa)

Architecture

$ uname -m
x86_64

Podman version : 7.0.2

$ podman version
Client:       Podman Engine
Version:      4.0.2
API Version:  4.0.2
Go Version:   go1.17.7
Built:      Fri Apr  1 18:29:15 2022
OS/Arch:    linux/amd64

Kind version: 0.20.0 (linux/amd64)

$ kind version
kind v0.20.0 go1.20.4 linux/amd64

systemd infos

NOTE: systemd user directory is on an NFS-mounted volume

Systemd Version

$ systemctl --version 
systemd 239 (239-78.el8)
+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy

User Service Status

$ systemctl --user status user@$(id -u).service
Warning: The unit file, source configuration file or drop-ins of [email protected] changed on disk. Run 'systemctl --user daemon-reload'>
● [email protected]
   Loaded: bad-setting (Reason: Unit [email protected] has a bad unit file setting.)
  Drop-In: /home/users/wli7/.config/systemd/user/[email protected]
           └─delegate.conf
   Active: inactive (dead)

Dec 13 17:45:34 dell5340dsy systemd[3453]: [email protected]: Service has no ExecStart=, ExecStop=, or SuccessAction=. Refusing.
Dec 13 17:45:37 dell5340dsy systemd[3453]: [email protected]: Service has no ExecStart=, ExecStop=, or SuccessAction=. Refusing.
Dec 13 17:46:22 dell5340dsy systemd[3453]: [email protected]: Service has no ExecStart=, ExecStop=, or SuccessAction=. Refusing.
Dec 13 17:48:21 dell5340dsy systemd[3453]: [email protected]: Service has no ExecStart=, ExecStop=, or SuccessAction=. Refusing.
Dec 13 17:49:17 dell5340dsy systemd[3453]: [email protected]: Service has no ExecStart=, ExecStop=, or SuccessAction=. Refusing.
Dec 13 17:49:44 dell5340dsy systemd[3453]: [email protected]: Service has no ExecStart=, ExecStop=, or SuccessAction=. Refusing.
Dec 13 18:04:05 dell5340dsy systemd[3453]: [email protected]: Service has no ExecStart=, ExecStop=, or SuccessAction=. Refusing.

Delegate settings

$ systemctl --user show user@$(id -u).service | grep Delegate

Delegate=yes
DelegateControllers=cpu cpuacct cpuset io blkio memory devices pid

s

Kind & Podman

podman version

$ podman version

Client:       Podman Engine
Version:      4.0.2
API Version:  4.0.2
Go Version:   go1.17.7

Built:      Fri Apr  1 18:29:15 2022
OS/Arch:    linux/amd64

Podman info

$ podman info

host:
  arch: amd64
  buildahVersion: 1.24.1
  cgroupControllers: []
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.0-1.module+el8.6.0+14673+621cb8be.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: 9d06693cb3781a627d20b8fdfd07be19072471ca'
  cpus: 32
  distribution:
    distribution: '"rhel"'
    version: "8.6"
  eventLogger: file
  hostname: dell5340dsy
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 50734
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 4.18.0-513.5.1.el8_9.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 61376577536
  memTotal: 66916802560
  networkBackend: netavark
  ociRuntime:
    name: runc
    package: runc-1.0.3-2.module+el8.6.0+14673+621cb8be.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.3
      spec: 1.0.2-dev
      go: go1.17.7
      libseccomp: 2.5.2
  os: linux
  remoteSocket:
    path: /home/WS/wli7/systemd/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.8-2.module+el8.6.0+14673+621cb8be.x86_64
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 16757288960
  swapTotal: 16757288960
  uptime: 41m 12.59s
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/users/wli7/.config/containers/storage.conf
  containerStore:
    number: 5
    paused: 0
    running: 0
    stopped: 5
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /home/duplo/wli7/podman/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 4
  runRoot: /home/duplo/wli7/podman/run/containers/storage
  volumePath: /home/duplo/wli7/podman/containers/storage/volumes
version:
  APIVersion: 4.0.2
  Built: 1648830555
  BuiltTime: Fri Apr  1 18:29:15 2022
  GitCommit: ""
  GoVersion: go1.17.7
  OsArch: linux/amd64
  Version: 4.0.2

kind version

$ kind version
kind v0.20.0 go1.20.4 linux/amd64

Information about cgroup2

$ mount | grep cgroup2
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)

Answer 1

Base on the docs you can check or do the following:

1. After modifying the path /etc/systemd/system/[email protected]/delegate.conf

[Service] Delegate=yes
You should do a sudo systemctl daemon-reload

2. Check the file /etc/modules-load.d/iptables.conf if the following content existed:

ip6_tables
ip6table_nat
ip_tables
iptable_nat

Note that there are restrictions regarding NFS on the docs.

Also, you might find this link helpful regarding your issue.