Why mannually setting the vlan id to the VF attached to a VM will not work?

Question

The working config is like this:

    <interface type='hostdev' managed='yes'>
  <mac address='52:54:00:6a:9d:7f'/>
  <driver name='vfio'/>
  <source>
    <address type='pci' domain='0x0000' bus='0xaf' slot='0x02' function='0x1'/>
  </source>
  <vlan>
    <tag id='4001'/>
  </vlan>
  <alias name='hostdev0'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>

If we use this config, the VMs can ping each other.

But , if we remove the vlan tag in the xml.

    <interface type='hostdev' managed='yes'>
  <mac address='52:54:00:6a:9d:7f'/>
  <driver name='vfio'/>
  <source>
    <address type='pci' domain='0x0000' bus='0xaf' slot='0x02' function='0x1'/>
  </source>
  <!--vlan>
    <tag id='4001'/>
  </vlan-->
  <alias name='hostdev0'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>

And then we try to set the vlan tag with ip command, it will not work, I don't know why, I think the IP command should work too, and the vlan id in the ip link command output is the same.

10: eth8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 89:41:33:95:5f:22 brd ff:ff:ff:ff:ff:ff
vf 0     link/ether 53:52:01:20:d9:da brd ff:ff:ff:ff:ff:ff, vlan 4001, spoof checking off, link-state auto, trust on
vf 1     link/ether 53:52:01:9c:d4:36 brd ff:ff:ff:ff:ff:ff, vlan 4001, spoof checking off, link-state auto, trust on