What happens i run 2 goroutines to modify a VM and destory the same VM using libvirt-go package on the same instance?

Question

As we know, libvirt is thread-safe. But running two goroutines concurrently acting on the same resource, such as modifying and deleting a VM leaves it in an ambiguous state. How does libvirt decide the order of execution of goroutines?

Here's the code for what i tried:

package main

import (
    "fmt"

    "github.com/libvirt/libvirt-go"
)

func main() {
    conn, err := libvirt.NewConnect("qemu:///system")
    if err != nil {
        fmt.Printf("Failed to connect to libvirt: %v\n", err)
        return
    }
    defer conn.Close()

    // Create a new VM
    domainXML := `
        <domain type='kvm'>
            <name>myvm</name>
            <memory unit='KiB'>1048576</memory>
            <vcpu placement='static'>1</vcpu>
            <os>
                <type arch='x86_64' machine='pc-i440fx-2.9'>hvm</type>
                <boot dev='hd'/>
            </os>
            <devices>
                <disk type='file' device='disk'>
                    <driver name='qemu' type='qcow2'/>
                    <source file='path/to/disk'/>
                    <target dev='vda' bus='virtio'/>
                    <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
                </disk>
            </devices>
        </domain>`

    dom, err := createVM(conn, domainXML)
    if err != nil {
        fmt.Printf("Failed to create VM: %v\n", err)
        return
    }

    go modifyVMMemory(dom, 2*1024*1024) // 2 GiB

    go deleteVM(dom)

}

func createVM(conn *libvirt.Connect, domainXML string) (*libvirt.Domain, error) {
    dom, err := conn.DomainCreateXML(domainXML, 0)
    if err != nil {
        return nil, err
    }
    return dom, nil
}

func modifyVMMemory(dom *libvirt.Domain, newMemory uint64) error {
    err := dom.SetMaxMemory(newMemory)
    if err != nil {
        return err
    }
    fmt.Print("Modified VM")
    return nil
}

func deleteVM(dom *libvirt.Domain) error {
    err := dom.Destroy()
    if err != nil {
        return err
    }

    err = dom.Undefine()
    if err != nil {
        return err
    }

    fmt.Print("Deleted VM")

    return nil
}

The program completes successfuly, hence the domain is destroyed and could be recreated but running this again results in the following error:

virError(Code=9, Domain=20, Message='operation failed: domain 'myvm' already exists with uuid 32c25acb-a4c5-4bfd-b2f5-f07b3d9b8eea')

Answer 1

As we know, libvirt is thread-safe. But running two goroutines concurrently acting on the same resource, such as modifying and deleting a VM leaves it in an ambiguous state. How does libvirt decide the order of execution of goroutines?

Thread-safety just means that the code is not going to have memory corruption problems when multiple threads use the same connection concurrently.

The semantic behaviour you are going to get will still be non-deterministic.

The libvirt QEMU/KVM driver uses an RPC layer between the client app and libvirtd (or virtqemud) daemons. So first you have non-determinism in which Goroutine runs first. When the libvirt-go-module APIs call into the C ibvirt.so library via CGo, they'll get locked to native OS threads, which will then synchronize inside libvirt.so to decide which gets to put its RPC message on the wire first. In the libvirtd daemon there are also many threads, and RPC messages are notionally processed FIFO, however, then the API logic inside libvirtd is still going to race for acquiring locks and so add more non-determinism when talking to/interacting with QEMU. Basically your SetMaxMemory and Destroy API calls can run in either order. If you need guaranteed ordering, you need to serialize them in your app, such that you only call Destroy after you've finished SetMaxMemory

Finally IIUC your Go code isn't robust as the main() method is spawning two goroutines, but is not waiting for either of them to complete. IOW, the Go process may well be exiting, before either goroutine gets to fully run. This is likely why you're getting the error message about the VM already existing - the deleteVM goroutine never got to run before the process exited.

Answer 2

You're running modifyVMMemory and deleteVM within a goroutine, and returning the error without checking it.

errCh := make (chan error, 1)
go func () {
    err := modifyVMMemory(dom, 2*1024*1024) // 2 GiB
    errCh <- err
}()

err := <- errCh
if err != nil {
    // handle the error
}

go func() {
    err := deleteVM(dom)
    errCh <- err
}()

err = <-errCh
if err != nil {
    // handle the error
}

Also, a tread-safe library does not necessarily mean the underlying software, like qemu is tread-safe.