Mockserver implementation fails with connection refused error

Question

We have implemented a mock server for unit tests in golang. If we run the test cases which use mockserver. it is inconsistent. Sometimes it fails with this error

dial tcp 127.0.0.1:46686: connect: connection refused

For port , we have this code :

listener, err := net.Listen("tcp", ":0")
    if err != nil {
        t.Log("Unable to initiate Listener", err)
    }
    port := listener.Addr().(*net.TCPAddr).Port
    err = listener.Close()
    if err != nil {
        t.Log("Unable to close Listener", err)
    }

we have two such mock servers for two packages. It consistently fails in gitlab pipeline when we run go test.

Can anyone suggest ways on how to prevent this error ?