Get directory entries for "slow protocols" asynchronously

Question

I want a function for getting directory entries on Linux. I use ioutil.ReadDir and usually it is fast. But if I want to read some mounted virtual file system on /run/user/1000/gvfs/, this function becomes slow. If the directory has many file entries I need to wait a long time.

I can use the ls command in a terminal and result will be the same. When I tried ls -U -a -p -1 I got line by line output immediately. I tried running this in Go with exec.Command, but it didn't work asynchronously. Go is waiting for full program output. What did I do wrong?

m.cmd = exec.Command("ls", "-U", "-a", "-p", "-1")
// for example some "slow" directory:
m.cmd.Dir = "/run/user/1000/gvfs/dav:host=webdav.yandex.ru,ssl=true,user=...../" 
reader, _ := m.cmd.StdoutPipe()
bufReader := bufio.NewReader(reader)
go func() {
  m.cmd.Start()
  for {
    line, _, err := bufReader.ReadLine()
    if err != nil {
      break
    }
    linestr := string(line)
    if linestr != "./" && linestr != "../" {
      fmt.Println(linestr)
    }
  }
}()

I need line by line printing immediately in Go.

Answer 1

Try ls -U -a -p 1 | cat to see if you get line-by-line output.

Go doesn't control ls; ls does line-by-line writing if ls chooses to do so, and ls chooses not to do that when its output is a pipe. You could allocate a pty pair and use that, but that's the wrong way to do this.

ioutil.ReadDir first reads the entire directory (by calling Readdir(-1)), then sorts the file names. If you use os.Open to open the directory, then call the Readdir or Readdirnames function with a small (but not negative) number, you should get something more to your liking.