How to issue a command that produces infinite output and return immediately

Question

When I write the following code (in ammonite but i don't think it matters)

("tail -f toTail.txt" lineStream) foreach(println(_)), the program give me the last line as intend but then hang, and even if i write more in the file, nothing come out.

How does the API support process that have unbounded output ?

I try to write val myStream = ("tail -f toTail.txt" lineStream_!) but it still does not return write away

Here is what the scala doc says:

lineStream: returns immediately like run, and the output being generated is provided through a Stream[String]. Getting the next element of that Stream may block until it becomes available.

Hence i don't understand why it blocks

By the way i am having exactly the same behavior with the Ammonite API

If type %%("tail", "-f", "toTail.txt") once again the method just hang and does not return immediately.

Answer 1

There is no issue with the ProcessBuilder (at least not one that stems from your use case). From the ProcessBuilder documentation:

Starting Processes

To execute all external commands associated with a ProcessBuilder, one may use one of four groups of methods. Each of these methods have various overloads and variations to enable further control over the I/O. These methods are:

• run: the most general method, it returns a scala.sys.process.Process immediately, and the external command executes concurrently.
• !: blocks until all external commands exit, and returns the exit code of the last one in the chain of execution.
• !!: blocks until all external commands exit, and returns a String with the output generated.
• lineStream: returns immediately like run, and the output being generated is provided through a Stream[String]. Getting the next element of that Stream may block until it becomes available. This method will throw an exception if the return code is different than zero -- if this is not desired, use the lineStream_! method.

The documentation clearly states that lineStream might block until the next line becomes available. Since the nature of tail -f is an infinite stream of lines lineBreak the program will block waiting for the next line to appear.

For the following assume I have a file: /Users/user/tmp/sample.txt and it's contents are:

boom
bar
cat

Why lineStream_! isn't wrong

import scala.language.postfixOps
import scala.sys.process._

object ProcessBuilder extends App {

  val myStream: Stream[String] = ("tail /Users/user/tmp/sample.txt" lineStream_!)
  println("I'm after tail!")
  myStream.filter(_ != null).foreach(println)
  println("Finished")
  System.exit(0)

}

Outputs:

I'm after tail!
boom
bar
cat
Finished

So you see that the lineStream_! returned immediately. Because the nature of the command is finite.

How to return immediately from a command that produces infinite output:

Let's try this with tail -f. You need more control over your process. Again, as the documentation states:

If one desires full control over input and output, then a scala.sys.process.ProcessIO can be used with run.

So for an example:

import java.io.{BufferedReader, InputStreamReader}

import scala.language.postfixOps
import scala.sys.process._

object ProcessBuilder extends App {

  var reader: BufferedReader = _
  try {

    var myStream: Stream[String] = Stream.empty
    val processIO = new ProcessIO(
      (os: java.io.OutputStream) => ??? /* Send things to the process here */,
      (in: java.io.InputStream) => {

        reader = new BufferedReader(new InputStreamReader(in))
        myStream = Stream.continually(reader.readLine()).takeWhile(_ != "ff")
      },
      (in: java.io.InputStream) => ???,
      true
    )
    "tail -f /Users/user/tmp/sample.txt".run(processIO)
    println("I'm after the tail command...")

    Thread.sleep(2000)
    println("Such computation performed while tail was active!")

    Thread.sleep(2000)
    println("Such computation performed while tail was active again!")

    println(
      s"Captured these lines while computing: ${myStream.print(System.lineSeparator())}")

    Thread.sleep(2000)
    println("Another computation!")

  } finally {
    Option(reader).foreach(_.close())
  }
  println("Finished")
  System.exit(0)

}

Outputs:

I'm after the tail command...
Such computation performed while tail was active!
Such computation performed while tail was active again!
boom
bar
cat

It still returns immediately and now it just hangs there waiting for more input. If I do echo 'fff' >> sample.txt from the tmp directory, the program outputs:

Another computation!
Finished

You now have the power to perform any computation you want after issuing a tail -f command and the power to terminate it depending on the condition you pass to the takeWhile method (or other methods that close the input stream).

For more details on ProcessIO check the documentation here.

Answer 2

I think, this has to do with how you are adding data to the file, not with the ProcessBuilder. If you are using it in an editor for example to add data, it rewrites the entire file every time you save, with a different inode, and tail isn't detecting that.

Try doing tail -F instead of tail -f, that should work.