How to fix hanging popen3 in Ruby?

Question

I am getting unexpected behaviour using popen3, which I want to use to run a command like tool ala cmd < file1 > file2. The below example hangs, so that stdout done is never reached. Using other tools than cat may cause hanging, so that stdin done is never reached. I suspect, I am suffering from buffering, but how do I fix this?

#!/usr/bin/env ruby

require 'open3'

Open3.popen3("cat") do |stdin, stdout, stderr, wait_thr|
  stdin.puts "foobar"

  puts "stdin done"

  stdout.each_line { |line| puts line }

  puts "stdout done"

  puts wait_thr.value
end

puts "all done"

Answer 1

stdout.each_line is waiting for further output from cat because cat's output stream is still open. It's still open because cat is still waiting for input from the user because its input stream hasn't been closed yet (you'll notice that when you open cat in a terminal and type in foobar, it will still be running and waiting for input until you press ^d to close the stream).

So to fix this, simply call stdin.close before you print the output.

Answer 2

Your code is hanging, because stdin is still open!

You need to close it with IO#close or with IO#close_write if you use popen3.

If you use popen then you need to use IO#close_write because it only uses one file descriptor.

 #!/usr/bin/env ruby
 require 'open3'

 Open3.popen3("cat") do |stdin, stdout, stderr, wait_thr|
   stdin.puts "foobar"

   stdin.close   # close stdin like this!  or with stdin.close_write

   stdout.each_line { |line| puts line }

   puts wait_thr.value
 end

See also:

Ruby 1.8.7 IO#close_write

Ruby 1.9.2 IO#close_write

Ruby 2.3.1 IO#close_write

Answer 3

The answers by Tilo and by sepp2k are right: If you close stdin, your simple test will end. Problem solved.

Though in your comment to the answer of sepp2k, you indicate that you still experience hangs. Well, there are some traps that you might have overlooked.

Stuck on full buffer for stderr

If you call a program that prints more to stderr than the buffer of an anonymous pipe can hold (64KiB for current Linuxes), the program gets suspended. A suspended program neither exits nor closes stdout. Consequently, reading from its stdout will hang. So if you want to do it right, you have to use threads or IO.select, non-blocking, unbuffered reads in order to read from both stdout and stderr in parallel or by turns without getting stuck.

Stuck on full buffer for stdin

If you try to feed more (much more) than "foobar" to your program (cat), the buffer of the anonymous pipe for stdout will get full. The OS will suspend cat. If you write even more to stdin, the buffer of the anonymous pipe for stdin will get full. Then your call to stdin.write will get stuck. This means: You need to write to stdin, read from stdout and read from stderr in parallel or by turns.

Conclusion

Read a good book (Richards Stevens, "UNIX Network Programming: Interprocess communications") and use good library functions. IPC (interprocess communications) is just too complicated and prone to indeterministic run-time behavior. It is for too much hassle to try to get it right by try-and-error.

Use Open3.capture3.