I have a Common Lisp program that behaves differently depending on how I use *standard-input*
. Here are the details:
(if input-stream?
(process)
(with-open-file (*standard-input* up :element-type 'unsigned-byte)
(process)))
The process
function starts multiple threads. Each thread reads part of the standard input, writes it in a file (all within a lock) and processes the resulting files in parallel (out of the lock). In fact it processes the resulting files in parallel only in case input-stream?
is false
otherwise it processes them sequentially.
(defun process ()
(let ((psize 4194304)
(stream *standard-input*)
(result-lock (bt:make-lock))
(input-stream-lock (bt:make-lock))
eof)
(flet ((add-job (fname)
(make-thread
#'(lambda ()
(do () (eof)
(when (bt:with-lock-held (input-stream-lock)
(unless eof
(setq eof (write-input-stream-to-file stream fname psize))
t))
(sleep 0.1)
(bt:with-lock-held (result-lock)
(display-progress))))))))
(mapcar
#'join-thread
(loop for i from 1 to 10
collect (add-job
(make-pathname :directory "/tmp"
:name "test"
:type (princ-to-string i))))))))
(let ((counter 0))
(defun display-progress ()
(if (zerop (mod (incf counter) 10))
(format t " ~a " counter)
(write-char #\+))))
(defun write-input-stream-to-file (stream fname psize-bytes)
(with-open-file (out fname
:direction :output
:element-type 'unsigned-byte
:if-exists :supersede)
(do ((byte (read-byte stream nil nil)
(read-byte stream nil nil))
(offset 0 (1+ offset)))
((or (= offset psize-bytes) (null byte)) (not byte))
(write-byte byte out))))
If we create a FIFO (with mkfifo), copy the file to it and run the program with it instead, we again observe parallelism.
The above program is built as a command line utility with ECL and runs on Linux. I run it in one of the following ways:
- cat "bigfile" | my-program
- my-program "bigfile"
Parallelism happens only in case 2.
The question is why the difference?
Update:
- I had mistake in my question. Now it is OK.
- Added the
process
function and described how I run the program