Not overwriting *standard-input* prevents multi-threading

110 Views Asked by At

I have a Common Lisp program that behaves differently depending on how I use *standard-input*. Here are the details:

(if input-stream?
    (process)
    (with-open-file (*standard-input* up :element-type 'unsigned-byte)
      (process)))

The process function starts multiple threads. Each thread reads part of the standard input, writes it in a file (all within a lock) and processes the resulting files in parallel (out of the lock). In fact it processes the resulting files in parallel only in case input-stream? is false otherwise it processes them sequentially.

(defun process ()
  (let ((psize 4194304)
        (stream *standard-input*)
        (result-lock (bt:make-lock))
        (input-stream-lock (bt:make-lock))
        eof)
    (flet ((add-job (fname)
             (make-thread
              #'(lambda ()
                  (do () (eof)
                    (when (bt:with-lock-held (input-stream-lock)
                            (unless eof
                              (setq eof (write-input-stream-to-file stream fname psize))
                              t))
                      (sleep 0.1)
                      (bt:with-lock-held (result-lock)
                        (display-progress))))))))
      (mapcar
       #'join-thread
       (loop for i from 1 to 10
          collect (add-job
                   (make-pathname :directory "/tmp"
                                  :name "test" 
                                  :type (princ-to-string i))))))))

(let ((counter 0))
  (defun display-progress ()
    (if (zerop (mod (incf counter) 10))
        (format t " ~a " counter)
        (write-char #\+))))

(defun write-input-stream-to-file (stream fname psize-bytes)
  (with-open-file (out fname
                       :direction :output
                       :element-type 'unsigned-byte
                       :if-exists :supersede) 
    (do ((byte (read-byte stream nil nil)
               (read-byte stream nil nil))
         (offset 0 (1+ offset)))
        ((or (= offset psize-bytes) (null byte)) (not byte))
      (write-byte byte out))))

If we create a FIFO (with mkfifo), copy the file to it and run the program with it instead, we again observe parallelism.

The above program is built as a command line utility with ECL and runs on Linux. I run it in one of the following ways:

  1. cat "bigfile" | my-program
  2. my-program "bigfile"

Parallelism happens only in case 2.

The question is why the difference?

Update:

  • I had mistake in my question. Now it is OK.
  • Added the process function and described how I run the program
0

There are 0 best solutions below