How to get a PHP script to execute while receiving a file?

Question

A desktop application uploads a file with arbitrary length (it really could be anything from a few MB to multiple GB) to a PHP endpoint.

The PHP server in question is running in Docker. Specifically, it's the Apache 2 + PHP combination (no FastCGI).

While uploading the file using the desktop application I observe following things:

• The network statistics show that there is network transmit activity of about 600 MB/s (fluctuates quite a bit) during the upload
• During the upload, the desktop application shows a progress bar indicating how much the upload has progressed. It aligns with the upload speed observed independently.
• There is no output in access/error log of Apache 2 nor is there any log output on PHP side during the upload
• I know that the PHP script is not running because I have added an error_log line at the very beginning and the log is not written to error log during upload.
• The error_log line is executed as soon as the upload reaches 100%. Network activity drops. Now the line appears in the error log of apache and the script starts executing.
• The request is sent with Transfer-Encoding: chunked. I checked this in the logs.
• RAM usage stays about the same during upload – only increases a bit during script execution.

Based on this I assume there are two places that could be causing this:

• Apache buffering the data before handing it off to PHP
• PHP buffering the data before starting script execution

This is an issue because the script uses php://input to process the upload and since it can be multiple GB, I would like the script to already start processing the data before the upload finished (let's leave the possibility of abrupt connection abortion aside).

How can I circumvent this?

I have searched everywhere on how to disable input buffering and similar but haven't found anything that would help. Everything only points to output buffering or adding lines to the script but that does not really help since the script only start execution when the file is uploaded so it can't be the code.

Answer 1

I'm going to answer my own question with the solution that I've come up with.

This will only work with POSIX systems (ones that support FIFOs/named pipes).

First, create a FIFO and start reading from it:

posix_mkfifo("/path/to/fifo");

$file = fopen("/path/to/fifo", "rb");
while (!feof($file)) {
  // fread etc.
}
fclose($file);

// You might want to keep reopening the fifo, depending on how you want to implement the write part

The script will start reading from the FIFO. Now all you need to do is write to this FIFO.

You could use a separate PHP script where you submit one chunk (i. e. 64 MB) per request and use regular fopen, fwrite to write to this FIFO.

Alternatively, you can also write a separate service in a different langauge (like Go, Rust, Kotlin, or whatever suits you) that writes into this FIFO in a continuous stream. (This is what I did)

PHP does not support streaming the request body as it is being uploaded by the client. The entire request is read into memory, always. So the solution is to delegate the upload implementation to a different script or service and only handle reading the FIFO in the long-running request.

---

The code example above is just a very simple example to illustrate the idea. Of course error checking, locking, etc. needs to be done here.