Read from live data feed php

Question

I am using something called DAP (https://github.com/rapid7/dap) which helps deal with large file handling and outputs an ever growing list of data.

For example:

curl -s https://scans.io/data/rapid7/sonar.http/20141209-http.gz | zcat | head -n 10 | dap json + select vhost + lines

This code correctly works and it will output 10 lines of IP addresses.

My question is how can I read this data from PHP - in effect where a data feed is continuous/live (it will end at some point) how can I process each line I'm given?

I've tried piping to it but I don't get passed the output. I don't want to use exec because the data is constantly growing. I think it could be a stream but not sure that is the case.

Answer 1

For anyone else that finds themselves in the same situation - here is the answer that works for me (can be run directly from the command line also):

curl -s 'https://scans.io/data/rapid7/sonar.http/20141209-http.gz' | zcat | head -n 1000 | dap json + select vhost + lines | while read line ; do php /your_script/path/file.php $line ; done

Then pull out $argv[1] and the data is all yours.