I'd like to build a RSS data retrieval process with PHP, MySQL, SimplePie, and jQuery, however I encounter problems with a crashing browser, limited server memory and error 500 when trying to retrieve and process a lot of data from external sources.
Is there a way to use jQuery (or another way) to "slow down" and "segment" the data retrieval and processing steps, i.e. can jQuery "split up" or "control" PHP output results into chunks, and force PHP to only process to the next remaining data retrieval chunk once the first has been finished, so that PHP has time to clear up server memory in between?
Basically my process is the following:
- get source xml feeds from database (MySQLi WHILE loop)
- initiate simplepie rss
- FOREACH feed, get latest rss items (title, content, image, date, etc)
- FOREACH rss item, check for images inside rss content element
- FOREACH rss item, check external source html for more images (via file_get_contents)
- check if images actually exist (getimagesize, get_headers)
Because I have a lot of FOREACH loops inside the MySQLi WHILE loop itself, the entire operation takes very long and is very memory intensive.
Example code:
$sql = "SELECT * FROM sources ORDER BY id DESC";
$query = mysqli_query($connection, $sql) or die (mysqli_error());
while ($row = mysqli_fetch_array($query)) {
$feed_url = $row["feed_url"];
$feed = new SimplePie();
$feed->set_feed_url($feed_url);
$feed->enable_cache(false);
$feed->init();
$feed->handle_content_type();
foreach ($feed->get_items(0, 20) as $item){
$link = $item->get_permalink();
$homepage = file_get_contents($link);
$doc = new DOMDocument();
libxml_use_internal_errors(true);
$doc->loadHTML($homepage);
libxml_clear_errors();
$tags = $doc->getElementsByTagName('img');
foreach ($tags as $tag) {
$image_url = $tag->getAttribute('src');
// process image further or return image here
} // end foreach tag
$tags = $doc->getElementsByTagName('a');
foreach ($tags as $tag) {
$image_url = $tag->getAttribute('href');
// process image further or return image here
} // end foreach tag
... more foreach loops here
} // end foreach rss item
} // end mysqli while
Is there a way to use jQuery to return data back to AJAX throughout the FOREACH loops and "control" the remaining PHP loops so that the server is not overloaded?