PHP Curl api post request gives back 25 results only, while there might be more

Question

I've wanted to scrape some useful data from a site. But the request only returned 25 results

with this:

    $url = 'https://api.test.org';
    $ch = curl_init();

    $jsonData = array(
        'limit' => 100, //user inputs pages * 5
        'listType' => 'taskSolutions',
        'task' => $taskid //taken from input user substr($_POST['link'],28);
        //'skip' => 25 $variable that increases by 25
    ); 

    curl_setopt($ch, CURLOPT_URL, $url);
    $jsonDataEncoded = json_encode($jsonData);
    curl_setopt($ch, CURLOPT_POST, 1);
    curl_setopt($ch, CURLOPT_POSTFIELDS, $jsonDataEncoded); // loop adding 25 each time to skip
    curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: application/json'));
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$data = json_decode(curl_exec($ch),true);

Now I looked at the website and they have the parameter 'skip' to get more results.

But now for the question:

How could I make a loop that adds 25 to the skip $variable and resends the CURLOPT_POSTFIELDS and add that data to $data

A variable $totalcount is available to check howmuch records there are.

Answer 1

You can do this with a loop. For instance, put the code above in a function called getData and pass it two arguments, $skip and $taskId :

function getData($skip, $taskid)
{
    $url = 'https://api.test.org';
    $ch = curl_init();

    $jsonData = array(
        'limit' => 100, //user inputs pages * 5
        'listType' => 'taskSolutions',
        'task' => $taskid //taken from input user substr($_POST['link'],28);
        'skip' => $skip
    );

    curl_setopt($ch, CURLOPT_URL, $url);
    $jsonDataEncoded = json_encode($jsonData);
    curl_setopt($ch, CURLOPT_POST, 1);
    curl_setopt($ch, CURLOPT_POSTFIELDS, $jsonDataEncoded); // loop adding 25 each time to skip
    curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: application/json'));
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    return json_decode(curl_exec($ch),true);
}

You can then write a loop to increment the $skip variable by 25 until it reaches $totalCount. In each iteration, add the returned elements to the $data array :

$data = [];
for($skip = 0; $skip < $totalCount; $skip += 25)
{
    foreach(getData($skip, $taskid) as $entry)
    {
        $data[] = $entry;
    }
}