SwifType Rate limit exceeded

Question

I currently have a demo account. I am trying to make a POC using SwifType for my employer. We have a very big database that is indexec every 1h and creates a JSON file. I thought the integration with Elastic would be very easy considering it's only a mater of sending the string when it's generated. I used PHP Curl and got a connection to the API. The code sends out part of the data and then freaks out with a "Rate limit exceeded" error.

How can I manage around that error and get the full JSON indexed?

My code looks like this at the moment:

// SENDING DATA TO ELASTIC SEARCH
$arr = array_change_key_case($arr, CASE_LOWER); // Keys to lower case
$arrlist = array_chunk($arr,100); // Split to chunks of 100
foreach($arrlist as $key=>$arr){
    $json = json_encode($arr); // Making the JSON string from the array
    $ch = curl_init('https://host-***.api.swiftype.com/api/as/v1/engines/***/documents');                                                                      
    curl_setopt($ch, CURLOPT_POST, 1);                                                                     
    curl_setopt($ch, CURLOPT_POSTFIELDS, $json);                                                                  
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);      
    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);                                                                
    $headers = array();
    $headers[] = 'Content-Type: application/json';
    $headers[] = 'Authorization: Bearer private-***';
    curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);

    $result = curl_exec($ch);
    if (curl_errno($ch)) {
        echo 'Error:' . curl_error($ch);
    }
    curl_close($ch);

    echo $result."<hr>";
}

Also, considering this code is going to be indexed every hour, if I am sending the same data over and over, will it UPDATE the previous one or will it duplicate it? If so, how can I manage that?

Answer 1

You need to look at the X-RateLimit-Remaining header (see details here) when it hits 0 (or gets close to it) or when you receive a 429 response, you sleep() and then retry. I'd recommend using exponential backoff if you receive multiple 429 responses in a row (double the sleep time every time you receive a 429, reset it down to default value when your request succeeds).