The situation is as follows:
I have a PHP website that connects to an Amazon like API to retrieve data of several thousants products.
For each product i get an XML. I 'file_get_contents' that XML and echo the data on my website. On the fly, so no backing up / saving those XML to my webserver. If the data (like stock info) changes at the remote site (Amazon like), the stockinfo is immediately visible on my website.
I hope you understand my situation. No the problem:
Since a few days, the Amazon like website returns a 'Too Many Requests 429' error and no data is on my website any more.
For 2 years there was no problem. But now i think the Amazon like website tighten there rate limits OR i get more and more visitors (real visitors or searchbot).
The rate limit at Amazon like is set to 1200 request per hour. My website got more than 2000 hits in 1 hour so i understand the couse of the error.
Now... what is the best way to avoid future errors?
Do i need to store those thousand XML files to my webserver or in a database so i do not have to connect to the API so many times.
Or is there any other solution? Maybe a time out? Or should i 'ask' google to not index my website so many times? Or can it be done with caching in httaccess maybe? If yes... could you please give me a piece of code i can use to help me out?
I use the following code. And all 2000 hits in 1 hour use this code:
$url = "https://amazonlikewebsite.com/catalog/v4/lists/?ids=123&limit=30&dataoutput=refinements&apikey=xxxxxxxxxxxxxxxxxxxxxxxxxxx&format=xml";
$content = file_get_contents($url);
$xml = simplexml_load_string($content);
foreach ($xml->products as $product) {
// echo data of the products
}