Using New York Times API with no returns

Question

I have tried to fetch news articles from New York Times but there were no return. No response at all. I troubleshooted and tested the URL on web browser, it does return the news articles. And on the NYT developer's 'PATHS' page, it also returned 200 for my API. I wonder if it's because the news articles I want from the past two months are too large, and the sleep time is not enough to limit the API rate? Can someone please help me with my code?

api_key = "my_NYT_api_key"
query = "Israel Palestine"
filter_query = "news_desk:(\"Foreign\"OR \"Politics\" OR \"World\") AND type_of_material:(\"News\" OR \"Article\") AND pub_date:([2023-10-07 TO 2023-11-21]) "
page = "0"
sort = "relevance"
facet_fields = "news_desk,section_name"
def send_request(query, filter_query, facet_fields, api_key):
   base_url = "https://api.nytimes.com/svc/search/v2/articlesearch.json"
   url = (f"{base_url}?q={query}&fq={filter_query}&facet_fields=facet_fields}&page={page}&sort={sort}&"
          f"facet=true&facet_filter=true&api-key={api_key}")
   headers = {
       'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36',
       'Content-Type': 'application/x-www-form-urlencoded;charset=UTF-8'
   }
   response = requests.get(url, headers=headers)
   print("HTTP Status Code:", response.status_code)  # Check HTTP status
   print("Raw Response:", response.text)  # Print raw response text
   time.sleep(6)
   return response.json()


def parse_response(response):
   data = {
       'headline': [],
       'pub_date': [],
       'web_url': [],
       'image_url': [],
       'timestamp': [],
   }

   articles = response['response']['docs']
   for article in articles:
       data['headline'].append(article['headline']['main'])
       data['pub_date'].append(article['pub_date'])
       data['web_url'].append(article['web_url'])
       data['timestamp'].append(article['pub_date'])

       multimedia = article.get('multimedia', [])
       image_url = multimedia[0]['url'] if multimedia else 'No image'
       data['image_url'].append(image_url)

   return pd.DataFrame(data)
def get_data(query, filter_query, facet_fields, api_key):
   response = send_request(query, filter_query, facet_fields, api_key)
   if response:  # Check if response is not empty
       df = parse_response(response)
       print(f"Number of articles collected: {len(df)}")
       return df
   else:
       print("No data received")
       return pd.DataFrame()

I wonder if it is because I have requested too many times in a day (although never returned any articles)? That is why NYT is limiting the rate from my api key?

Answer 1

i had the same problem. In the FAQ of NYTs API you find this:

11. Is there an API call limit? Yes, there are two rate limits per API: 500 requests per day and 5 requests per minute. You should sleep 12 seconds between calls to avoid hitting the per minute rate limit. If you need a higher rate limit, please contact us at [email protected].