How to scrape multiple pages using the apple i tunes api

Question

I'm trying to scrape the Itunes API to get information for all of the podcasts available in the Apple iTunes store. Currently, I'm only able to pull 200 at a time. When I try to grab the next 200 podcasts in the list, I'm getting the same 200 as before.

https://itunes.apple.com/search?term=podcast&limit=2 https://itunes.apple.com/search?term=podcast&limit=2&offset=1

Any suggestions would be appreciated.

import requests
import pandas as pd
import time
import json

url = 'https://itunes.apple.com/search?term=podcast&limit=2' 
res = requests.get(url,headers={'User-agent': 'project'})
res.status_code
current_url = None
posts = []
the_offset = 0
for _ in range(2):
    if current_url == None:
        current_url = url

    else:
        current_url = url +'&offset={}'.format(the_offset)
        res = requests.get(current_url)
    if res.status_code != 200:
        print('Error',res.status_code)
        break
    the_offset += 1
    current_dict = res.json()
    current_posts = {k:v for (k,v) in current_dict.items()}
    posts.extend(current_posts['results'])
    print(current_url)
    time.sleep(3)

Answer 1

Try changing the offset parameter:

results = 100
limit = 10

pages = int(results / limit)

for i in pages:
    offset = i+1
    request(offset)