Request status update Twitter stream data

Question

I retrieved Twitter data via the streaming API on Python, however, I am also interested in how the public metrics evolve during the time. As a result, I would like to request on a daily basis the metrics.

Unfortunately, the API for the status update can only handle 100 requests at a time. I have a list of all id's, how is it possible to automatically split the string of id's so that all of them will be requested, always in batches of 100?

Thank you a lot in advance!

Answer 1

Keep it as list of IDs instead of single string.

And then you can use range(len(...)) with [n:n+100] like

# example data
all_ids = list(range(500))

SIZE = 100
#SIZE = 10  # test on smaller size

for n in range(0, len(all_ids), SIZE):
    print(all_ids[n:n+SIZE])

You can even use yield to create special function for this

def split(data, size):
    for n in range(0, len(data), size):
        yield data[n:n+size]

# example data
all_ids = list(range(500))

SIZE = 100
SIZE = 10
    
for part in split(all_ids, SIZE):
    print(part)

---

Eventually you can get [:100] and slice [100:] but this destroy list so you have to do it on copy of this list

# example data
all_ids = list(range(500))

SIZE = 100
#SIZE = 10  # test on smaller size

all_ids_copy = all_ids.copy()

while all_ids_copy:
    print(all_ids_copy[:SIZE])
    all_ids_copy = all_ids_copy[SIZE:]

---

You can also use some external modules for this.

from toolz import partition

# example data
all_ids = list(range(500))

SIZE = 100
#SIZE = 10  # test on smaller size
 
for part in partition(SIZE, all_ids):
    print(part)

---

If you will have list of strings then you can convert back to single string using join()

    print( ",".join(part) )

For list of integers you may need to convert integers to strings

    print( ",".join(str(x) for x in part) )