How to block requests while analyzing 1000 links?

140 Views Asked by At

How i can block "grequests" when i found "target file"? I need to analyze like 1000 links, but just one is that good.

If I have found that link after e.g. 10 URLs, I need to block the grequests but for now him still continue until it analyze all the links.

How I can edit or fix it? Thanks

#!/usr/bin/python

import grequests

urls = [
'http://example.com/1.php',
'http://example.com/2.php',
'http://example.com/3.php',
'http://example.com/4.php'
]


def do_something(response, *args, **kwargs):
    print response
    print response.url
    if "html" in response.text:
        print response.url
    
    
async_list = []

for u in urls:
    action_item = grequests.get(u, timeout=10, hooks = {'response' : do_something})
    async_list.append(action_item)

grequests.map(async_list,size=10)
0

There are 0 best solutions below