PRAW Reddit API - How do I speed up this web scraping code?

45 Views Asked by At

I'm gathering recent cross posts of about a 1000 popular subreddits, But the code takes a while to run.. How do I speed up this process? Please help!

submission_data = []
sub_count = 0
for sub in popsublist:
    count = 0
    sub_count += 1
    print('============================')
    print('subs-looped count:',sub_count)
    print('current sub:',sub)
    print('============================')
    sub_loop = 0
    for post in reddit.subreddit(sub).hot(limit=500):
        sub_loop += 1
        print("posts-looped count",sub_loop)
        if hasattr(post, "crosspost_parent"):
            count += 1
            print('posts-loop count [ADDED!]:',count)
            op = reddit.submission(id=post.crosspost_parent.split("_")[1]).subreddit
            submission_data.append({
                'SOURCE_SUB': str(post.subreddit),
                'TARGET_SUB': str(op),
                'POST_ID': str(post.id),
                'POST_TITLE': str(post.title),
                'POST_DATE': datetime.utcfromtimestamp(int(post.created_utc)).strftime('%Y-%m-%d %H:%M:%S'),
                'POST_LINK': str('http://www.reddit.com'+post.permalink),
                'POST_SCORE': post.score,
                'POST_NSFW': post.over_18,
            })

This is my code. How do I accelerate this process?

0

There are 0 best solutions below