Rq: Insert Multiple rows

53 Views Asked by At

I have a 5gb file. I want to extract from each row some info to store into a db. I tried to iterate over all the file rows, and to insert one by one by it takes more than 2 days. I was thinking to use an "asynchronous approach" with rq ( I want to divide the rows in n queue of rows_num / n jobs). This is what I tried:

def do_work(file_name, queue_length,func,redis_connection): 
    with open(file_name, 'r') as f: 
        queue_num = 1
        q = Queue(str(1), connection=redis_connection)
        rows = []
        for i, line in enumerate(f): 
            if i // queue_length == queue_num:
                rows.append(Queue.prepare_data(func, (line,), job_id=str(i)))
            else:
                q.enqueue_many(rows)          
                q = Queue(str(queue_num), connection=redis_connection)
                rows = [Queue.prepare_data(func, (line,job_id=str(i)))]
        q.enqueue_many(rows)

If I am not mistaken, this will not work as I expect (not in an asynchronous way) as it will execute the first queue than when it finishes, it will execute the second one, then the third one and so on.

How can I reach what I am expecting?

0

There are 0 best solutions below