How can I ensure that this Python RQ task on a Redis server is being executed?

483 Views Asked by At

In a Flask app, I have a function that takes a list of lists and adds each as a row to an Excel file. This function works fine on a development server when the request won't time out. Here it is wrapped in a try/except block. This is how I need to call it on the prod server as a queued task.

def make_excel_file(file_title, lists_to_print):

  try:
    job = get_current_job()

    # returns a list of lists
    # the first list is the header row (titles of attributes)
    # the following lists are all lists of the relevant object attributes
    # as well as some joined object attributes
    # (e.g. user ID, user email, drive year)
    rows = wish_list_rows(lists_to_print)

    wb = Workbook()
    ws = wb.active

    tempfile = NamedTemporaryFile()
    tempfile.name = file_title

    num_rows = len(rows)

    _set_task_progress(0, "excel_progress") #This results in a valid notification
                                            # so I know the function reaches at least this point


    for r in rows:
      ws.append(r)
      wb.save(tempfile)
      tempfile.seek(0)

      _set_task_progress(100.0 * (num_rows/rows.index(r)), "excel_progress") #the number never
                                                                             #updates, so the for loop
                                                                             #must never begin
    stream = tempfile.read()

    response = make_response(stream)
    response.headers['Content-Disposition'] = "attachment; filename={}".format(
      file_title)
    response.mimetype = 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'
  except:
    flash("Lists could not be exported to Excel.", "warning")
  finally:
    _set_task_progress(100, "excel_progress")
  if response:
    return response

Using the local server's Redis CLI and the command redis-cli monitor, I can see the following output:

~$ redis-cli monitor
OK
1594476959.432554 [0 [::1]:61680] "HGETALL" "rq:job:cfabaad5-b586-4aba-90b1-61addb5c9ea9"
1594476959.485371 [0 [::1]:61680] "MULTI"
1594476959.485408 [0 [::1]:61680] "LREM" "rq:queue:spur-hcd-tasks" "1" "cfabaad5-b586-4aba-90b1-61addb5c9ea9"
1594476959.487109 [0 [::1]:61680] "EXEC"
1594476976.799187 [0 [::1]:61680] "MULTI"
1594476976.801143 [0 [::1]:61680] "SADD" "rq:queues" "rq:queue:spur-hcd-tasks"
1594476976.803906 [0 [::1]:61680] "HSET" "rq:job:18be4075-8e3c-409c-a5cf-f82f3d11ba42" "status" "queued"
1594476976.803958 [0 [::1]:61680] "HMSET"
"rq:job:18be4075-8e3c-409c-a5cf-f82f3d11ba42"
"created_at" "2020-07-11T14:16:15.310435Z"
"data" "x\x9c\x94\xbd[\xcc / GIANT BYTE STRING / "
"origin" "name-of-queue"
"description" "app.tasks.make_excel_file(1, file_title='All lists', lists_to_print=[\n            column_1_data, (\"column_2_data\"), column_3_data, column_5_data.\n            , \n..." "enqueued_at" "2020-07-11T14:16:16.749772Z"
"started_at" "" "ended_at" ""
"timeout" "180" "status" "queued"
1594476976.841367 [0 [::1]:61680] "RPUSH" "rq:queue:spur-hcd-tasks" "18be4075-8e3c-409c-a5cf-f82f3d11ba42"
1594476976.841410 [0 [::1]:61680] "EXEC"
1594476977.433481 [0 [::1]:61680] "HGETALL" "rq:job:18be4075-8e3c-409c-a5cf-f82f3d11ba42"

I'm not sure how to interpret this. I also would like help understanding how to further debug and view what is happening in that function on the local Redis server.

Edit: I see now that there may be something wrong with the queue that is configured to the app. When I get jobs from RQ using get_current_job(), I can access it. However, the registries for the app's queue are all empty. What would prevent the app's queue from having the job, if it does actually exist on the Redis server both in the redis-cli and for Python RQ?

0

There are 0 best solutions below