deserialization error when calling method in python rq distributed queue

Question

I am trying to test how to pass a python object to a rq worker process. I have the following classes in common.py

class Input:
    def __init__(self, arr_list: List):
        self.arr_list = arr_list

    def compute_sum(self):
        sum = 0
        for num in self.arr_list:
            sum += num
        time.sleep(10)
        return Output(sum)


class Output:
    def __init__(self, sum_result):
        self.sum_result = sum_result

    def __str__(self):
        return str(self.sum_result)


def calculate_sum(input_obj: Input) -> Output:
    return input_obj.compute_sum()

I am calling this method calculate_sum to calculate sum of 100 elements using rq.

from common import Input, calculate_sum


class TestRqJobSharding:

    def test_job_submission(self):
        index = 0
        batch = 10
        list_obj = [1] * 100
        jobs = []
        results = []
        while index < len(list_obj):
            new_batch = list_obj[index: index + batch]
            input_obj = Input(new_batch)
            job = get_queue().enqueue(calculate_sum, args=(input_obj,))
            index = index + batch
            jobs.append(job)
        results = submit_and_wait_till_completed(jobs)
        for result in results:
            print(f'{result}')
        print(results)

I am seeing the following deserialization error in worker process.

Traceback (most recent call last):
  File "/home/sshil/venv3.7/lib/python3.7/site-packages/rq/worker.py", line 1056, in perform_job
    self.prepare_job_execution(job)
  File "/home/sshil/venv3.7/lib/python3.7/site-packages/rq/worker.py", line 937, in prepare_job_execution
    self.procline(msg.format(job.func_name, job.origin, time.time()))
  File "/home/sshil/venv3.7/lib/python3.7/site-packages/rq/job.py", line 297, in func_name
    self._deserialize_data()
  File "/home/sshil/venv3.7/lib/python3.7/site-packages/rq/job.py", line 265, in _deserialize_data
    raise DeserializationError() from e
rq.exceptions.DeserializationError
Traceback (most recent call last):
  File "/home/sshil/venv3.7/lib/python3.7/site-packages/rq/job.py", line 262, in _deserialize_data
    self._func_name, self._instance, self._args, self._kwargs = self.serializer.loads(self.data)
ModuleNotFoundError: No module named 'common'

I tried to use Json serializer, I am still seeing this error. This happens when I pass a python object. It works fine with int or string objects.

Answer 1

I had to set PYTHONPATH environment variable. It was not able to find the python packages. It has nothing to do with rq serialization