Passing multiple arguments to retry_on_exception argument of retrying in python

Question

I have a usecase where a method which adds a row to a table needs to be retried in case of any exception. I am using @retry decorator of retrying to achieve this.

One of the cases is when there is a change of password of the db. I am using the arg retry_on_exception to catch whatever exception. The issue I am facing is, I want the password fetch to happen in the method I am passing to retry_on_exception.

My code so far,

@retry(stop_max_attempt_number=3, retry_on_exception=retry_if_db_error_or_passwd_change)
def add_request_to_db(self, req_id):
    with make_session(self.session_maker) as session:
        session.merge(RequestHolder(request_id=req_id))
        session.commit()

and retry_if_db_error_or_passwd_change is like

def retry_if_db_error_or_passwd_change(exception, class_reference):
    is_db_exception = isinstance(exception, DBRetryExceededException)
    is_operational_error = isinstance(exception, OperationalError)

    if is_db_exception:
        return True
    elif is_operational_error or 'Access denied ' in exception:
        fetch_password(class_reference)
        return True

My question is how do I send the class reference also with retry_if_db_error_or_passwd_change when passing the callable to retry_on_exception ?

Answer 1

Solution 1

Instead of using the raised exception as the basis of the retry via retry_on_exception, you can change it to be based on the return via retry_on_result. Here, we can pass the self argument.

1. Wrap the whole functionality to be retried inside a try-except block
2. Catch all exceptions
3. If an exception occurred, wrap the details of the exception and the class reference to an object and return the object.
   • The configured receiver of retry_on_result would then receive the object and can act accordingly.
4. If no exception occurred, proceed as usual. Return anything or none.
   • The configured receiver of retry_on_result would still receive the response but should just ignore and wouldn't retry.

from dataclasses import dataclass
import random
from typing import Any

from retrying import retry


# This could be a dataclass or just an ordinary class
@dataclass
class RetryInfo:
    exception: Exception
    class_reference: Any


def retry_if_db_error_or_passwd_change(retry_info):
    """Return True if we should retry, False otherwise."""
    if not isinstance(retry_info, RetryInfo):
        # If the response isn't intended for the retry mechanism, ignore and just continue.
        print("No retry needed")
        return False

    print("Evaluate retry for:", type(retry_info.exception), retry_info.class_reference.value)

    # Let's say we will stop the retries if there is a RuntimeError
    return not isinstance(retry_info.exception, RuntimeError)


class MyClass:
    def __init__(self, value):
        self.value = value

    @retry(retry_on_result=retry_if_db_error_or_passwd_change)
    def add_request_to_db(self, raise_exception):
        try:
            # Wrap the whole code inside a try-except block to catch all exceptions.
            print("Called add_request_to_db")
            self.value += 1
            if raise_exception:
                raise random.choice([IndexError, KeyError, RuntimeError, TypeError, ValueError])
        except Exception as error:
            # Instead of raising the exception to trigger a retry, just return the contained values.
            return RetryInfo(error, self)

print("Call 1...")
MyClass(0).add_request_to_db(True)

print("\n==========\n")

print("Call 2...")
MyClass(0).add_request_to_db(False)

$ python3 script.py 
Call 1...
Called add_request_to_db
Evaluate retry for: <class 'ValueError'> 1
Called add_request_to_db
Evaluate retry for: <class 'TypeError'> 2
Called add_request_to_db
Evaluate retry for: <class 'ValueError'> 3
Called add_request_to_db
Evaluate retry for: <class 'TypeError'> 4
Called add_request_to_db
Evaluate retry for: <class 'KeyError'> 5
Called add_request_to_db
Evaluate retry for: <class 'TypeError'> 6
Called add_request_to_db
Evaluate retry for: <class 'RuntimeError'> 7

==========

Call 2...
Called add_request_to_db
No retry needed

• For "Call 1", the retry_if_db_error_or_passwd_change was able to receive both the exception and the reference to self since it correctly printed the current value of self.value (increasing number) and was able to stop when RuntimeError occurred.
• For "Call 2", the behavior isn't changed if there are no raised exceptions, no retries are made.

Solution 2

This is inspired by the answer from @blhsing. Wrap the class method during class initialization so that you can inject the class reference self.

import random

from retrying import retry


def retry_if_db_error_or_passwd_change(exception, class_reference):
    """Return True if we should retry, False otherwise."""
    print("Evaluate retry for:", type(exception), class_reference.value)

    # Let's say we will just retry if any kind of exception occurred
    return isinstance(exception, Exception)


class MyClass:
    def __init__(self, value):
        self.value = value

        # Decorate functions to be retried
        retry_decorator = retry(
            retry_on_exception=lambda exc: retry_if_db_error_or_passwd_change(exc, self),
        )
        self.add_request_to_db = retry_decorator(self.add_request_to_db)

    def add_request_to_db(self, anything):
        self.value += 1
        chosen_exc = random.choice([IndexError, KeyError, RuntimeError, TypeError, None])
        if chosen_exc:
            print("Called add_request_to_db failed")
            raise chosen_exc
        else:
            print("Called add_request_to_db successful")


MyClass(0).add_request_to_db(None)

$ python3 script2.py 
Called add_request_to_db failed
Evaluate retry for: <class 'TypeError'> 1
Called add_request_to_db failed
Evaluate retry for: <class 'KeyError'> 2
Called add_request_to_db failed
Evaluate retry for: <class 'TypeError'> 3
Called add_request_to_db failed
Evaluate retry for: <class 'RuntimeError'> 4
Called add_request_to_db failed
Evaluate retry for: <class 'RuntimeError'> 5
Called add_request_to_db successful

• The class reference was correctly passed since it was able to print the current value of self.value (increasing number) and the retries stopped when when an exception wasn't raised anymore.

Answer 2

You can preserve the class reference as a default parameter for a wrapper function that calls retry_if_db_error_or_passwd_change with this additional argument:

# assign cls with the desired class reference first

@retry(
    stop_max_attempt_number=3,
    retry_on_exception=lambda exc, cls=cls: retry_if_db_error_or_passwd_change(exc, cls)
)
def add_request_to_db(self, req_id):
    ...