Python (pylint): Catching general exceptions in validation procedures

Question

Conventional wisdom states that you shall never catch a general exception in python, except for very rare circumstances. However, there is a noticeable lack of advice regarding what constitutes "very rare circumstances."

I am interested in the use case where a function is performing some sort of validation by calling various functions on a target object. If those functions raise an error, the object must not be valid.

In my particular case, I am trying to verify that a .json file is in fact valid json. It looks something like this:

def validate_json(file_path: str):

    try:
        with open(file_path) as f:
            json.load(f)
    except Exception:
        return False

    return True


def list_json_files(root_dir: str):
    return [file for file in glob.glob(root_dir + "*.json") if validate_json(file)]

Note that I didn't test the code above (I quickly wrote it to illustrate the scenario). In many cases like this, it seems unnecessary to track down all the possible exceptions that might happen while trying to validate a particular object. Catching specific exceptions also makes the code less future-proof, for instance if an outside library were to add a new exception that may be raised.

The main reason I'm curious about this use case is because it doesn't seem all that rare to me, yet it's never mentioned.

More importantly, as a pylint user I have to deal with those pesky broad-exception-caught warnings that I can't get rid of without littering my code with #pylint: disable...s.

I guess my question is: Is this a valid reason to catch a general exception? Otherwise, what would be a better approach to solve this type of problem, particularly in a way that makes pylint happy?

Answer 1

Your example is perhaps even more instructive than you intended!

It is certainly true that there are times when you want to catch any exception at all, and those times might make you curse pylint for being an over-fussy nanny. However, it is only behaving as a useful nag to ask if you are really sure that you don't want to be more specific.

But in your example validate_json function, what if you misspell the filename, or don't have permission to read the file? You now believe you have invalid JSON on your hands, whereas an uncaught FileNotFoundError or PermissionError would have told you what was really wrong.