Using great expectations for date validation

Question

We are using great_expectations to validate data using Apache Spark. We are unable to validate columns which have the DATE or DATETIME type.

We use the below configuration to check date entries in a table if they are recent or not.

[
                {
                    "expectation_type": "expect_column_min_to_be_between",
                    "kwargs": {
                        "column": "started",
                        "min_value": "2020-12-13"
                         },
                    "parse_strings_as_datetimes": True
                }]

But it throws the error: not supported between instances of 'str' and 'datetime.datetime'

Since str failed, we tried to use

[
            {
                "expectation_type": "expect_column_min_to_be_between",
                "kwargs": {
                    "column": "started",
                    "min_value": datetime.fromisoformat("2020-12-13")
                     },
                "parse_strings_as_datetimes": True
            }]
               

But here we get the same error: not supported between instances of 'str' and 'datetime.datetime'

We also tried it after removing the parse_strings_as_datetimes argument but it still throws the same error.

We have also tried asking the great_expectations team in slack, but they didn't respond. Does anyone have a clue what is wrong?

Answer 1

This appears to be a bug. There is an existing Github Issue open for this, and the internal team will be working on it. I also tried searching the Great Expectations Slack and couldn't find a question like this, so for the future, you may want to confirm that you have actually posted the question in the Support channel of the Slack.