Great_Expectations Conditional Expectation in Spark 3.2.1 with Pandas API in DataBricks

Question

We want to implement Great_Expectations' in DataBricks with Conditional Expectation. According to GE's documentation https://docs.greatexpectations.io/docs/reference/expectations/conditional_expectations is only available for Pandas this argument must be set to "pandas" by default, thus, demanding the appropriate syntax. Other engines might be implemented in the future.

Does anyone knows if it can be implemented in Spark 3.2.1 which integrated with Pandas APIs? if not, any suggestion for handle conditional expectation in DataBricks with Spark 3.2.1?

Answer 1

This functionality is now experimentally supported w/ Spark. The documentation is still being updated, but you should now be able to set a row_condition on an Expectation against a Spark datasource by passing great_expectations__experimental__ as the condition_parser.