Great Expectation using Cloud Data Store

11 Views Asked by At

I am programming a project in python 3.7. I use Great Expectations to verify some expectation and validation on my datasets. I am running the project on a containerized environment where I don't have access to file system. Moreover, I have access to S3 as permanent memory. I need to store my validation artifacts, what is the best way here? I have seen that great expectations has a CloudDataContext but I have not found a good document or example. Does anyone here know how I can use it? I have (an endpoint_url for my s3 too)

I found this code on the internet but it does not work too

context_config = {
    "store_backend": {
        "class_name": "TupleS3StoreBackend",
        "bucket": "mybucket",
        "prefix": "myprefix",
        "protocol": "s3",
    },
    "store_backend_options": {
        "aws_access_key_id": "YOUR_ACCESS_KEY_ID",
        "aws_secret_access_key": "YOUR_SECRET_ACCESS_KEY",
        "endpoint_url": "YOUR_S3_ENDPOINT_URL",
    }
}
ured storage backend
data_context = DataContext(context_config)
0

There are 0 best solutions below