I am attempting to read a particular version from the Databricks Delta Sharing CDF (Change Data Feed) API.
Though version contains only one data file, an error occurs due to a timeout with following message:
"errorCode": "RESOURCE_LIMIT_EXCEEDED",
"message": "A timeout occurred when processing the table after 3 updates across 1 iterations. If it continues to happen, please contact your data provider and request them to optimize their table so that it has fewer files."
API(You will not be able to use it, as token is required):
https://eastus2-c2.azuredatabricks.net/api/2.0/delta-sharing/metastores/949ae2f5-8c64-4c5c-83a6-a4532259a54f/shares/demo/schemas/appevents/tables/auditlogs/changes?startingVersion=40000&endingVersion=40000
The 40000th version has:
- data size ~ 10KB
- And only 1 file.
Any help is appreciated!!