Is it necessary to have md5-content property for a file to be auto-ingested by snowpipe from azure adls gen2?

Question

I am trying to load data through snowpipes and somehow all the files are not getting loaded, these files are from different sources including automated jobs, manual uploads, data extracts. The one thing I am seeing in common is - the files that do not have md5 are not ingested automatically, while the files that do are. So wanted to confirm if anybody knows that md5 is necessary for a file to be auto-ingested?

Thanks

Answer 1

Turns out ADLS gen2 API needs explicit flush close in order to create a notification for snowflake which leads to auto ingest. Added this to the azure function and it seems to be working fine now

Answer 2

No, it is not necessary to have a MD5 (file) for auto-ingest. It relies on notification services (SNS, SQS, Event Grid).

I suggest you to query SYSTEM$PIPE_STATUS to understand how many files are pending, timestamp of the oldest file in the queue, timestamp of last forwarded notification message:

https://docs.snowflake.com/en/user-guide/data-load-snowpipe-auto-s3.html#system-pipe-status-output