I was recently exploring Azure Purview and was trying to push lineage information from ADF to Azure purview. However, it seems when we sink data in Delta Format using dataflow in ADF (Which is a inline format for data flow), it doesn't capture the lineage information. Whereas, it captures the lineage information when sinking using "dataset" in "Parquet" format. Is it that I am missing something here or its a bug or a feature that needs to be addressed?
I guess we need some optimization here.
On
The documentation indicates that as at today only sources and sinks are supported:
It also says the "lineage for Dataflow transformation is not supported yet". So I'm not sure what you mean by "inline format" - does that count as a transformation? If so, that would indicate why the lineage with Azure Data Factory (ADF) and Purview is not yet working, but indicates it may work at some point in the future.
Keep an eye on the documenation for updates:
https://learn.microsoft.com/en-us/azure/data-factory/turorial-push-lineage-to-purview
Delta Lake lineage from ADF data flow is not supported in Azure Purview right now. This is a known limitation and we are working to get it released. There is no ETA at this time.
In response to one of the comments, Inline datasets are sources and sinks in mapping data flows that do not require a dataset resource.