Writing Python validation script for different data files from different data sources on premises for different rules

Question

i am new to python and i want to move data files from different on premises data sources to azure data lake storage and i want to validate these data files for different rules of validation before the get moved using azure data factory pipeline and azure databrick

I am aware how to create custome python copy activity in databricks for running python script

i need help with writing the script for validation

Answer 1

You might want to have a look at Data Factory to move files between different storages. It's pretty straightforward and scalable and you can even create Data Flow pipelines based on a Spark Cluster without having to write a single line of code.