Can we use spark pool to process data from dedicated SQL pool and is that a good architecture?

Question

Is it a good design to use spark pool for processing data which comes in dedicated SQL pool and again write back to dedicated SQL pool and to adls.

As of now everything we r doing with dedicated SQL pool so if we add spark pool so will it be more efficient or it will just be burden to existing dedicated SQL pool.

Answer 1

Yes, you can use spark pool to process data from dedicated SQL pool and is that a good architecture as there it is recommended and directly support by Microsoft Officials.

The Synapse Dedicated SQL Pool Connector is an API that efficiently moves data between Apache Spark runtime and Dedicated SQL pool in Azure Synapse Analytics. This connector is available in Scala.

If your project required large scale streaming you can definitely go for Apache Spark. There won't be any burden on existing architecture. You will get expected results.

Refer: Azure Synapse Dedicated SQL Pool connector for Apache Spark