I don't work with database so want to confirm best practices before I start. I have been given a project to where I am supposed to take few differently modeled data files, process them and show some standard reporting.
It will be batch ingestion, can be weekly or daily max. The cloud we will be using is Microsoft Azure. I am thinking of below tools for the job from my development experience. But will love some advise from you guys:
-> Ingestion [and stage] => write a rest api which takes the data and stage it in Elastic
-> Processing -> Azure HDInsight.. I haven't used it but from brief reading, that sounds like the recommended choice.
-> Staging and Reporting -> back to Elastic search, with Kibana/Graphana. I have used these tools so preferring it.
I also read some articles recommending Azure Data Factory. Look forward to do some suggestion.
Regards, Gaurav Sharma
For the ingestion you can use Data Factory, it supports calling a rest api with a WebActivity. https://learn.microsoft.com/en-us/azure/data-factory/control-flow-web-activity
For processing, have you considered storing the data in a Data Lake Store and using U-SQL to make the transformations? I'm just mentioning it because you may have skipped considering it. https://learn.microsoft.com/en-us/azure/data-lake-analytics/
For reporting Microsoft offers Power Bi, it can connect to a lot of sources included rest APIs. It may or may not be useful for you, depending on the requirements of your project. https://app.powerbi.com
Hope this helps!