Running an .exe in Azure Data

Question

So I am working on migrating a project from one of our (to-be deprecated) platforms to Azure Data Factory. That platform can run .exe files. Here is a brief summary of the system I am migrating:

1. Reads multiple TSV files, each file larger than 4 GB
   
2. Run the C# .exe which on the surface does the following:
   - Extra a base64string from each row from each TSV file
   - Decode the base64string into a json and populate an object model using this
   - Create metrics using some details in the object model
   - Write these metrics to a TSV file.
   

Requirement: To migrate the above system to ADF and add some additional components to it.

Steps 1 and 2 remain the same from above.
3. Read the metrics from each row from each of the TSV files.
4. Perform some data transformations with read metrics
5. Write them to Azure Data Explorer

I know how to do steps 1, 4, 5 but have no clue on how to trigger a .exe in Azure Data Factory and how to read the output of it.

Converting the C# .exe into data flows is not possible having failed pursuing that option. I have very limited expertise in Azure Services in general. Can someone explain if there is any way to get this done? If possible I would like to use the .exe as is and just supply the inputs to trigger the.exe

Answer 1

You can use Custom Activity and Azure Batch to run your EXE. Have the EXE write the results to a known location (like Storage or SQL) and then subsequent activities can read those results.