Ingesting data from an API and webactivity

Question

I need to ingest data from 3 endpoints. my pipeline is as follows: a web activity to obtain authentication to the api via a post request and 3 web activities to retrieve the data via a get request. I would need to ingest data from these 3 web activities to a location in the raw zone . How can I achieve this?

I have managed to make the API calls and get a response. But have no knowledge on how to proceed further

Answer 1

You can directly use copy activity with Http connector to call your endpoints. and add it as source and the select appropriate sink. and copy the data

• sample liked service:

• Source example

OR

• Use copy activity with a sample .csv file with one column as the source dataset . Add additional column and provide a new column named Webactvityoutput with value = @string(activity('Web1').output) .
• In the sink, use another dataset with delimited text and pass file name as with .json extension. set Escape character as No escape character and Quote character as No quote character.
• In the mapping tab, click on Import schema and delete the column from dummy file and only keep the additionally added column.
• Execute the pipeline and check if the webActivity output got stored in the .json in the ADLS