Add additional header from previous activity to rest call in copy data activity

Question

I've a pipeline which should sync data from a REST API source to a SQL table. There are 2 steps in this pipeline:

1. Get the last changed datum field from the data set in the previous run, so that I know that I have to sync all records which got changed after this date.
2. The copy data activity, which should perform a REST API call (with the date filter in there as a param so that it only retrieves the changed records from the source) and sink them to a SQL table.

The problem is that the additional header parameter (the DATECHANGED field) is not correctly added to my REST call.

I've set the a pram in the connection settings and added an expression in the data set properties. I've done the same thing for other familiar pipelines with pagination, there it's working fine. I've also tried to removing the params from the connection and set additional headers in the source settings to make it work but it didn't make a difference.

What is the way to solve this issue? Or what am I doing wrong?

Answer 1

I tried to copy the API data with dynamic query parameter using ADF. Below is the approach to do it.

• Below is the sample API dataset. Parameter is created and given in the relative URL.

Base URL: https://swapi.dev/api/peopele/
Relative URL: ?created={@dataset().created}

• Lookup activity output is given as the value for dataset parameter created.

• When pipeline is executed, data from API with query parameter is copied to sink without error.