REST call in Copy Activity under Azure Synapse / ADF

Question

I'm trying to use the Copy Activity to get data from a REST API. I have a problem with the paging functionnality and I'm wondering that if I can use it considering the available functionnalities.

I'm using an http request as follow:

https://MyAPI.com/API/TableName?$orderby=Field1,Field2,Field3&$format=json

Then I get my json response with

{
    "value": [
        data that I'm interested in here...
    ],
    "odata.nextLink": "TableName/?$orderby=Field1,Field2,Field3&$skiptoken=156464,165464,1,84,-1,3373"
}

Therefore it means that for my next URL, I have to take part of my original link: https://MyAPI.com/API/ and add to it the odata.nextLink content and also add my &$format=json at the end.

My guess is that it's now possible to do this using the Copy Activity and I would have to rely on a Web Activity in a loop where I can fully control the URL but that's really not an avenue I would like to pick.

Any help would be greatly appreciated. Thanks

I tried using a variable within my dataset like this:

And my pagination rule:

But then I get the following error: Failure happened on 'Source' side. ErrorCode=RestInvalidPaginationRule,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Invalid PaginationRule, RuleKey='AbsoluteUrl.{nextLink}', RuleValue='$.'odata.nextLink'',Source=Microsoft.DataTransfer.ClientLibrary,'

Answer 1

haven't tested my answer, but based on your data,

1. go to your copy activity in ADF > under Source > Pagination rules,
2. select "AbsoluteUrl",
3. leave the adjacent next box empty
4. under Value select "Body"
5. type "$.'odata.nextLink'" Although this part could differ based on your mapping as well.

You could also troubleshoot this yourself as well, when you hover over your copy activity on the canvas in ADF you can select the little "</>" button next to the litterbin & duplicate icons. In this, under the "translator" json code node, you can find out your mapping and what ADF expects to succesfully address the odata.nextline.

Hoping to have helped!