How can I test my api response for pagination?

Question

I'm integrating QRadar with ForgeRock Identity Cloud which is not supported natively by IBM. Instead IBM has a Universal Cloud REST API protocol which can do this through a workflow xml (defines api call) and a environment vars xml (keys, secrets, etc).

The ForgeRock api for logs doesn't explicitly say how to implement pagination besides entering a token on every request after the first request (token provided by ForgeRock). I've compared the two json events objects and they are different but I'm still unsure because this token should be null when there are no more results.

Manually running this process starting from current time to last 24 hours on Postman 4-5 times never results in a null token. I have a certain date range which I know have 1,060 events total (1,000 on 1st response, 60 on 2nd response) that I've paginated using the method they document for paginating the tail logs on the recommended logging endpoint instead of the tail endpoint, and this terminates on the 2nd call with a null token.

enter image description here

QRadar documentation on how the xml tags work for the Workflow are kinda sparse. I'm not sure if ` auto encodes any of this. The forgerock docs do say the token should be uri encoded.

Right now, I'm using QRadar CE so I'm rate limited to 50 eps. ForgeRock API has a request limit of 1,000 eps and the largest time window is 24 hours.

I've tried to replicate this logic in postman but the rate limiting is preventing from confirming in QRadar. Unfortunately the client will not give me access as I develop and test this integration.

I've looked at example implementations from IBM on the QRadar side and tail pagination repo.

PS: Also, on the client's system, I've added the log sources but events are not appearing. On the IBM forum, this guy found that he had to postevents to source instead of host. I've set mine to post events to the host which is the fqdn to ForgeRock. It looks like they're already consuming a bunch of sources.

Answer 1

I got it. The ForgeRock documentation is incomplete regarding paging normal audit and debug logs.

You need the query params:

_queryFilter=true,

beginTime and endTime must be defined,

source=<log_src>,

_pagedResultsCookie which must be present but empty on on the initial call.

On the initial call, the page cookie will be present but empty. The responses will contain a cookie until it turns null indicating no more pages.

Fill the _pagedResultsCookie param with the cookie value on every call after the initial call to page through all results in a given time range.