How to power a windowed virtual list with cursor based pagination?

Question

Take a windowed virtual list with the capability of loading an arbitrary range of rows at any point in the list, such as in this following example.

The virtual list provides a callback that is called anytime the user scrolls to some rows that have not been fetched from the backend yet, and provides the start and stop indexes, so that, in an offset based pagination endpoint, I can fetch the required items without fetching any unnecessary data.

const loadMoreItems = (startIndex, stopIndex) => {
  fetch(`/items?offset=${startIndex}&limit=${stopIndex - startIndex}`);
}

I'd like to replace my offset based pagination with a cursor based one, but I can't figure out how to reproduce the above logic with it.

The main issue is that I feel like I will need to download all the items before startIndex in order to receive the cursor needed to fetch the items between startIndex and stopIndex.

What's the correct way to approach this?

Answer 1

After some investigation I found what seems to be the way MongoDB approaches the problem:

https://docs.mongodb.com/manual/reference/method/cursor.skip/#mongodb-method-cursor.skip

Obviously he same approach can be adopted by any other backend implementation.

They provide a skip method that allows to skip an arbitrary amount of items after the provided cursor.

This means my sample endpoint would look like the following:

/items?cursor=${cursor}&skip=${skip}&limit=${stopIndex - startIndex}

I then need to figure out the cursor and the skip values.

The following code could work to find the closest available cursor, given I store them together with the items:

// Limit our search only to items before startIndex
const fragment = items.slice(0, startIndex);
// Find the closest cursor index
const cursorIndex = fragment.length - 1 - fragment.reverse().findIndex(item => item.cursor != null);
// Get the cursor
const cursor = items[cursorIndex];

And of course, I also have a way to know the skip value:

const skip = items.length - 1 - cursorIndex;