Retry failed API call in for...of loop

Question

I am trying to build a wrapper over Notion JS SDK's iteratePaginatedAPI that handles errors as well. I feel particularly lost on how do I catch API errors in such a way that I can actually retry them (aka retry the iteration that failed). Here's my attempt:

async function* queryNotion(listFn, firstPageArgs) {
  try {
    for await (const result of iteratePaginatedAPI(listFn, firstPageArgs)) {
      yield* result
    }
  } catch (error) {
    if (error.code === APIErrorCode.RateLimited) {
      console.log('rate_limited');
      console.log(error);
      sleep(1);
      // How would I retry the last iteration?
    }
  }
}

Coming from the Ruby world, there is a retry in a rescue block. Any help would be appreciated!

Answer 1

Very interesting problem. The issue is that the exception comes from the for await itself, not from its body, so you cannot catch it there. When the exception hits, loops are over.

Note that the iterator might be done after a rejection/exception, in which case there is nothing you can do except starting a new one.

That said, you can always call Iterator.next() yourself and process the result manually. The next() call of an async iterator will return an object like {value: Promise<any>, done: boolean}, and when running it in a loop, you can await the promise in a try..catch and only exit the loop when done becomes true:

async function* queryNotion(listFn, firstPageArgs) {
  const asyncGenerator = mockIteratePaginatedAPI(listFn, firstPageArgs)
  while (true) {
    const current = asyncGenerator.next()
    if (current.done) {
      break
    }
    try {
      yield* await current.value
    } catch (e) {
      console.log(`got exception: "${e}" - trying again`)
      continue
    }
  }
}

function* mockIteratePaginatedAPI(a, b) {
  for (let i = 0; i < 8; i++) {
    yield new Promise((resolve, reject) => setTimeout(() => [3, 5].includes(i) ? reject(`le error at ${i}`) : resolve([i]), 500))
  }
}

(async function() {
  for await (const n of queryNotion('foo', 'bar')) {
    console.log(n)
  }
})()

If we keep a reference to the generator, we can also put it back into a for async. This might be easier to read, however a for await ...of will call the iterator's return() when exiting the loop early, likely finishing it, in which case, this will not work:

async function* queryNotion(listFn, firstPageArgs) {
  const asyncGenerator = mockIteratePaginatedAPI(listFn, firstPageArgs)
  while (true) {
    try {
      for await (const result of asyncGenerator) {
        yield* result
      }
      break
    } catch (e) {
      console.log('got exception:', e, 'trying again')
    }
  }
}

function* mockIteratePaginatedAPI(a, b) {
  for (let i = 0; i < 8; i++) {
    yield new Promise((resolve, reject) => setTimeout(() => [3, 5].includes(i) ? reject(`le error at ${i}`) : resolve([i]), 500))
  }
}

(async function () {
  for await (const n of queryNotion('foo', 'bar')) {
    console.log(n)
  }
})()

Answer 2

Simply add a continue statement inside your if

async function* queryNotion(listFn, firstPageArgs) {
  try {
    for await (const result of iteratePaginatedAPI(listFn, firstPageArgs)) {
      yield* result
    }
  } catch (error) {
    if (error.code === APIErrorCode.RateLimited) {
      console.log('rate_limited');
      console.log(error);
      await sleep(1);
      continue; // retry the last iteration
    }
  }
}