Checking response code of all URLs in a column [Airtable database]

Question

We have an airtable database of over 24000 records. These records are websites, and many now have errors in them (missing "/", extra space...). We are trying to detect the websites that have these errors before manually fixing them.

What we have tried So far, we have used the fetch method to call each URL and report back on the error status . This is the script we have used:

const inputConfig = input.config(); 
const url = inputConfig.url;
let status;

try {
    const response = await fetch(url);
    status = response.status; } catch (error) {
    status = 'error'; }

output.set('status', status);

Issues we ran into

1. The script won't follow redirects, so it reports "error" back if there is a redirect even if the URL is working.
2. The output now is either "200" meaning the URL works, or "error". We don't get the actual response code of the error, which we ideally would like to get.

Any help would be appreciated! Thanks

Answer 1

There's some nuance to how fetch works. If you review Mozilla's documentation they say:

The Promise returned from fetch() won't reject on HTTP error status even if the response is an HTTP 404 or 500. Instead, as soon as the server responds with headers, the Promise will resolve normally (with the ok property of the response set to false if the response isn't in the range 200–299), and it will only reject on network failure or if anything prevented the request from completing.

So you have to do an extra check in your code to determine if the request was successful or not and throw your own error. In your case, you don't necessarily need to throw an error at all and can just rely on ok property of the response.

const config = input.config();
const url = config.url;

let status = null;

const response = await fetch(url);

if(response.ok) {
    status = response.status
} else {
    status = 'error'
}

output.set('status', status);