Googlebot throws "TypeError: Failed to fetch" errors on POST despite good CORS response headers

104 Views Asked by At

I have a try/catch around a fetch JavaScript call that reports caught errors. I periodically get the following report about "TypeError: Failed to fetch" exceptions from Googlebot requests:

[2023-09-21T00:00:00.031Z] POST https://findhumane.com/find_humane/api/get_location_details?v=12.1.20230917
[2023-09-21T00:00:00.034Z] handleErrors: Failed to fetch (class: TypeError)
...
      HTTP_USER_AGENT: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.5938.88 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

While there are many potential causes for TypeError during fetch, the most common is improper CORS response headers, but I don't see any problems from Chrome network inspector when I perform the same use case manually; for example, it shows good CORS headers:

POST https://findhumane.com/find_humane/api/get_location_details?v=12.1.20230917
200 OK

Response Headers:

Access-Control-Allow-Methods: GET, POST, PATCH, PUT
Access-Control-Allow-Origin: *
Access-Control-Max-Age: 7200
...

The odd thing is that since I'm getting the exception email from the exception catch that is being POSTed to the same domain, then I don't know what the difference is between that fetch and this application fetch.

Any ideas or how I can debug further? I have two ideas:

  1. Run a tcpdump on my backend all the time between my reverse proxy (HAProxy) and app server without TLS and then correlate to the Googlebot exception email to confirm 100% that the right CORS response headers are being sent.
  2. Maybe I can somehow route the console into an in-memory string and report other console warnings/errors in my exception email?

I do have a 60 second timeout abort on the fetch:

try {
  const controller = new AbortController();
  const timeoutId = setTimeout(() => controller.abort(), 60000);
  const response = await fetch(
    uri,
    {
      method: "POST",
      headers: {
        "Accept": "application/json",
        "Content-Type": "application/json",
      },
      signal: controller.signal,
    },
  );
  ...

However, the timestamps above show the TypeError occurred about 3 milliseconds after the request started.

0

There are 0 best solutions below