404 errors calling BigQuery, Pub/Sub and Secret Manager after deployment

Question

Starting at 6:30pm yesterday, every BigQuery, Pub/Sub and Secret Manager request started failing with a 404 error in every API call. I deployed a new version around this time – so that seems like a potential trigger for the issue – but the change was trivial (a change to a logging message). A deployment 5 hours earlier worked fine.

The environment is Nodejs 20 and the APIs use Google Cloud Functions (a mix of 1st and 2nd Gen). The APIs are failing when called via https, pub/sub and callable functions. The 404 errors occur when calling one of these three Google services (and perhaps others). There are no deployment errors and updating @google-cloud/bigquery and other packages to the latest versions did not resolve the issue.

Here's an example of a simple BigQuery request to check if a table exists. The error message is shown below.

import { BigQuery } from '@google-cloud/bigquery';
let bigquery: BigQuery;

export async function checkTableExists(dataset: string, table: string): Promise<boolean> {
  bigquery = new BigQuery();
  const tableExists = await bigquery.dataset(dataset).table(table).exists();   // <-- GETS 404 HERE
  return tableExists[0];
}

The resulting error message indicates there might be an authentication issue, but this is a stable API that's been running for several years. Nothing changed in the configuration, service accounts, etc. between the two deployments yesterday.

Unhandled error GaxiosError: Unsuccessful response status code. Request failed with status code 404
    at Gaxios._request (/workspace/node_modules/gcp-metadata/node_modules/gaxios/build/src/gaxios.js:141:23)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async metadataAccessor (/workspace/node_modules/gcp-metadata/build/src/index.js:110:21)
    at async GoogleAuth._GoogleAuth_getUniverseFromMetadataServer (/workspace/node_modules/google-auth-library/build/src/auth/googleauth.js:777:26)
    at async GoogleAuth.getUniverseDomain (/workspace/node_modules/google-auth-library/build/src/auth/googleauth.js:186:168)
    at async GoogleAuth.getApplicationDefaultAsync (/workspace/node_modules/google-auth-library/build/src/auth/googleauth.js:252:42)
    at async GoogleAuth.getClient (/workspace/node_modules/google-auth-library/build/src/auth/googleauth.js:674:17)
    at async GoogleAuth.authorizeRequest (/workspace/node_modules/google-auth-library/build/src/auth/googleauth.js:715:24)
    at async Promise.all (index 1)
    at async prepareRequest (/workspace/node_modules/@google-cloud/common/build/src/util.js:442:61) {
  config: {
    url: 'http://169.254.169.254/computeMetadata/v1/universe/universe_domain',
    headers: { 'Metadata-Flavor': 'Google' },
    retryConfig: {
      noResponseRetries: 3,
      currentRetryAttempt: 0,
      retry: 3,
      httpMethodsToRetry: [Array],
      statusCodesToRetry: [Array]
    },
    params: {},
    responseType: 'text',
    timeout: 0,
    paramsSerializer: [Function: paramsSerializer],
    validateStatus: [Function: validateStatus],
    method: 'GET',
    errorRedactor: [Function: defaultErrorRedactor]
  },
  response: {
    config: {
      url: 'http://169.254.169.254/computeMetadata/v1/universe/universe_domain',
      headers: [Object],
      retryConfig: [Object],
      params: {},
      responseType: 'text',
      timeout: 0,
      paramsSerializer: [Function: paramsSerializer],
      validateStatus: [Function: validateStatus],
      method: 'GET',
      errorRedactor: [Function: defaultErrorRedactor]
    },
    data: '404 page not found\n',
    headers: {
      'content-length': '19',
      'content-type': 'text/plain; charset=utf-8',
      date: 'Thu, 30 Nov 2023 13:44:09 GMT',
      'x-content-type-options': 'nosniff'
    },
    status: 404,
    statusText: 'Not Found',
    request: {
      responseURL: 'http://169.254.169.254/computeMetadata/v1/universe/universe_domain'
    }

Any ideas on what might cause this sudden and complete breakage of the API? Google Support isn't aware of any outages on their side.

Answer 1

Google has identified and resolved the issue: https://github.com/googleapis/google-auth-library-nodejs/issues/1706.

In my case, I had to add install google-auth-library pinned to v9.2. It's worth noting that this issue occurred even though I don't directly use this library.

It appears that v9.4.1 fixes this issue, but I have not tested this option.