Persistently receiving 14 UNAVAILABLE: Stream refused by server. Create BigTable client per-request?

Question

It's happening daily an only on our save operation:

// the record is this small and `myValue` is a sting < 32 characters in length.

const rowToInsert = {
  data: {
    myKey: { value: `${params.myValue}` },
  },
};

await table.row(rowId).save(rowToInsert, gaxOptions);

Which leads me to believe that clearly something is wrong and it's probably not with BigTable. Using nodejs, with the client provided from @google-cloud/bigtable the operations are invoked with the following GAX options:

import { status } from '@grpc/grpc-js';
import { CallOptions, createRetryOptions, createBackoffSettings } from 'google-gax';

// https://cloud.google.com/bigtable/docs/status-codes
  const retries = 4;
  const timeout = 1000;
  const retryCodes = [
    status.CANCELLED,
    status.UNKNOWN,
    status.DEADLINE_EXCEEDED,
    status.FAILED_PRECONDITION,
    status.ABORTED,
    status.INTERNAL,
    status.UNAVAILABLE,
    status.DATA_LOSS,
  ];

const initialRetryDelayMillis = 100;
const retryDelayMultiplier = 1.3;
const maxRetryDelayMillis = 1000;
const initialRpcTimeoutMillis = null;
const rpcTimeoutMultiplier = null;
const maxRpcTimeoutMillis = null;
const totalTimeoutMillis = timeout;

const backoffSettings = createBackoffSettings(
    initialRetryDelayMillis,
    retryDelayMultiplier,
    maxRetryDelayMillis,
    initialRpcTimeoutMillis,
    rpcTimeoutMultiplier,
    maxRpcTimeoutMillis,
    totalTimeoutMillis,
  );

  const options: CallOptions = {
    // https://github.com/googleapis/gax-nodejs/blob/889730c1548a6dcc0b082a24c59a9278dd2296f6/src/gax.ts#L158-L159
    // ignored when using retry
    // timeout: ###
    maxRetries: retries,
    // https://github.com/googleapis/gax-nodejs/blob/889730c1548a6dcc0b082a24c59a9278dd2296f6/src/gax.ts#L349-L351
    retry: createRetryOptions(safeRetryCodes, backoffSettings),
  };

I would assume 4 retries should be plenty for any instability especially with the backoff. Reviewing the monitoring of my BigTable instance, it is not remotely under any strenuous load.

This leads me to the idea that maybe it's because I am initializing my client as a singleton? Maybe I should initialize the client per-request instead? Maybe the client is connecting and then something is timing out and the next request fails?

So should a BigTable client be created per-request? Is something else suspect in the GAX options above?

The examples: https://github.com/googleapis/nodejs-bigtable/blob/master/samples/tableadmin.js

seem to show making a new client per action, but I don't know if that's just for example purposes. Same with the example on the npm page:

https://www.npmjs.com/package/@google-cloud/bigtable/v/0.16.0#using-the-client-library

Answer 1

I have found the same error on this GitHub Issue. The reported issue is for the Firestore API, but if you continue reading; they said that the main issue is with the @grpc/grpc-js library.

The proposed workaround is to update the library and try again, just have in mind that the issue is still open and the GCP Engineers are still working on it.

Also, you can try to open a similar issue but in the BigTable API GitHub Issue Page

Just as a quick refresh, you can update it with the command

npm update -g @grpc/grpc-js