How to fix Firebase cloud function logs table too many columns

Question

I have a firebase project where I have a number of functions logging data.

Every now again again I see this error:

Too many total leaf fields: 10002, max allowed field count: 10000

What I believe has happened is that over time the jsonPayload.data column has gotten wider and wider (more properties) as the application logged all kinds of different objects there.

I believe it was my mistake to log objects where the properties were dynamic (i.e. the structure of the objects changes over time). I've updated the application to stop doing this now.

However, I still need to fix the table.

In an attempt to resolve this problem, I considered dropping the nested columns below jsonsPayload.data, but it appears dropping nested columns is not supported in BigQuery.

screenshot of Cloud Function log table schema in BigQuery

There are suggestions to create a new table and exclude the columns that are not required. But this raises the question that the new table will have a different name, and I'm not sure how to repoint cloud function logs to the new table (or even if that's the right thing to do).

I also considered just dropping the whole table and hoping a new one would be created (I'm ok with losing the data). But I'm not sure if this would break something else since the table was created by Firebase (I think) when I created the functions.

Some code to replicate the issue:

export const fooFunction = functions.https.onRequest((req, resp) => {
  functions.logger.info({
   event: 'test-event',
   data: {
    // nested object with properties that change over time
   }
  });
})

When this function called, it will insert rows in to a BigQuery table called:

<project_id>.<title>_cloud_function_logs.cloudfunctions_googleapis_com_cloud_functions

The schema will look like this:

screenshot of BigQuery table schema for cloud function log

So my question boils down to this, how do I drop either:

a) the whole table, or b) the columns below jsonPayload.data

But without breaking cloud function logging?

Answer 1

I came to the conclusion that the table cannot be repaired.

I decided to drop the whole table, remove the sink, and create it again from scratch

The documentation for managing the sink is at: https://cloud.google.com/logging/docs/export/configure_export_v2#managing_sinks