BigQuery and 'bq load' - Overriding the default dataset CSEK with a GMEK for some tables

Question

I have a dataset in Big Query with a default Customer-Managed Encryption Key (CMEK) specified. This is a requirement and ensures, by default, that new tables are encrypted using a CMEK (unless specifically exempt).

I have some data I want to ingest into the dataset that doesn't require a CMEK and can instead use a Google-Managed Encryption Key (GMEK).

The main reason is to be able leverage features not available when using a CMEK, such as being able to use wildcards for tables in SQL within Big Query.

My question is: Using the bq load command from the command line, how can I override the default key with a GMEK when I load data to Big Query?

For example, if it were the other way around, I could use --destination_kms_key and specify my CMEK, but I see no option in the documentation that lets me specify a GMEK.

Is this possible?

Answer 1

When you have a default KMS key on your dataset, you can only use this key or to specify another one. You can't create table with GMEK (with KMS key in fact).

For that, you need to remove the default KMS key on the dataset and then to specify when you want to use a KMS key at table creation. If you specify nothing, GMEK is used.

Note: don't mix CMEK and CSEK. It's 2 different things not to mix (and CSEK isn't available on BigQuery)

---

EDIT 1

It's not mentioned how to removed the default KMS key on a dataset. I did that in my tests and it worked

bq update --default_kms_key= --dataset <myDataset>