Copy Firestore Database Data to Bigquery using Cloud Shell Terminal

Question

Does anyone know how I can manually copy/transfer data from Firestore database to Bigquery using Cloud Shell Terminal?

I did this in the past but I'm unable to find the documentation/video that I used. I find a lot that states that once Bigquery is connected to Firebase, it should be automatic but mine is not.

When I ran code in the Cloud Shell Terminal to pull data from Firebase the collection was copied as a table into a Bigquery dataset. Two tables were created and "raw_latest" and "raw_changelog" were created.

I'm not sure how to transfer another collection now. I specifically need to transfer data from a subcollection in the Firestore database.

Answer 1

1. You can now export data from Cloud Firestore to BigQuery with a Firebase Extension. To import all the previous data you will need first to install the extension because all the writes while doing the export first without installing the extension will be lost.

   See: https://firebase.google.com/products/extensions/firestore-bigquery-export

2. Firestore allows import / export data to BigQuery using a GCS bucket. The data is exported to a Cloud Storage bucket and from there it can be imported into Big Query.
   The gcloud commands for the same are :

export data :

gcloud beta firestore export --collection-ids=users gs://my bucket/users

load backup into bq :

bq load --source_format=DATASTORE_BACKUP mydataset.users gs://gs://mybucket/users/all_namespaces/kind_users/all_namespaces_kind_users.export_metadata

3. Here are some links that might be helpful:

• https://firebase.google.com/docs/firestore/manage-data/export-import
• https://cloud.google.com/bigquery/docs/loading-data-cloud-datastore
• https://github.com/firebase/extensions/blob/master/firestore-bigquery-export/guides/IMPORT_EXISTING_DOCUMENTS.md