I am building a similar approach to bring all the logs into BigQuery dataset. As per this https://dev.to/alvardev/gcp-cloud-logging-the-basis-45eh I am able to bring the project log details into the corresponding BigQuery. But how do we bring all the projects logs into a single Bigtable i.e for example lets say An organisation has 50 projects and with minimal configuration I want all the logs from 50 projects to come under a single Bigtable as consolidated project logs.
Is this possible in GCP, can someone please help me on this
Currently, the only supported destinations based on the documentation are Cloud Storage, Pub/Sub, BigQuery and another Cloud Logging Bucket . However I found this another approach where the architecture is: Cloud Logging -> Pub/Sub -> DataFlow -> BigTable.
Basically, you will create a sink and choose Pub/Sub topic as the destination then from there, you can stream your data from Pub/Sub to Bigtable using Dataflow. You can check this SO answer that you can use as reference on how to stream data from Pub/Sub to BT.