How to set up a custom log destination using Google Cloud Stackdriver Log Export service

Question

I'm trying to set up a custom log destination sink using the Stackdriver Log Export service. But I'm able to find any documentation about how to implement this. Does anyone know where to find information about how to implement a custom destination?

I've previously successfully set up the Cloud Storage and Cloud Pub/Sub sink destinations.

Here's a screenshot of the feature I'm referring to --

Answer 1

Finally was able to get an answer from Google directly and they have an internally open issue as to the custom destination, and the stackdriver team is working on a fix at it but there is no ETA on this. FWIW the "Custom destination" will still be one of the three original channels (GCS, BQ or Pub/Sub), but will allow you to specify another project to export to. However, it may be possible to export your logs through a Cloud Pub/Sub topic and the third party receives your logs by subscribing to the same topic per these instructions [1] since Stackdriver Logging supports logging integration with third parties such as Splunk, Logentries, and Tenable.

[1] https://cloud.google.com/logging/docs/export/using_exported_logs#integrate-thru-pubsub

Answer 2

The way to define custom destination is mentioned in this post: How do I coalesce Stackdriver logs/sinks into one BigQuery project/dataset?

We have to select Custom Destination when creating an export in Google Stackdriver Logging viewer. Then we have to give destination in the below formats:

• For BigQuery:
  bigquery.googleapis.com/projects/[PROJECT_ID]/datasets/[DATASET_ID]
• For PubSub: pubsub.googleapis.com/projects/[PROJECT_ID]/topics/[PUBSUB_Topic]
• For GCS: storage.googleapis.com/[BUCKET_NAME]

Add Unique Writer Identity generated after sink creation to IAM & Admin and give BiqQuery Data Editor role, PubSub Editor role and all storage permissions depending on the type of sink (BigQuery, PubSub and GCS respectively).