Google Stackdriver log entry record field logging.googleapis.com/severity vs severity

Question

I have a general log package that wraps logs for Stackdriver- Gets context, severity etc and transform it to LogEntry i.e:

func Log(ctx context.Context, severity Severity, format string, a ...interface{}) {
    log.Println(Entry{Severity: severity, Message: fmt.Sprintf(format, a...), Component: "arbitrary-property", Trace: GetTraceId(ctx)})
}

The entry struct lookgs like this

type Entry struct {
    Message  string   `json:"message"`
    Severity Severity `json:"severity,omitempty"`
    Trace    string   `json:"logging.googleapis.com/trace,omitempty"`

    // Logs Explorer allows filtering and display of this as `jsonPayload.component`.
    Component string `json:"component,omitempty"`
}

This package is global and used for multiple services both in Cloud Run and in Compute Engine.

The logs in Compute Engine is ingested with Google Ops Agent If I want the logs severity to work with ops agent, I need to use this json key logging.googleapis.com/severity according to the docs: https://cloud.google.com/stackdriver/docs/solutions/agents/ops-agent/configuration#special-fields Otherwise the severity remains a part of jsonPayload, and the severity is not applying on the log entry. But this conflicts with the docs here: https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry Here it says the json field should be simply severity without the prefix logging.googleapis.com/. If I use this struct

type Entry struct {
    ...
    Severity Severity `json:"severity,omitempty"`
}

It's not working with Ops Agent, if I use this struct

type Entry struct {
    ...
    Severity Severity `json:"logging.googleapis.com/severity,omitempty"`
}

It works with Ops Agent, but not working in Cloud Run (I simply see logging.googleapis.com/severity as part of jsonPayload, the severity is not applying). Is it the expected behaviour? Is there a workaround for this without creating two packages?

Answer 1

This is the expected behavior. There's no workaround for this partícular scenario, so you need to build two packages.

In Cloud Run, as you can read here about Special JSON fields in messages, when you provide a structured log as a JSON dictionary, some special fields are stripped from the jsonPayload and are written to the corresponding field in the generated LogEntry as described in the documentation for special fields.

For example, if your JSON includes a severity property, it is removed from the jsonPayload and appears instead as the log entry's severity. The message property is used as the main display text of the log entry if present. For more on special properties, read the Logging Resource section.

Answer 2

I ended up writing two structs and using fmt.Stringer to write only one log function. This is bad IMO because it requires to handle two structs (i.e you add field to one, you need to remember to add to the second one also), but I couldn't find anything better. It look to me like some kind of mistake in GCP, I guess it written by two different teams, otherwise I don't see any reason why not using the same json field name.

var useOpsAgent bool

func SetUseOpsAgent(v bool) {
    useOpsAgent = v
}

// for cloud run
type Entry struct {
    Message  string            `json:"message"`
    Severity Severity          `json:"severity,omitempty"`
    Trace    string            `json:"logging.googleapis.com/trace,omitempty"`
    Component string `json:"component,omitempty"`
}

// for compute engine
type EntryOA struct {
    Message   string            `json:"message"`
    Severity  Severity          `json:"logging.googleapis.com/severity,omitempty"`
    Trace     string            `json:"logging.googleapis.com/trace,omitempty"`
    Component string            `json:"component,omitempty"`
}

func (e Entry) String() string { ...

func (e EntryOA) String() string { ...

func Log(ctx context.Context, severity Severity, format string, a ...interface{}) {
    var logEntry fmt.Stringer = Entry{Severity: severity, Message: fmt.Sprintf(format, a...), Component: "arbitrary-property", Trace: GetTraceId(ctx)}
    if useOpsAgent {
        logEntry = EntryOA{Severity: severity, Message: fmt.Sprintf(format, a...), Component: "arbitrary-property", Trace: GetTraceId(ctx)}
    }
    log.Println(logEntry)
}

Then when using in the main.go:

func init() {
    loglib.SetUseOpsAgent(true)
}