How to get google-cloud-ops-agent logging to recognize special "severity" field

Question

I am running ZooKeeper on a google compute instance and trying to setup google-cloud-ops-agent to properly parse the logs. I am most of the way there, but am having trouble getting it to recognize the severity field.

The docs state that severity is a special field that will get extracted from the jsonPayload, but I am not seeing that happening.

• special fields documentation
• logEntry documentation

My /etc/google-cloud-ops-agent/config.yaml

logging:
  receivers:
    zookeeper:
      type: files
      include_paths:
      - /zookeeper/logs/zookeeper.log
  processors:
    zookeeper:
      type: parse_regex
      field: message
      regex: '^(?<time>.{23}) \[(?<zknode>[^\]]+)] - (?<severity>\S+)\s+ \[(?<class>[^\]]+)] - (?<msg>.*)$'
      time_key: time
      time_format: "%Y-%m-%d %H:%M:%S,%L"
  service:
    pipelines:
      zookeeper:
        receivers: [zookeeper]
        processors: [zookeeper]

evidence the parse_regex is working properly (but notice severity still exists in jsonPayload)

Answer 1

I fixed it by describing in processors:

    change_severity:
  type: modify_fields
  fields:
    severity:
      copy_from: jsonPayload.severity

and then to service -> pipelines:

      web_access:
    receivers:
      - web_access
    processors:
      - change_severity

Answer 2

When parsing an unstructured log, the Ops Agent places all fields under jsonPayload unless the field has one of the special names from that table, as you've pointed out from the docs. However, in that table severity is the destination field, not the source. The correct source field is spelled logging.googleapis.com/severity.

Unfortunately the regex engine does not allow special characters in destination field names. If it did, you could simply write your regex as e.g. regex: '^(?<time>.{23}) \[(?<zknode>[^\]]+)] - (?<logging.googleapis.com/severity>\S+)\s+ \[(?<class>[^\]]+)] - (?<msg>.*)$' and the problem would be fixed.

Because you can't write that regex, you need an additional processor to lift the jsonPayload.severity field to the top-level severity field:

logging:
  ...
  processors:
    move_severity:
      type: modify_fields
      fields:
        severity:
          move_from: jsonPayload.severity
  service:
    pipelines:
      ..
        processors:
          ..
          - move_severity

Additionally, severity only recognizes specific inputs, e.g. a warning must be spelled as WARNING and not WARN or W. If you're collecting logs that spell the severity level differently, you can add a corresponding map_values section to the move_severity processor.