How to write a custom multiline parser for Google Cloud Ops Agent

434 Views Asked by At

I am trying to configure google-cloud-ops-agent to collect log records from an application. Logs are one line or multiline.

Example multiline log:

[2023-02-08 10:32:23] production.ERROR: App\Jobs\Message::handle (Symfony\\Component\\Mime\\Exception\\RfcComplianceException(code: 0): Email \"3712\" does not comply with addr-spec of RFC 2822. at /var/www/html/vendor/symfony/mime/Address.php:54)
[stacktrace]
#0 /var/www/html/vendor/laravel/framework/src/Illuminate/Mail/Message.php(244): Symfony\\Component\\Mime\\Address->__construct()
#1 /var/www/html/vendor/laravel/framework/src/Illuminate/Mail/Message.php(108): Illuminate\\Mail\\Message->addAddresses()
[2023-02-08 20:53:30] production.ERROR: RuntimeException: User not found in /var/www/html/app/Services/UserService.php:82
Stack trace:
#0 /var/www/html/app/Services/UserService.php(63): App\Services\UserService->init()
#1 /var/www/html/app/Helpers/UserHelper.php(128): App\Services\UserHelper->__construct()

The configuration I have is as follows:

logging:
  receivers:
    customFile:
      type: files
      include_paths:
      - /var/www/html/storage/logs/error.log
  processors:
    extract_structure:
      type: parse_regex
      field: message
      regex: "^\[(?<time>[\d-]*\s[\d:]*)\]\sproduction.(?<severity>[^ :]*):\s(?<message>[\s\S]*)$"
      time_key: time
      time_format: "%Y-%m-%d %H:%M:%S"
    move_severity:
      type: modify_fields
      fields:
        jsonPayload."logging.googleapis.com/severity":
          move_from: jsonPayload.severity
  service:
    pipelines:
      default_pipeline:
        receivers: [customFile]
        processors: [extract_structure, move_severity]

The logs appear in Log Explorer as one line per entry.

Expecting to see all the lines starting from the first date and time until the next date and time occurrence in one Log Explorer entry

Supported languages are: java, python and go. I need for PHP

I've tried the Java multiline parse config

parse_java_multiline:
      type: parse_multiline
      match_any:
      - type: language_exceptions
        language: java

It didn't work

0

There are 0 best solutions below