getting logs from a file with Ops Agent

Question

I have a python script on a vm that writes logs to a file and I want to use them in the google logging.

I tried this config yaml:

logging:
  receivers:
    syslog:
      type: files
      include_paths:
      - /var/log/messages
      - /var/log/syslog
    etl-error-logs:
      type: files
      include_paths:
      - /home/user/test_logging/err_*
    etl-info-logs:
      type: files
      include_paths:
      - /home/user/test_logging/out_*
  processors:
  etl_log_processor:
    type: parse_regex
    field:       message
    regex:       "(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})\s(?<severity>INFO|ERROR)\s(?<message>.*)"
    time_key:    time
    time_format: "%Y-%m-%d %H:%M:%S"
  service:
    pipelines:
      default_pipeline:
        receivers: [syslog]
      error_pipeline:
        receivers: [etl-error-logs]
        processors: [etl_log_processor]
        log_level: error
      info_pipeline:
        receivers: [etl-info-logs]
        processors: [etl_log_processor]
        log_level: info
metrics:
  receivers:
    hostmetrics:
      type: hostmetrics
      collection_interval: 60s
  processors:
    metrics_filter:
      type: exclude_metrics
      metrics_pattern: []
  service:
    pipelines:
      default_pipeline:
        receivers: [hostmetrics]
        processors: [metrics_filter]
      error_pipeline:
        receivers: [hostmetrics]
        processors: [metrics_filter]
      info_pipeline:
        receivers: [hostmetrics]
        processors: [metrics_filter]

and this is an example of the logs: 2021-11-22 11:15:44 INFO testing normal

I didn't fully understand the google docs so I created the yaml as best as I understood and with a reference to their main example but I have no idea why it doesn't work

environmen:GCE VM

You want to use those logs in GCP Log Viewer: yes

Which docs did you follow: https://cloud.google.com/stackdriver/docs/solutions/agents/ops-agent/configuration#logging-receivers

How did you install OpsAgent: in gce I entered each vm instance went to observability and there was the option to install ops agent in cloud shell

What logs you want to save: I want to save all of the logs that are being written to my log file live.

specific applications logs: its an etl process that runs in python and saves its logs to a local file on the vm

Answer 1

sudo journalctl -xe | grep "google_cloud_ops_agent_engine"

Try out this command it should show you the exact(almost) error