I need to treat each build log as a single event in Splunk

Question

I am using Splunk App for Jenkins. I need to treat each build log as a single event in Splunk. Every time I get the data in Splunk, it breaks it on timestamp.

Jenkins console log

Started by timer
Running as SYSTEM
Building in workspace /var/lib/jenkins/workspace/Common/Server_UI_UAT_UD
[Server_UI_UAT_UD] $ /bin/sh -xe /tmp/jenkins48606894436586402.sh
+ export PYTHONPATH=:/var/www/html/atc/
+ PYTHONPATH=:/var/www/html/atc/
+ /var/www/html/venv_atc/bin/python /var/www/html/atc/csweep/scripts/scheduler/system_health/send_alert.py

Splunk logs

Expected: I need to treat each build log as a single event in Splunk. Every time I get the data in Splunk, it breaks it on timestamp.

Answer 1

This is not how Splunk works.

The behavior you get is expected and it's correct.

Splunk agent streams the diff of the log data to the Splunk server (or log collector that can perform pre-processing and routing) once new entries appear on the log. That ensures timely data ingestion, reasonable usage of the resources (both network and Splunk internal DB), and prevents the events from being lost.

Every full build log size could be hundreds of MBs so what you want is not feasible.

To distinct one build log from another, you are provided with source field.