how i can write a shell script to submit job in splunk and wait to job to be finish and after download the csv file from splunk

Question

I am running below shell script command to submit job in splunk ,after first curl command execution, on splunk UI it is taking some time (not constant) to complete the job then only report.csv file will be ready on splunk to download. I have given sleep for 3600s but some time it is taking more than that . How i can achieve this.

code ####### #!/bin/bash

jobid=curl -sS -k -u username:password https://XXXXX.splunkcloud.com:8098/XXXXXjobs/ -d search="search index="prod" source="*log*" earliest=-1d@d latest=-0d@d | timechart count span=1s | sort - count | head 1"

jobid=${sid#*}

jobid=echo $sid | cut -d"<" -f1

sleep 3600

curl -sS -k -u username:password https://XXXXX.splunkcloud.com:8098/XXXXX/jobs/$jobid/results?output_mode=csv&count=0" > report.csv

I tried putting sleep more time but the problem is on splunk UI after some time job will be overwritten by recent job , so only way to achieve it , download the file as soon as job finished on splunk.

Answer 1

Use splunk REST API

curl --location --request POST 'https://splunkhost:8389/services/search/jobs' \
--header 'Authorization: Basic XXXXXXX' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'search=search index="test" | table *' \
--data-urlencode 'exec_mode=oneshot' \
--data-urlencode 'output_mode=json'

more details - https://docs.splunk.com/Documentation/Splunk/9.0.3/RESTREF/RESTsearch#search.2Fjobs