How to batch the app events with max of 1000 events per second when using Splunk HTTP Event Collector (HEC)

Question

I need to send the batched events to Splunk HTTP Event Collector, say 1000 events per second.

Below is the example of 5 log events that are sent to Splunk HEC -

% curl "https://splunk-example.com:8088/services/collector/raw?channel=093DCD-BC98-8UET-8AFE-8413C3825C4C&sourcetype=test_type&index=test_index"
-H "Authorization: Splunk ******-****-****-****-*********" -d '<log line 1>   <log line 2>  <log line 3>  <log line 4>

  Output:  {"text":"Success","code":0}%

So how do I configure to send the application logs as 1000 events/seconds to Splunk HEC?

Answer 1

In your props.conf file for the sourcetype you should use "SHOULD_LINEMERGE = false". That'll break each log line into separate events.

Now, what if your event is multi line?

Then this document should help: https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/Configureeventlinebreaking

Answer 2

The send rate configuration i.e throttling should be done on the client side. I would assume that you will not be using "plain" curl, but rather it will be some sort of script/application. So your throttling needs to be done there.