How to get log message in separated field whih logstash

Question

I'm trying to get my log in a specific way in kibana from logstash but I'm not succeedinig.

I will explain by example. Then if I have in my log file this line :

2015-06-17 13:08:45      INFO    connectionpool  Starting new HTTPS connection (1): SSX9FSMULE.algolia.net

I would like to have these fields and values in kibana :

DateTime : 2015-06-17 13:08:45
LogType : INFO
ModuleName : connectionpool
Message :   connectionpool  Starting new HTTPS connection (1): SSX9FSMULE.algolia.net

The datetime field is not really important as by default ES include a timestamp

I try this code :

if [type] == "mylog" {
    grok {
      match => { "message" => "%{MYLOGTIME:DateTime} %{LOGLEVEL:LogType} %{WORD:ModuleName}  %{GREEDYDATA:Message}" }
    }
  }

with a file mylog in the patterns directory containing this line :

MYLOGTIME %{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}

But it didn't work and I got a no one of the above desired fields in kibana, only the message

2015-06-17 13:08:45      INFO    connectionpool  Starting new HTTPS connection (1): SSX9FSMULE.algolia.net

I tried this :

if [type] == "autosuggest" {
    grok {
      match => { "message" => "%{GREEDYDATA:Message}" }
      add_field => [ "LogType", "%{LOGLEVEL}" ]
      add_field => [ "ModuleNAme", "%{WORD}" ]
    }
  }

and got this in kibana :

LogType : %{LOGELEVEL}
ModuleName : %{WORD}
Message :  2015-06-17 13:08:45      INFO    connectionpool  Starting new HTTPS connection (1): SSX9FSMULE.algolia.net

And in the standard kibana message field I have the message above :

 2015-06-17 13:08:45      INFO    connectionpool  Starting new HTTPS connection (1): SSX9FSMULE.algolia.net

While I would like to have only the last part of my message (the part after module name) as message.

I'm googling since several hours whitout success

Please can some one explain how to get the part of the message I want with grok, and how to achieve what I'm trying to do here.

Thanks

Answer 1

Finally I did this, and it works :

if [type] == "mylog" {
    grok {
      match => { "message" => "%{MYLOGDATETIME:my_time}\s+ %{LOGLEVEL:my_type}\s+ %{WORD:my_module_name}\s+ %{GREEDYDATA:my_msg}" }
      add_field => [ "DateTime", "%{my_time}" ]
      add_field => [ "LogType", "%{my_type}" ]
      add_field => [ "ModuleNAme", "%{my_module_name}" ]
      add_field => [ "Message", "%{my_msg}" ]
    }
  }

Answer 2

It looks like the problem you are having is related to spaces. If you use a pattern like this:

%{DATESTAMP:DateTime}\s+%{LOGLEVEL:LogType}\s+%{WORD:ModuleName}\s+%{GREEDYDATA:Message}

It should skip over those multiple spaces. \s+ means one or more white spaces (which can include tabs).