Logstash replace @timestamp with syslog date

Question

I'm a bit confused. I'm trying to pull out the syslog date (backfilling the logstash) and replace the @timestamp with it. I've tried almost everything.

This is my filter

filter {
   if [type] == "syslog" {
   grok {
     match => {
"message" => ["%{SYSLOGTIMESTAMP:DATETIME} %{WORD:SERVER} (?<BINARY>(.*?)(php\-cgi|php))\: %{DATA:PHP_ERROR_TYPE}\:\s\s(?<PHP_ERROR_DESC>(.*?)(e\s\d))""]
  }
}

date {
  match => { "DATETIME" => [ "MMM  d HH:mm:ss", "MMM dd HH:mm:ss", "ISO8601" ] }
  target => "@timestamp"
  add_tag => [ "tmatch" ]
}

if !("_grokparsefailure" in [tags]) {
  mutate {
    replace => [ "@source_host", "%{SERVER}" ]
  }
}
mutate {
  remove_field => [ "SERVER" ]
}
}
}

sample output:

{
    "message" => "Sep 10 00:00:00 xxxxxxx",
    "@timestamp" => "2013-12-05T13:29:35.169Z",
      "@version" => "1",
          "type" => "xxxx",
          "host" => "127.0.0.1:xxx",
      "DATETIME" => "Sep 10 00:00:00",
        "BINARY" => "xxxx",
"PHP_ERROR_TYPE" => "xxxx",
"PHP_ERROR_DESC" => "xxxxx",
          "tags" => [
    [0] "tmatch"
],
  "@source_host" => "xxx"
}

tmatch is in the tags so I assume that the date filter works, but why do I still have:

@timestamp => "2013-12-05T13:29:35.169Z"

?

Thanks for help (my logstash is logstash-1.2.2-flatjar.jar)

Answer 1

Let's take a look at your date filter:

date {
  match => { "DATETIME" => [ "MMM  d HH:mm:ss", "MMM dd HH:mm:ss", "ISO8601" ] }
  target => "@timestamp"
  add_tag => [ "tmatch" ]
}

In particular, the match parameter:

match => { "DATETIME" => [ "MMM  d HH:mm:ss", "MMM dd HH:mm:ss", "ISO8601" ] }

Match expects an array. I'm not sure what you're passing, exactly, but it's definitely not an array. I tried running this with -v, and I'm surprised to see it doesn't complain.

You probably mean something closer to this:

match => ["DATETIME", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss", "ISO8601"]

Note the first element of the array is the target field; additional elements are pattern(s) to match against.

Past that, you really only need to pass the one format you expect, but it looks like that's included among the three you're sending.

Answer 2

If you want the timestamp showed as your timezone format, instead of UTC time, you can do like this

ruby {
    code => "event['@timestamp'] = event['@timestamp'].local('-08:00')"
}

Before:@timestamp => "2013-12-05T13:29:35.169Z"

After :@timestamp => "2013-12-05T05:29:35.169-08:00"

Updated: The local method can't work in version 1.4.2. So, change another API

ruby {
    code => "event['@timestamp'] = event['@timestamp'].getlocal"
}