Large log aggregation and count, what tools should I choose?

72 Views Asked by At

Input: Apache Kafka with Ngnix log(milliseconds level) from multiple servers, the logs is generally ordered by time, but not exactly.

Proceed: What tools or tech. should I choose ???

Output: Text with format: URL time(minute level) count

e.g.

Input: 2015-06-24 12:23:34.001 /rest/login?user=xxx HTTP/1.0 ...

Output: /rest/login "2015-06-24 12:23" 2

Thanks very much for your reply

BTW. parsing the log is not the problem, but how to do the aggregation and count.

0

There are 0 best solutions below