How to view the log rows in group/session in Kibana's Discover UI?

Question

I'm using elasticsearch + kibana + logstash + filebeat latest 6.4.1 to collect and analyze web logs. The columns of my log are like:

timestamp, http_method, request_uri, http_status, host, user_agent, client_ip, client_port

I have configured ELK to show my logs in Kibana. But now I want to see my logs in sessions. I hope the log lines can be grouped by session and shown in Kibana's Discover page. In my scenario, the log lines with the same (host, client_ip) belong to the same session.

I hope the Discover with session/grouping UI can show all the sessions. And I can still see the log lines inside one session when I click it.

Is it possible to do it? Or what's the best way to do it? Thanks.

Answer 1

Discover shows you the log entries as raw documents as they are stored, you can save the search on Discover, even without filtering anything. Through Vizualize you can create widgets, these can aggregate data like sessions. I usually make one or two "visual builder" for timelines and a data table for (request_uri, host, user_agent, client_ip) and a pie for http_status. With these visuals you can create a Dashboard and include the search result from discover. On this dashboard you can search and filter and if you add the search from discover, the dashboard will show the raw data for each will still show. I only use discover after adding new indices.

https://www.elastic.co/guide/en/kibana/current/visualize.html

https://www.elastic.co/guide/en/kibana/current/dashboard.html

If you really want the index contain the host/client_ip as a aggregate you have to group the data before you store it in elasticsearch. A Dashboard sounds like a better idea.

EDIT: As Waleed Ali described, the visualize could look like this

Answer 2

as @sleepyhead suggested, Discover is not for the purpose, you are looking for. You can create a Datatable from Visualize in Kibana (which, I guess, you have already done).

Next, In the Bucket Aggregations section, you can choose Split Rows and go for Terms Aggregation. Select your desired field (host) there. Then add a sub-bucket and again, choose Split Rows then Terms Aggregation and select the other field (client_ip).

This will create one row in your data table for each unique set of both the fields (host, client_ip), which is what you require as far as I have understood your question and comment.

By default, there will only be a single column (Count), But, You can obviously add more columns by adding Metric Aggregations.