How to do live data analysis?

Question

I have worked with R to do data analysis for making prediction model. Now I need to build solution for live data analysis. That's find patterns from data and show an alert if some condition is going to occur.

Like if I am receiving RSSI from the sensors. First find patterns if it has any relation with battery level and show alert if the live data shows that pattern from live RSSI data.

I did a little search and in one reply on Quora, a linked tool Kafka is mentioned for doing such stuff. I don't have any experience of working with Kafka.

How can this be done?

Answer 1

You could start by exploring KSQL? Not knowing your schema here's a naive query against a theoretical Kafka topic with live RSSI data:

CREATE STREAM rssi_data 
    (device_id VARCHAR,
     battery_level INT,
     SIGNAL FLOAT)
 WITH (TOPIC='rssi_data', VALUE_FORMAT='json', KEY='device_id');

and then query it:

CREATE STREAM low_battery_devices AS SELECT * FROM rssi_data WHERE battery_level < 20;

That would create a new Kafka topic called low_bettery_devices which would contain events for the devices who's battery is lower. You can then of course consume that topic with another KSQL query, or an R program or some other downstream Kafka compatible system.