OpenSearch - Anomaly Detector

Question

Context:
There are four fields in my data stream (text), id, userName, loginDateTime, loginGeoLocation. A user shouldn't be allowed to log in from different machines (geoLocation), but they are logging in from multiple machines. Data Stream is coming from many external systems & we don't have any control over it, and landing on a Kafka topic, and a logstash pipeline is picking up from and pushing to OpenSearch Indice.

Problem Statement: I want to identify (alert) if some user login from a totally different location than is frequent. For example user-a login from Redwood, CA - but suddenly login from Boston, MS. On this action an alert trigger and send email/push/notify etc. How can we achieve this using pipeline, logstash or any method available with OpenSearch other than development or interceptor on stream.

Answer 1

After spending a lot of time, figured out two ways for this,

• Create an anomaly detection (https://www.elastic.co/guide/en/machine-learning/7.17/ml-configuring-detector-custom-rules.html)
• Use fingerprint filter from logstash while ingressing data (https://www.elastic.co/blog/logstash-lessons-handling-duplicates)
• Customise the plugin to use custom written rules (https://github.com/logstash-plugins/logstash-input-java_input_example)