How can we monitor a process with cloudwatch

Question

I have a Java process that runs on EC2 I would like to setup an alert in Cloudwatch when the process goes down or is in a bad state (e.g does not send heartbeat to Cloudwatch for the last 10 secs or so).

What is the best way to do this ? I think I need the custom metrics, but did not find any documentation for specifically monitoring a process.

I can use the AWS SDK if needed.

Answer 1

You can write a custom script with ps or jps and push that metric to Cloudwatch. BUT if you are looking for 10 seconds granularity, then Cloudwatch is not the right solution since its minimum granularity is 60 seconds.

From: AWS Resource and Custom Metrics Monitoring

Q: What is the minimum granularity for the data that Amazon CloudWatch receives and aggregates?

The minimum granularity supported by CloudWatch is 1 minute data points. Many metrics are received and aggregated at 1-minute intervals. Some are received at 3-minute or 5-minute intervals.

Though it is possible to create an alarm using CLI and SDK, I suggest you use the AWS Cloudwatch dashboard. Wait for your custom metric to appear in Cloudwatch dashboard. After you see your custom metrics in Cloudwatch, click on CreateAlarm and select your metric. After that define your alarm. The attached image shows Applications as the metric. In your case, it will be whatever name you choose to call it. Under Actions, create a new notification and specify your email. Now if the count goes below 1 for one period, you will get an alarm.

Answer 2

The best way to monitor a process will be using AWS CloudWatch procstat plugin. First create a CloudWatch configuration file with PID file location from EC2 and monitor the memory_rss parameter of process. The idea is memory consumption metric will never go below or equal to zero for a running process.

{
    "agent": {
            "run_as_user": "cwagent"
    },
    "metrics": {
            "metrics_collected": {
                "procstat": [
                {
                    "pid_file": "/var/run/sshd.pid",
                    "measurement": [
                        "cpu_usage",
                        "memory_rss"
                    ]
                }
                ]
            }
    }

}

Later start the CloudWatch Agent and configure the ALARM using this AWS documentation!

Answer 3

AWS Custom Metrics can be used to publish the health of the Program.

Below Java Code can be used to Publish the Heart Beat. Using Custom Metrics Alarm can be configured in CloudWatch.

AmazonCloudWatch amazonCloudWatch = AmazonCloudWatchClientBuilder.standard().
                                            withEndpointConfiguration(new AwsClientBuilder.
                                                    EndpointConfiguration("monitoring.us-west-1.amazonaws.com","us-west-1")).build();
        PutMetricDataRequest putMetricDataRequest = new PutMetricDataRequest();
        putMetricDataRequest.setNamespace("CUSTOM/SQS");


        MetricDatum metricDatum1 = new MetricDatum().withMetricName("MessageCount").withDimensions(new Dimension().withName("Personalization").withValue("123"));
        metricDatum1.setValue(-1.00);
        metricDatum1.setUnit(StandardUnit.Count);
        putMetricDataRequest.getMetricData().add(metricDatum1);

        PutMetricDataResult result = amazonCloudWatch.putMetricData(putMetricDataRequest);