How to monitor API calls on EC2?

Question

I have a requirement to find response time for API (Rest API calls to external instances) calls going out of an amazon EC2 instance (there is an application running on EC2 making these calls). It will be great if I could also filter the calls based on a regex or complete urls. We have been thinking of logging the calls and analyzing the data or using tools like Dynatrace, Nagios so that code changes are not required. If someone has implemented such a solution, please let us know.

Answer 1

This may not be a complete answer, but following the idea of starting with logs, I would recommend looking into using Cloudwatch: https://aws.amazon.com/cloudwatch/details/#log-monitoring

Answer 2

Depending on the level of access you have, you use something like https://www.wireshark.org/ to monitor all the traffic and do url/protocol filtering. You might try a log aggregator like http://papertrailapp.com/ which has filtering.

As you stated you could also use some form of APM like dynatrace, statsd, instrumental[1], newrelic, datadog, etc. to do monitoring inside your application.

---

[1] I work for instrumental.

Answer 3

You can setup a simple proxy that would measure web requests time and write metrics for this measures.

There are some some nice tools, take a look at http://datadoghq.com/product or http://devmetrics.io/logslib

For example, simple nodejs proxy with devmetrics lib:

var httpProxy = require('http-proxy');
var logger = require('devmetrics-core');
var http = require('http');
var proxy = new httpProxy.createProxyServer({});
var proxyServer = http.createServer(function (req, res) {
  // now
  var start_time = new Date().getTime();
  proxy.web(req, res, { target: 'http://localhost:80' });
  res.on('finish', function() {
    var latency = new Date().getTime() - start_time;
    console.log("The request was proxied in " + latency + "ms");
    logger.appGauge('web_request', latency);
  });
});
proxyServer.listen(3000);