I've been using appstats to examine some calls. I have a simple view that makes a memcache call and returns the result. Appstats tells me that the total time (Grand Total) is around 15ms. However, what I observe in the browser is more like 242ms or so. And actually, I also get the same result using apache bench. I tried another network (using an ec2 instance) to see if I would see different round trip time and also got about the same result. A ping to the server takes about 13 or 14ms.
I am definitely missing something here. There is around 180-200ms of latency that I can't account for. Based on experience w/ app engine and appstats, I'm hoping someone can help me open my eyes to what I am missing.
Some environment details if interested ... python 2.7; used both webapp2 and Flask in tests; my apache bench tests have all been single threaded hitting 100 times.
Thanks for any insight.
In my experience, you will always see a difference between appstats and total time. You can see this in the logs. Where the total latency is listed. Appstats gives you the time it took for your code to process the request but the logs list total i/o time. From what I can workout the increased time is down to load balancing and the way app engine internally queues requests and assesses if it needs to scale or not. Also if you send a burst of 100 request you may actually make the difference more noticeable because depending on your app settings it is most likely going to serve those request on the minimum instances before a new instance has started.
A good rule of thumb if you want your app to perform well is have appstats below 100ms and the rest is down to how much money you want to burn.
Would be interested in the official line but that's what it seems like to me after 3 years working with app engine