Azure Cloud Service Performance Differences

Question

We currently have the following setup for our webapp:

Pre-Production:

1 Medium instance (web role running windows 2012) 1 Web Tier SQL Server Storage account for blobs

Live:

2 Large Instances (web role running windows 2012) 1 Web Tier SQL Server Storage account for blobs (shares the same account as pre-prod)

All the servers, storage accounts are located in the same region (North Europe)

The problem we have is that the pre-production server is faster at serving pages than the live setup but we cannot figure out why. They are both setup exactly the same, they simply point to different databases.

Any example of the difference is that the main dashboard page takes 300ms in the pre-prod box and 600 ms on the live. This was tested when bot apps were not being used by anyone else apart from us so it can't be down to load on the box.

We also have in-role caching on both setups (30% of the memory) Looking at the server it's self neither CPU usage or RAM usage is excessive.

Does anyone know of anything obvious that we should be checking as to why the live deployment is so much slower than pre-prod?

Answer 1

The actually issue turned out to be some logging we had going on in the app. The log server was in the US and on a shared platform which explained why we had the delay on live. It looks like when we first checked this as something that might be causing it the performance of the logging setup was fine but think that was probably due to the time of day it was checked.

Answer 2

Hope all is well! :)

There are two main things I'd be looking at:

1. Depending on how you manage your pre-production DB and how old your site is, your production database might have a lot more data in it - so when you're calculating your stats it takes a lot longer to do. In my experience, the Web and Standard databases have weak performance - worse than you'd expect from that 10 year old server running SQL 2000. So when you are running calculations or gathering lots of data it really does make a big difference on those versions. The Business edition is much faster and similar to the P1, but with variable performance so it might be worth upgrading to that.
2. Is the database hosted on the same server? The main (official) reason for moving from the web/business model to the basic/standard/premium pricing models was to try to give consistent and reliable performance. If you've got one of your databases on another server, you could be sharing those resources with someone that is hammering the database and slowing down your performance.

I have found that the database is the bottleneck on Azure and that 2 Medium instances cope comfortably with 10,000 or so visitors a day, so you might be able to lower the cost of those instances to balance out the raised costs on using a Business database.

The only other thing that might be useful is that, in the windows azure management, under database > monitor, they upgraded the metrics a little while back to include things like CPU %, IO %, etc but they're not on the graph by default. If you add those metrics in it might give you a bit more information to debug.

Was there also a reason for going for North Europe over West Europe?