Azure ML: Getting Error 503: NoMoreResources to any web service API even when I only make 1 request

Question

Getting the following response even when I make one request (concurrency set to 200) to a web service.

{ status: 503, headers: '{"content-length":"174","content-type":"application/json; charset=utf-8","etag":"\"8ce068bf420a485c8096065ea3e4f436\"","server":"Microsoft-HTTPAPI/2.0","x-ms-request-id":"d5c56cdd-644f-48ba-ba2b-6eb444975e4c","date":"Mon, 15 Feb 2016 04:54:01 GMT","connection":"close"}', body: '{"error":{"code":"ServiceUnavailable","message":"Service is temporarily unavailable.","details":[{"code":"NoMoreResources","message":"No resources available for request."}]}}' }

The request-response web service is a recommender retraining web service with the training set containing close to 200k records. The training set is already present in my ML studio dataset, only 10-15 extra records are passed in the request. The same experiment was working flawlessly till 13th Feb 2016. I have already tried increasing the concurrency but still the same issue. I even reduced the size of the training set to 20 records, still didn't work.

I have two web service both doing something similar and both aren't working since 13th Feb 2016.

Finally, I created a really small experiment ( skill.csv --> split row ---> web output ) which doesn't take any input. It just has to return some part of the dataset. Did not work, response code 503.

The logs I got are as follows

{ "version": "2014-10-01", "diagnostics": [{ ..... { "type": "GetResourceEndEvent", "timestamp": 13.1362, "resourceId": "5e2d653c2b214e4dad2927210af4a436.865467b9e7c5410e9ebe829abd0050cd.v1-default-111", "status": "Failure", "error": "The Uri for the target storage location is not specified. Please consider changing the request's location mode." }, { "type": "InitializationSummary", "time": "2016-02-15T04:46:18.3651714Z", "status": "Failure", "error": "The Uri for the target storage location is not specified. Please consider changing the request's location mode." } ] }

What am I missing? Or am I doing it completely wrong?

Thank you in advance.

PS: Data is stored in mongoDB and then imported as CSV

Answer 1

This was an Azure problem. I quote the Microsoft guy,

We believe we have isolated the issue impacting tour service and we are currently working on a fix. We will be able to deploy this in the next couple of days. The problem is impacting only the ASIA AzureML region at this time, so if this is an option for you, might I suggest using a workspace in either the US or EU region until the fix gets rolled out here.

To view the complete discussion, click here