How to deploy different endpoints of different models to a single virtual machine in azure ml

Question

I have 6 different models trained. I registered models and created endpoints separately in azure ml and able to access endpoints and getting outputs. But noticed that each endpoint took one virtual machine separately. 6 endpoints took 6 virtual machines.

How can create or deploy my 6 endpoints to a single virtual machine. Because in future my models may increase.

Note: all my 6 models are different and each endpoint output is different.

Answer 1

If you have more than one model, then one possible alternate solution can be by creating a multi-model deployment.

With this all-model file are registered as a single model asset on Azure and loaded simultaneously in the scoring script and the scoring script parses each request for a "model" field and routes the payload accordingly.

Below is sample code snippet for the custom scoring script:

import joblib
import os
import pandas as pd
from pathlib import Path
import json

models = None

def init():
    global models
    model_dir = Path(os.getenv("AZUREML_MODEL_DIR")) / "models"
    models = {m[:-4]: joblib.load(model_dir / m) for m in os.listdir(model_dir)}


def run(data):
    data = json.loads(data)
    model = models[data["model"]]
    payload = pd.DataFrame(data["data"])
    try:
        ret = model.predict(payload)
        return pd.DataFrame(ret).to_json()
    except KeyError:
        raise KeyError("No such model")

For detailed example code, you can check this notebook.