V2 Inference Protocol - PyTorch flavour datatype error

57 Views Asked by At

I have an mlserver server running and I am looking to make an inference request with a PyTorch Model. It is a basic hello-world model.

The signature is:

signature:
  inputs: '[{"name": "seq_input", "type": "tensor", "tensor-spec": {"dtype": "float32",
    "shape": [-1, 5, 3]}}, {"name": "static_input", "type": "tensor", "tensor-spec":
    {"dtype": "float32", "shape": [-1, 1]}}]'

I make a POST request:

{
   "id": "model",
  "inputs": [
    {
      "name": "seq_input",
      "datatype": "FP32",
            "parameters": {
        "content_type": "np"
      },
      "shape": [1, 5, 3],
      "data": [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 1.1, 1.2, 1.3, 1.4, 1.5]
    },
    {
      "name": "static_input",
      "datatype": "FP32",
      "shape": [1, 1],
                  "parameters": {
        "content_type": "np"
      },
      "data": [0.1]
    }
  ]
}

The error from the server is:

{
    "error": "builtins.TypeError: The PyTorch flavor does not support List or Dict input types. Please use a pandas.DataFrame or a numpy.ndarray"
}

I don't understand why this would not work. If I changed content_type to pd then I get:

{
    "error": "mlflow.exceptions.MlflowException: Failed to enforce schema of data '{'seq_input': TensorData(__root__=[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 1.1, 1.2, 1.3, 1.4, 1.5]), 'static_input': TensorData(__root__=[0.1])}' with schema '['seq_input': Tensor('float32', (-1, 5, 3)), 'static_input': Tensor('float32', (-1, 1))]'. Error: This model contains a tensor-based model signature with input names, which suggests a dictionary input mapping input name to a numpy array, but a dict with value type <class 'mlserver.types.dataplane.TensorData'> was found."
}

Documentation for serving PyTorch models with the mlflow runtime is quite scarce. Any help appreciated.

0

There are 0 best solutions below