I have an mlserver
server running and I am looking to make an inference request with a PyTorch
Model. It is a basic hello-world model.
The signature is:
signature:
inputs: '[{"name": "seq_input", "type": "tensor", "tensor-spec": {"dtype": "float32",
"shape": [-1, 5, 3]}}, {"name": "static_input", "type": "tensor", "tensor-spec":
{"dtype": "float32", "shape": [-1, 1]}}]'
I make a POST request:
{
"id": "model",
"inputs": [
{
"name": "seq_input",
"datatype": "FP32",
"parameters": {
"content_type": "np"
},
"shape": [1, 5, 3],
"data": [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 1.1, 1.2, 1.3, 1.4, 1.5]
},
{
"name": "static_input",
"datatype": "FP32",
"shape": [1, 1],
"parameters": {
"content_type": "np"
},
"data": [0.1]
}
]
}
The error from the server is:
{
"error": "builtins.TypeError: The PyTorch flavor does not support List or Dict input types. Please use a pandas.DataFrame or a numpy.ndarray"
}
I don't understand why this would not work. If I changed content_type
to pd
then I get:
{
"error": "mlflow.exceptions.MlflowException: Failed to enforce schema of data '{'seq_input': TensorData(__root__=[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 1.1, 1.2, 1.3, 1.4, 1.5]), 'static_input': TensorData(__root__=[0.1])}' with schema '['seq_input': Tensor('float32', (-1, 5, 3)), 'static_input': Tensor('float32', (-1, 1))]'. Error: This model contains a tensor-based model signature with input names, which suggests a dictionary input mapping input name to a numpy array, but a dict with value type <class 'mlserver.types.dataplane.TensorData'> was found."
}
Documentation for serving PyTorch
models with the mlflow
runtime is quite scarce. Any help appreciated.