Error when serving attention_ocr model ("error": "Expected one or two output Tensors, found 17")

Question

I'm trying to serve attention_ocr model on docker with tensorflow/serving image.

First, I trained this model with own dataset and get a good result with demo_inference.py

So, I'm export the trained model with export_model.py
python export_model.py --checkpoint=model.ckpt-111111 --export_dir=/tmp/mydir

Then, run docker container for serving the model.
docker run -it --rm -p 8501:8501 -v /tmp/mydir:/models/aocr -e MODEL_NAME=aocr --gpus all tensorflow/serving

And this is my python client script.

data_dir = '/root/src/models/research/attention_ocr/python/datasets/data/demo/'

data_files = os.listdir(data_dir)

with open(data_dir + "0.jpg", "rb") as image_file:
    encoded_string = base64.b64encode(image_file.read())

## Some requests I tried ##
# predict_request = '{"examples": [{"inputs": ["%s"]}]}' % encoded_string
# predict_request = '{"examples": [{"inputs": "%s"}]}' % encoded_string
predict_request = '{"examples": [{"inputs": {"b64": ["%s"]}}]}' % encoded_string

r = requests.post('http://MY_IP_ADDR:8501/v1/models/aocr:classify', data=predict_request)
print(r.text)

Result.. "error": "Expected one or two output Tensors, found 17"

This is the first time using tensorflow/serving. I can't handle this error.

Please help this newbie.. Thanks in advance.

Answer 1

Thank you for reporting this issue. I filed a bug (#9264) on Github on your behalf. The issue is that the default signature includes all the endpoints that the model provides. If you want to use the Serving's Classification API, we need to modify the export_model script to export just the 2 tensors expected by the classification API (i.e., predictions and scores). In the meantime, you can use the Predict API, which supports an arbitrary number of output tensors. Please note that when using the predict API via GRPC you can specify output_filter, but the RESTful API does not have that option, so the response is pretty heavy, since it sends back all the attention masks and the raw image. In case somebody else is trying to figure out how to run inference, here are steps that worked for me.

1. Export the model:

wget http://download.tensorflow.org/models/attention_ocr_2017_08_09.tar.gz
tar -xzvf attention_ocr_2017_08_09.tar.gz
python model_export.py --checkpoint=model.ckpt-399731 \
  --export_dir=/tmp/aocr_b1 --batch_size=1

Note that the --batch_size=1 is needed due to a bug in model_export.py. I'll take care of it when send the PR for the signature issue.

2. Run the docker container for serving the model.

sudo docker run -t --rm -p 8501:8501 \
  -v /tmp/aocr_b1:/models/aocr/1 -e MODEL_NAME=aocr tensorflow/serving

Please note that the path needs to contain a version number /models/aocr/1. If you don't append /1 the server complains that it could not find any versions.

3. Run the script

python send_serving_request.py --image_file=testdata/fsns_train_00.png

Here are the results

Prediction: Rue de la Gare░░░░░░░░░░░░░░░░░░░░░░░
Confidence: 0.899479449

Here is the code:

send_serving_request.py

from absl import app
from absl import flags
import base64
import json
import os
from PIL import Image
import numpy as np
import requests
import tensorflow as tf

flags.DEFINE_string('image_file', None,
                    'Name of file containing image to request.')


def create_serialized_tf_example(image):
  """Create a serialized tf.Example proto for feeding the model."""
  example = tf.train.Example()
  example.features.feature['image/encoded'].float_list.value.extend(
      list(np.reshape(image, (-1))))
  return example.SerializeToString()


def main(_):
  pil_image = Image.open(flags.FLAGS.image_file)
  encoded_string = base64.b64encode(
      create_serialized_tf_example(np.asarray(pil_image)))
  predict_request = (
      b'{"instances": [{"inputs": {"b64": "%s"}}]}') % encoded_string
  r = requests.post(
      'http://localhost:8501/v1/models/aocr:predict', data=predict_request)
  data = json.loads(r.text)
  print('Prediction:', data['predictions'][0]['predicted_text'])
  print('Confidence:', data['predictions'][0]['normalized_seq_conf'])


if __name__ == '__main__':
  flags.mark_flag_as_required('image_file')
  app.run(main)