Issue converting tensorflow into unity backend (barracuda)

Question

I'm trying to convert a UNet model that I created with Keras into a .nn for use in unity's neural networking backend. However I'm getting this error. For my model export I exported an '.h5' which I converted into a binary '.pb', and later I used the tensorflow_to_barracuda.py. Is there maybe someone with a working segmentation program in unity?

Converting unet_person.bytes to unet_person.nn
IGNORED: PlaceholderWithDefault unknown layer
IGNORED: Switch unknown layer
IGNORED: Switch unknown layer
IGNORED: Shape unknown layer
IGNORED: Switch unknown layer
IGNORED: Merge unknown layer
IGNORED: Shape unknown layer
IGNORED: Shape unknown layer
---------------------------------------------------------------------------
UnboundLocalError                         Traceback (most recent call last)
<ipython-input-22-d09d8c6d2c1a> in <module>
      1 from mlagents.trainers import tensorflow_to_barracuda as tb
      2 
----> 3 tb.convert('unet_person.bytes', 'unet_person.nn')

/anaconda3/lib/python3.6/site-packages/mlagents/trainers/tensorflow_to_barracuda.py in convert(source_file, target_file, trim_unused_by_output, verbose, compress_f16)
938     o_model = barracuda.Model()
939     o_model.layers, o_input_shapes, o_model.tensors, o_model.memories = \
--> 940         process_model(i_model, args)
941 
942     # Cleanup unconnected Identities (they might linger after processing complex node patterns like LSTM)

/anaconda3/lib/python3.6/site-packages/mlagents/trainers/tensorflow_to_barracuda.py in process_model(model, args)
870                 nodes = nodes_as_array[node_index:pattern_end]
871                 name = nodes[-1].name
--> 872                 var_tensors, const_tensors = get_tensors(nodes)
873                 if args.print_patterns or args.verbose:
874                     print('PATTERN:', name, '~~', pattern_name, pattern, '<-', var_tensors, '+', [t.name for t in const_tensors])

/anaconda3/lib/python3.6/site-packages/mlagents/trainers/tensorflow_to_barracuda.py in get_tensors(pattern_nodes)
845                 tensor_nodes = [n for n in pattern_nodes if n.op == 'Const']
846                 tensors = [Struct(name = n.name, obj = n.attr["value"].tensor, shape = get_tensor_dims(n.attr["value"].tensor), data = get_tensor_data(n.attr["value"].tensor))
--> 847                     for n in tensor_nodes]
848 
849                 # TODO: unify / reuse code from process_layer

/anaconda3/lib/python3.6/site-packages/mlagents/trainers/tensorflow_to_barracuda.py in <listcomp>(.0)
845                 tensor_nodes = [n for n in pattern_nodes if n.op == 'Const']
846                 tensors = [Struct(name = n.name, obj = n.attr["value"].tensor, shape = get_tensor_dims(n.attr["value"].tensor), data = get_tensor_data(n.attr["value"].tensor))
--> 847                     for n in tensor_nodes]
848 
849                 # TODO: unify / reuse code from process_layer

/anaconda3/lib/python3.6/site-packages/mlagents/trainers/tensorflow_to_barracuda.py in get_tensor_data(tensor)
492     if tensor.bool_val:
493         data = np.array(tensor.bool_val, dtype=float)
--> 494     return np.array(data).reshape(dims)
495 
496 def flatten(items,enter=lambda x:isinstance(x, list)):

UnboundLocalError: local variable 'data' referenced before assignment

Answer 1

In Barracuda 1.0, there is a way to convert Keras (.h5) models into ONNX models with the use of the Keras2ONNX pip package.

You install keras2ONNX and then run

import keras2onnx
onnx_model = keras2onnx.convert_keras(unet, name='unet')
keras2onnx.save_model(onnx_model, "unet.onnx")

Note that you made need the following flag: channel_first_inputs=[unet.layers[0].layers[0]]

onnx_model = keras2onnx.convert_keras(unet, name='unet')

Since Barracuda inputs are channel first, meaning that say for a batch_size x width x height x rgb image, the ordering is rgb x width x height x batch_size.

Answer 2

I found out that this framework wasn't developed far enough yet. What worked for me was compiling the Tensorflow Lite source for all platforms and using that backend. Converting to Tensorflow Lite is still a bit tricky, because only certain layers are supported. Lastly you need to wrap the C binaries in C# which is already partially done for you here: https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/experimental/examples/unity

Compiling is relatively easy.