Tensorflow: What's the accepted "clean" way to organize a large network?

Question

I have a pretty large fully connected network and I've started getting bothered by the fact that I store my weights and biases in a dictionary, and then compute each layer

layer_i+1 = relu(add(matmul(layer_i, weights['i']), biases['i']))

Surely there must be some "cleaner" way to do this? Or am I overthinking things?

Answer 1

I manage my networks the following way:

layers.py

vision = [
    ('conv', [5,5, 3,32], [32]),
    ('conv', [3,3,32,32], [32]),
    ('conv', [3,3,32,32], [32]),
    ('pool', 2),
    ('conv', [3,3,32,64], [64]),
    ('conv', [3,3,64,64], [64]),
    ('pool', 2),
    ('conv', [3,3,64,128], [128]),
    ('pool', 2),
    ('reshape', [-1,6*128]),
    ('dense', [6*128, 512], [512])
]


counter = [
    ('dense', [512, 256], [256]),
    ('dense', [256, max_digits], [max_digits])
]

tfmodel.py

def conv2d(x, W, b, strides=1, act='relu', name='convolution'):
    x = tf.nn.conv2d(x, W, strides=[1,strides,strides,1], padding="VALID", name=name)
    x = tf.nn.bias_add(x, b)

    if act=='relu':
        return tf.nn.relu(x)
    elif act=='tanh':
        return tf.nn.tanh(x)
    elif act=='softmax':
        return tf.nn.softmax(x)

def maxpool2d(x, k=2):
    return tf.nn.max_pool(x, ksize=[1,k,k,1], strides=[1,k,k,1], padding="VALID")

def process_network(X, layers, dropout, scope):
    with tf.variable_scope(scope):
        h = X
        i=0
        for layer in layers:
            if layer[0]=='conv':
                nameW = 'conv{}W'.format(i)
                nameb = 'conv{}b'.format(i)
                h = conv2d(h, tf.get_variable(nameW, layer[1], initializer=tf.random_normal_initializer()), tf.get_variable(nameb,layer[2], initializer=tf.random_normal_initializer()))
            elif layer[0]=='pool':
                h = maxpool2d(h, layer[1])
            elif layer[0]=='dense':
                nameW = 'dense{}W'.format(i)
                nameb = 'dense{}b'.format(i)
                h = tf.add(tf.matmul(h, tf.get_variable(nameW, layer[1], initializer=tf.random_normal_initializer())), tf.get_variable(nameb,layer[2], initializer=tf.random_normal_initializer()))
            elif layer[0]=='reshape':
                h = tf.reshape(h, layer[1])
            i = i+1
        h = tf.identity(h, 'out')
        return h

And while creating the graph, simply call like this:

h = tfmodel.process_network(image, layers.vision, 0.1, 'vision')
c_ = tfmodel.process_network(h, layers.counter, 0.1, 'counter')

This also creates a clean graph in TensorBoard. Its not complete but I'm sure you got the idea.

Another clean way is to use Keras to define layers or models. Check out Keras as a simplified interface to TensorFlow: tutorial