Tensorflow tf.nn.embedding_lookup

Question

is there a small neural network in tf.nn.embedding_lookup?? When I train some data, a value of the same index is changing. So is it trained also? while I'm training my model

I checked the official embedding_lookup code but I can not see any tf.Variables for train embedding parameter. But when I print all tf.Variables then I can found a Variable which is within embedding scope

Thank you.

Answer 1

Yes, the embedding is learned. You can look at the tf.nn.embedding_lookup operation as doing the following matrix multiplication more efficiently:

import tensorflow as tf
import numpy as np

NUM_CATEGORIES, EMBEDDING_SIZE = 5, 3
y = tf.placeholder(name='class_idx', shape=(1,), dtype=tf.int32)

RS = np.random.RandomState(42)
W_em_init = RS.randn(NUM_CATEGORIES, EMBEDDING_SIZE)
W_em = tf.get_variable(name='W_em',
                       initializer=tf.constant_initializer(W_em_init),
                       shape=(NUM_CATEGORIES, EMBEDDING_SIZE))

# Using tf.nn.embedding_lookup
y_em_1 = tf.nn.embedding_lookup(W_em, y)

# Using multiplication
y_one_hot = tf.one_hot(y, depth=NUM_CATEGORIES)
y_em_2 = tf.matmul(y_one_hot, W_em)

sess = tf.InteractiveSession()
sess.run(tf.global_variables_initializer())
sess.run([y_em_1, y_em_2], feed_dict={y: [1.0]})
# [array([[ 1.5230298 , -0.23415338, -0.23413695]], dtype=float32),
#  array([[ 1.5230298 , -0.23415338, -0.23413695]], dtype=float32)]

The variable W_em will be trained in exactly the same way irrespective of whether you use y_em_1 or y_em_2 formulation; y_em_1 is likely to be more efficient, though.