(Tensorflow)Does the op assign change the gradient computation?

Question

I use the op "assign" to change the value of variables instead of "=", but I found the gradient I got is quite different. Could anyone tell me the difference and why? thanks! Like change w = w1 to op1 = tf.assign(w, w1) sess.run(op1)

Answer 1

= and tf.assign are different operations.

= is a python operation, in which you assign a python value to a python variable

tf.assign is a Tensorflow operation that assigns the value to the variable ref and returns the assign operation.

= is executed in python and doesn't affect the computation graph. tf.assign is a node in the computational graph.

To understand, let's run this simple script

import tensorflow as tf

x = tf.Variable(1)
y = tf.Variable(2)

x = y

print(x.name, y.name)

a = tf.Variable(1)
b = tf.Variable(2)

# override a, otherwise a content is 1
a = a.assign(b)
print(a.name, b.name)

init = tf.global_variables_initializer()
with tf.Session() as sess:
    sess.run(init)
    print(sess.run([x, y, a, b]))

print(x.name, y.name) outputs Variable_1:0 Variable_1:0

because = is executed in python and you've overwritten the variable x.

print(a.name, b.name) outputs Assign:0 Variable_3:0 because you defined an assign op in the computational graph, now a is an assign op.

When you run the defined graph, you get:

[2, 2, 2, 2]

But these values are computed differently: one is a computation in the graph, the others no.

If you forgot to assign a to the assign op created with tf.assign (thus you change the line a = a.assign(b) to a.assign(b)), then when you evaluate the graph, you'll get:

[2, 2, 1, 2]