How do I exclude bias from regularization using TensorFlow's `matrix_solve_ls`?

Question

Given a design matrix (with an initial column of 1s corresponding to a bias factor), I can perform standard NN-style L2-Regularized parameter estimation, in which only the weights are regularized, with a variety of Python tools such as

scipy.optimize.minimize(lambda p: np.mean((np.dot(dmat, p) - y_data)**2) + lamb*np.dot(p[1:], p[1:]), np.zeros(n_params).T).x

or

scipy.optimize.minimize(lambda p: np.mean((np.dot(dmat, p) - y_data)**2) + 2*lamb*tf.nn.l2_loss(p[1:]).eval(), np.zeros(n_params).T).x

But I see no way to exclude the bias from regularization with TensorFlow's tf.matrix_solve_ls which treats

tf.matrix_solve_ls(dmat, np.array([y_data]).T, l2_regularizer=lamb*len(dmat)).eval()

as equivalent to

scipy.optimize.minimize(lambda p: np.mean((np.dot(dmat, p) - y_data)**2) + lamb*np.dot(p, p), np.zeros(n_params).T).x

How do I exclude bias (or, by construction, the first parameter) from regularization using tf.matrix_solve_ls?

Answer 1

Unfortunately, this is not possible with the current version of matrix_solve_ls. On the other hand, the default (fast) path in matrix_solve_ls was recently changed to simply be a composite Python function:

https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/ops/linalg_ops.py#L293

You can easily make a version of this code where the l2_regularizer is a vector (with a separate value for each parameter). It would probably be a good idea to implement such a generalization in matrix_solve_ls in general. Perhaps you can submit this change as a pull request?