He-initialization for Caffe?

Question

I know that I can either initialize the weights of a convolutional network in caffe with xavier or gaussian. Sadly I can not find any other initialization methods implemented in caffe. Especially, I would like to use the PReLU initialization as suggested in this paper: https://arxiv.org/pdf/1502.01852.pdf

Is this shomewho implemented in caffe and pycaffe?

Answer 1

The MSRA (for Microsoft Research Asia) filler was implemented in Caffe a few months after the paper was released. You can use it with the msra parameter.

See the documentation for more information.

Answer 2

He et al use an initialization scheme which samples each weight w of a given layer independantly identically distributed (iid):

However, you might know that Var(aX) = a^2 Var(X). Hence:

So you can simply initialize the weights with gaussian and then multiply them with 2 / n_l.