Why does gradient perturbation to input blob in overrided Solver::Callback::after_step() cause the input blob altered

Question

I'm trying to implement the "Adversarial Data Augmentation" algorithm. To do this, I registered a callback class GradPerturb which is publically inherited from caffe::Solver<float>::Callback and overrided the function GradPerturb::before_step() and GradPerturb::after_step(). The function after_step() is as follows:

caffe::Blob<float>* input_blob = solver_.net()->input_blobs()[0];
caffe::Blob<float>* label_blob = solver_.net()->input_blobs()[1];
boost::shared_ptr<caffe::Blob<float>> first_feat = solver_.net()->blob_by_name("conv1_n");
boost::shared_ptr<caffe::Layer<float>> first_layer = solver_.net()->layer_by_name("conv1_n");
std::vector<bool> propagate_down(1, true);
const int input_data_count = input_blob->count();
std::vector<caffe::Blob<float>*> top_vecs(1, first_feat.get());
std::vector<caffe::Blob<float>*> bottom_vecs(1, input_blob);
for (int i = 0; i < perturb_iters_; ++i)
{
    caffe::caffe_gpu_set(first_feat->count(), 1.f, first_feat->mutable_gpu_diff());
    float loss = solver_.net()->ForwardBackward();
    first_layer->Backward(top_vecs, propagate_down, bottom_vecs);
    caffe::caffe_gpu_axpy(input_data_count, perturb_scale_, input_blob->gpu_diff(), input_blob->mutable_gpu_data());
    solver_.net()->ClearParamDiffs();
}

But after calling caffe_gpu_axpy, data in input_blob changed. I mean it's not the same batch of images as before.

I have tried to print the pointer input_blob, it never changed. Also, I tried to call the counterpart cpu version caffe_axpy and even calculate in a for loop like this:

for (int j = 0; j < input_data_count; ++j)
{
    input_blob->mutable_cpu_data()[j] = input_blob->cpu_diff()[j] * perturb_scale_ + input_blob->cpu_data()[j];
}

More surprisingly, data in input_blob has not changed, but label in label_blob changed after first several iterations. In fact, when gradient perturbation was moved to before_step(), this phenomenan didn't occur. So I guess the mechanism of Solver::Callback or multithreading in caffe caused this, but I don't understand the actual mechanism.