I've been wondering whether there might be a correlation between the "distance" among neural network weights and their performance.
To elaborate, consider the following scenario:
We have three models, M1, M2, and M3, all with identical structures, each trained on its respective dataset, D1, D2, and D3. Post-training, let's say the "distance" between M1 and M2 is 5, while between M1 and M3 it's 20. In essence, M1 and M2 are closer "spatially".
What I would except is that if we evaluate M1 on D2 and D3, its performance on D2 should be higher since M1 is closer to M2, and M2 was trained on D2. However, some experiments contradict this hypothesis.
I intentionally enclose "distance" in quotes because I'm uncertain about the appropriate metric to employ in this context.
I found some papers on the topic, but they don't seem to meet my needs.
Can anyone help me better understand whether there is a relationship between "distance" and performance?
Thank you very much!