std::unordered_map vs std::map have different performances depending on the compiler

Question

I have integer keys which have to be associated to std::vector<T>, with T an iterator-like object (you can safely assume each element of the vector is a pointer).

The obvious candidate from the STL is a std::unordered_map<int,vector<T>>, mainly because I should not get hash collisions as my keys are just integers. In my particular case each key maps to a vector with potentially "many" elements which are pushed back. To give an example with a std::unordered_map<int,vector<double>>

constexpr size_t map_size = 360;
constexpr size_t vec_size = 1000;

std::unordered_map<int, std::vector<double>> grid_unmap_int;
for (size_t i = 0; i < map_size; ++i)
  {
    for (size_t j = 0; j < vec_size; ++j)
      {
        grid_unmap_int[i].push_back(j);
      }
  }

If I compare the performance of the insertion using the code above with a std::map<int,std::vector<double>>, I get quite different results which seems to depend on the chosen compiler. In particular, the result for the std::unordered_map using clang turns out to be much faster. Why does the unordered_map behaves so poorly with GCC ? I am afraid it's related to the fact my values are vector with no known size...

Here's a link to quick_bench: https://quick-bench.com/q/fgQ9XuZ9tmeXcKYEGuJOW165FtE

Here are the associated results (-std=c++17 and -O3) Output with GCC 9.4:

Output with CLANG 14.0: