Compile time grows after every file change

Question

I've changed my sass workflow recently to more modular way - a lot of partials in directories, instead of single stylesheet - and i've noticed, that time of compilation growed really bad. It continously grows when I apply some changes in my partials (it can grows even to 10 sec for single compilation!):

My main stylesheet contains only imports of partials e.g.:

@import 'components/menu';

and so on...

This is my gulpfile:

gulp.task('sass', function() {
  return gulp.src('scss/**/*.scss') 
    .pipe(sourcemaps.init())
    .pipe(sass())
    .pipe(autoprefixer('> 0.5%'))
    .pipe(sourcemaps.write('.'))
    .pipe(gulp.dest('css'))
});
gulp.task('watch:sass', function () {
    gulp.watch('scss/**/*.scss', gulp.series('sass'));
});
gulp.task('browser-sync', function() {
    browserSync.init({
        proxy: site,
        port: 8080,
        open: true,
        notify: false,
        files: ['./css/*.css', './js/*.js', '*.php', './include/**/*.php']
    });
});
gulp.task('default', gulp.parallel('watch:sass', 'browser-sync'));

Interesting thing: when i've moved all partials from internal directories to main /scss directory, problem doesn't appear. I really want to preserve that modular directory tree and not go for a flat structure.