Azure DevOps VsTest multi-agent parallel

Question

I use Azure DevOps Server 2020 with self hosted agents and created a CI pipeline which executes all tests in parallel on one agent. The ~5000 tests (no UI tests) need around 7min to complete. That's way to slow for our needs, so to speed things up I added 3 agents, put the VsTest task into another job in the same pipeline with parallel: 4. All 4 agents first download the build artifacts and run a slice of the tests. Unfortunately this actually made it worse. Now the test run needs around 8 min on each agent.

My vstest yaml for 1 agent

- task: VSTest@2
  displayName: 'Run tests'
  inputs:
      testSelector: 'testAssemblies'
      testAssemblyVer2:
            **\*test*.dll
            !**\*TestAdapter.dll
            !**\*TestFramework.dll
            !**\obj\**
      searchFolder: '$(System.ArtifactsDirectory)'
      runInParallel: true
      codeCoverageEnabled: false
      rerunFailedTests: false

My vstest yaml for 4 agents

- task: VSTest@2
  displayName: 'Run tests'
  inputs:
      testSelector: 'testAssemblies'
      testAssemblyVer2:
            **\*test*.dll
            !**\*TestAdapter.dll
            !**\*TestFramework.dll
            !**\obj\**
      searchFolder: '$(System.ArtifactsDirectory)'
      runInParallel: true
      codeCoverageEnabled: false
      distributionBatchType: 'basedOnExecutionTime'
      rerunFailedTests: false

I even tried batching by assembly and based on number of tests + number of agents, test run time still sits at ~8min.

Comparing this to our old UI based CI pipeline, with multi-config and multiplier on a variable with 4 TestCategories, which runs even more tests ~10000 (including the 5000 of the new pipeline) but these are distributed by TestCategory on the same 4 agents (Cat1 on agent1, Cat2 on agent2 and so on), the agents average on ~5min each.

The yaml of the UI based one looks like this:

steps:
- task: VSTest@2
  displayName: 'Run tests'
  inputs:
    searchFolder: '$(Build.BinariesDirectory)'
    testFiltercriteria: 'TestCategory=$(Tests)'
    runInParallel: true
    codeCoverageEnabled: false

I think I have to be missing something obvious.

Thanks in advance!

Edit 1:

I connected to my agents with RDP and inside the task manager there are multiple instances of testhost.x86 running, up to 8 simultaneously but not constantly. If I start my tests locally the 8+ instances of testhost.x86 are up almost all the time and rarely vanish at all. If that's any help.