We are trying to have dedicated AWS EC2 instance used for all of our organizations docker builds. We are using build_kit and the goal we are trying to reach here is having a common cache amongst all of us. However we notice that when multiple docker build's are occurring at once there is a significant increase in latency(up to 200%) of the docker builds execution time. We've significantly increased the size of the instance with higher RAM, CPU, and IO Internet speeds. However, much of those resources aren't being used and our profiling results are very similar.
If there are multiple docker build's happening on a machine at once, what is the bottle neck? Is there some queuing happening on the docker daemon? This is assuming resources aren't the issue.
The machine is using linux so should natively be using all the available resources. We are not seeing a limit set anywhere.