How to build large software projects without cloning the entire repository?

Question

Thinking of big companies like Google and Facebook, what methodology is used when these companies create big projects without cloning all the code on each computer of each developer?

Answer 1

There are a variety of approaches that companies, and large software projects in general, take.

First, many large projects use multiple repositories. A developer need not clone all the repositories, only the ones that they're working on. For example, at work, I have about thirty repositories on my machine, only a handful of which I access frequently, and there are probably multiple hundreds maintained by other teams which I will never clone or otherwise touch. These projects build independently as part of a larger system, so there's no need to clone all of the code at once.

Second, many projects expose a library interface, and as such they can be installed using the language's preferred package manager (with an internal package server) without cloning based on the latest release. Cloning the repository is only necessary if a developer needs to make a change to that library.

Third, if people do have large monorepos, individual developers can use techniques like sparse checkout and partial clone to avoid cloning or checking out unneeded data, and CI and CD systems can use shallow clones to build. This dramatically reduces the amount of data that needs to be present on the system and both build times and developer experience. The experience usually is (IMHO) worse than with multiple repositories, but some people prefer monorepos nevertheless.