I’m designing a multi-pass compiler for a language and the AST is completely kept in memory. This means that I’m saving the entire AST in memory, no matter how many files the program is composed of.
I’m thinking about potential BIG programs that are composed by 15k+ files (Google Chrome is the first examples that comes in my mind, which is in C++ and C/C++ compilers have separate compilation so it’s not a problem for such languages).
May a compiler that keeps the entire AST of Google Chrome run out of memory?
Of course it is possible to run out of memory. However, carefully done, an AST does not require much more memory than the source code itself, because common symbols (all types, methods, structures, etc.) can be de-dup'd and pointer-referenced.
The problem tends to be that inefficient (large generic) data structures are used, when custom data structures (AST nodes that act as their own linked lists, for example) would be able to reduce memory utilization by 1-2 orders of magnitude.