Loading performance

Question

For (incremental) loading performance I want to split a huge (believe me), generated BUILD.bazel into smaller .bzl files.

In each .bzl I then plan to have a Macro foo, which contains the actual rule calls:

def foo():
    foorule("a")
    foorule("b")
    ...

In the BUILD.bazel I then would have (a lot of) loads like:

load("foo.bzl", foo_0 = "foo")
load("other/foo.bzl", foo_1 = "foo")
...

and then trigger the rules in BUILD.bazel via:

foo_0()
foo_1()

Is this supposed to be faster than evaluating all rules inside of a symbol in .bzl?

foo = [
     foorule("a"),
     foorule("b"),
]

Or is there even a better way to load all the info in parallel?

Answer 1

If I am not mistaken this should be slightly faster but the best approach would be to split the rules into several packages so the package loading itself can be parallelized.

Here just the Skylark loading would be parallelized.

Answer 2

Each .bzl is file can be loaded and evaluated in parallel. Evaluating a .bzl file consists in evaluating all top-level statements. If you have CPU-intensive computations, you can do them in the top-level and store the result in a global value.

However, if .bzl files simply define some functions, there's almost nothing to parallelize (just loading the file from the disk and parsing it). I would expect no visible speedup in your case. Macros will not be evaluated in parallel.

Do you have more data? How many rules are there in your file (run bazel query :all | wc -l)? How long does it take to load the file? Are you sure the bottleneck is the loading phase?

If your BUILD file is huge, I would encourage you to split it. Create multiple BUILD files instead, if it's possible. You'll get more parallelism.