Toolchain not downloading the tool

Question

Hi I'm trying to set up a toolchain for the Fn project. The approach is to set up a toolchain per binary available in GitHub and then, in theory use it in a rule.

I have a common package which has the available binaries:

default_version = "0.5.44"

os_list = [
    "linux",
    "mac",
    "windows"
]

def get_bin_name(os):
    return "fn_cli_%s_bin" % os

The download part looks like this:

load(":common.bzl", "get_bin_name", "os_list", "default_version")

_url = "https://github.com/fnproject/cli/releases/download/{version}/{file}"
_os_to_file = {
    "linux" : "fn_linux",
    "mac" : "fn_mac",
    "windows" : "fn.exe",
}

def _fn_binary(os):
    name = get_bin_name(os)
    file = _os_to_file.get(os)
    url = _url.format(
        file = file,
        version = default_version
    )
    native.http_file(
        name = name,
        urls = [url],
        executable = True
    )

def fn_binaries():
    """
    Installs the hermetic binary for Fn.
    """
    for os in os_list:
        _fn_binary(os)

Then I set up the toolchain like this:

load(":common.bzl", "get_bin_name", "os_list")

_toolchain_type = "toolchain_type"

FnInfo = provider(
    doc = "Information about the Fn Framework CLI.",
    fields = {
        "bin" : "The Fn Framework binary."
    }
)

def _fn_cli_toolchain(ctx):
  toolchain_info = platform_common.ToolchainInfo(
      fn_info = FnInfo(
          bin = ctx.attr.bin
      )
  )
  return [toolchain_info]

fn_toolchain = rule(
    implementation = _fn_cli_toolchain,
    attrs = {
        "bin" : attr.label(mandatory = True)
    }
)

def _add_toolchain(os):
    toolchain_name = "fn_cli_%s" % os
    native_toolchain_name = "fn_cli_%s_toolchain" % os
    bin_name = get_bin_name(os)
    compatibility = ["@bazel_tools//platforms:%s" % os]

    fn_toolchain(
        name = toolchain_name,
        bin = ":%s" % bin_name,
        visibility = ["//visibility:public"]
    )

    native.toolchain(
        name = native_toolchain_name,
        toolchain = ":%s" % toolchain_name,
        toolchain_type = ":%s" % _toolchain_type,
        target_compatible_with = compatibility
    )

def setup_toolchains():
    """
    Macro te set up the toolchains for the different platforms
    """
    native.toolchain_type(name = _toolchain_type)

    for os in os_list:
      _add_toolchain(os)

def fn_register():
    """
    Registers the Fn toolchains.
    """
    path = "//tools/bazel_rules/fn/internal/cli:fn_cli_%s_toolchain"

    for os in os_list:
      native.register_toolchains(path % os)

In my BUILD file I call setup_toolchains:

load(":toolchain.bzl", "setup_toolchains")
setup_toolchains()

With this set up I have a rule which looks like this:

_toolchain = "//tools/bazel_rules/fn/cli:toolchain_type"

def _fn(ctx):
  print("HEY")
  bin = ctx.toolchains[_toolchain].fn_info.bin
  print(bin)

# TEST RULE
fn = rule(
    implementation = _fn,
    toolchains = [_toolchain]
)

Workpace:

workspace(name = "basicwindow")

load("//tools/bazel_rules/fn:defs.bzl", "fn_binaries", "fn_register")
fn_binaries()
fn_register()

When I query for the different binaries with bazel query //tools/bazel_rules/fn/internal/cli:fn_cli_linux_bin they are there but calling bazel build //... results in an error which complains of:

ERROR: /Users/marcguilera/Code/Marc/basicwindow/tools/bazel_rules/fn/internal/cli/BUILD.bazel:2:1: in bin attribute of fn_toolchain rule //tools/bazel_rules/fn/internal/cli:fn_cli_windows: rule '//tools/bazel_rules/fn/internal/cli:fn_cli_windows_bin' does not exist. Since this rule was created by the macro 'setup_toolchains', the error might have been caused by the macro implementation in /Users/marcguilera/Code/Marc/basicwindow/tools/bazel_rules/fn/internal/cli/toolchain.bzl:35:15
ERROR: Analysis of target '//tools/bazel_rules/fn/internal/cli:fn_cli_windows' failed; build aborted: Analysis of target '//tools/bazel_rules/fn/internal/cli:fn_cli_windows' failed; build aborted
INFO: Elapsed time: 0.079s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded, 0 targets configured)

I tried to follow the toolchain tutorial in the documentation but I can't get it to work. Another interesting thing is that I'm actually using mac so the toolchain compatibility seems to also be wrong.

I'm using this toolchain in a repo so the paths vary but here's a repo containing only the fn stuff for ease of read.

Answer 1

Two things:

One, I suspect this is your actual issue: https://github.com/bazelbuild/bazel/issues/6828 The core of the problem is that, is the toolchain_type target is in an external repository, it always needs to be referred to by the fully-qualified name, never by the locally-qualified name.

The second is a little more fundamental: you have a lot of Starlark macros here that are generating other targets, and it's very hard to read. It would actually be a lot simpler to remove a lot of the macros, such as _fn_binary, fn_binaries, and _add_toolchains. Just have setup_toolchains directly create the needed native.toolchain targets, and have a repository macro that calls http_archive three times to declare the three different sets of binaries. This will make the code much easier to read and thus easier to debug.

For debugging toolchains, I follow two steps: first, I verify that the tool repositories exist and can be accessed directly, and then I check the toolchain registration and resolution.

After going several levels deep, it looks like you're calling http_archive, naming the new repository @linux, and downloading a specific binary file. This isn't how http_archive works: it expects to fetch a zip file (or tar.gz file), extract that, and find a WORKSPACE and at least one BUILD file inside.

My suggestions: simplify your macros, get the external repositories clearly defined, and then explore using toolchain resolution to choose the right one.

I'm happy to help answer further questions as needed.