building the hex registry file offline

Question

Adapting Elixir and all the tools in its ecosystem to work with a different build system.

In this system, the packages and their dependencies are managed separately and Hex is made to work in offline mode. (grab the tarballs)

It's working with one caveat: every time I import a new package I need to also import the latest registry file from hexpm and I cannot use packages that are not published through hex unless they are at the top level in the deps chain.

Given a bunch of tarballs (and assuming that the dependencies between them are satisfied, how would one go about building a hex registry file that works with them.

What I have so far:

• looked at the registry file format and seen it's an ets file. Can load and inspect it; now I need to generated
• looked at how the website builds the registry file, but it's super complicated for my needs
• I struggle a bit to understand why there is a need for a registry file (and if there is, why can't each package contain the needed info in the metadata, making the need for a central registry obsolete)

Anyway, if anyone played with Hex and can provide some guidance on how to do this I would appreciate it.

Answer 1

It's a bit hard to give good information and advice without more information on your use case. Could you elaborate a bit more on what you are doing and why you are doing it? I will try my best to answer the question though.

Here is the specification for the registry format: https://github.com/hexpm/specifications/blob/master/registry.md.

The format is fairly simple and it would not require too much code to build the ETS file yourself.

I struggle a bit to understand why there is a need for a registry file (and if there is, why can't each package contain the needed info in the metadata, making the need for a central registry obsolete)

The registry is needed for the dependency resolution in the Hex client. It is possible for the resolver to try many different versions of packages, if the client had to fetch each package version to see if it resolved a lot of useless HTTP requests would have to be made. The registry is there as an optimization so we only have to fetch a single file to do the full resolution.

I think what you may want is to depend on local package tarballs directly since you imply you do the dependency resolution yourself. Is that correct? I have opened an issue on the client to support this: https://github.com/hexpm/hex/issues/261

Answer 2

For future generations that ended up here, here is a working registry builder:

defp string_files(files) do
  Enum.into(files, %{}, fn {name, binary} ->
    {List.to_string(name), binary}
  end)
end

defp decode(string) when is_binary(string) do
  string = String.to_char_list(string)
  case :safe_erl_term.string(string) do
    {:ok, tokens, _line} ->
      try do
        terms = :safe_erl_term.terms(tokens)
        result = Enum.into(terms, %{})
        {:ok, result}
      rescue
        FunctionClauseError ->
          {:error, "invalid terms"}
        ArgumentError ->
          {:error, "not in key-value format"}
      end

    {:error, reason} ->
      {:error, inspect reason}
  end
end

def build_registry(hex_home) do
  # find the tars
  tars = Path.wildcard(Path.join(hex_home,"packages/*.tar"))

  # initialize the ets table used to build the registry
  :ets.new(:myr, [:named_table])
  :ets.insert(:myr, {:"$$version$$", 4})

  # go through the tars, extract the info needed and populate
  # the registry
  Enum.each(tars, fn filename ->
      {:ok, files} = :erl_tar.extract(String.to_char_list(filename), [:memory])
      files = string_files(files)
      {:ok, metadata} = decode(files["metadata.config"])
      name = metadata["app"]
      version = metadata["version"]
      build_tools = metadata["build_tools"]
      checksum = files["CHECKSUM"]
      deps = []
      if metadata["requirements"], do: deps = metadata["requirements"]
      reg_deps = Enum.map(deps, fn
          {name, depa} ->
              depa = Enum.into(depa, %{})
              [name, depa["requirement"], depa["optional"], depa["app"]]
          depa ->
              depa = Enum.into(depa, %{})
              [depa["name"], depa["requirement"], depa["optional"], depa["app"]]
      end)
      IO.puts "adding dependency"
      IO.inspect {name, [[version]]}
      IO.inspect {{name, version}, [reg_deps, checksum, build_tools]}
      :ets.insert(:myr, {name, [[version]]})
      :ets.insert(:myr, {{name, version}, [reg_deps, checksum, build_tools]})
  end)

  # persist the registry to disk and remove the table
  registry_file = Path.join(hex_home, "registry.ets")
  IO.puts "Writing registry to: #{registry_file}"
  :ets.tab2file(:myr, String.to_char_list(registry_file))
  :ets.delete(:myr)
  registry_file_gzip = registry_file <> ".gz"
  IO.puts "Gzipping registry to: #{registry_file_gzip}"
  gzipped_content = File.read!(registry_file) |> :zlib.gzip
  File.write!(registry_file_gzip, gzipped_content)
end

For more context:

• the registry is an ETS table. See: http://erlang.org/doc/man/ets.html
• the registry format is outlined here: https://github.com/hexpm/specifications/blob/master/registry.md
• private fn in this module are inspired by: https://github.com/hexpm/hex_web/blob/master/scripts/rebuild_tarballs.exs
• safe_erl_term.xrl (needs to be under src/) is borrowed from: https://raw.githubusercontent.com/hexpm/hex_web/master/src/safe_erl_term.xrl
• xrl is an input to leex. see here (takes spec, generates erlang code): http://erlang.org/doc/man/leex.html