Makefile.am: How to list all files (of specific extension) in `foo_HEADERS`?

Question

In a Makefile.am, there is the following rule that mentions all *.h files in that specific folder (several hundreds of them):

foo_HEADERS = file1.h file2.h ...

As the project evolves, new header files are being added to that folder, and I am looking for a method that picks up all .h files automatically. What doesn't work is to use Makefile stuff like $(wildcard).

Respective Makefile is one of the AC_CONFIG_FILES in configure.ac so I came up with the following:

In Makefile.am:

foo_HEADERS = @FOO_INCLUDE_HFILES@

EXTRA_HEADERS =

In configure.ac:

Add AC_SUBST for FOO_INCLUDE_HFILES:

# Build a list of all .h files in  $(srcdir)/include/foo  as using
# make variables like $(wildcard) won't pan out as desired.
ac_include_foo_h_files=`ls $srcdir/include/foo/*.h`
ac_include_foo_h_files=`basename -a $ac_include_foo_h_files`
AC_SUBST(FOO_INCLUDE_HFILES, "`echo $ac_include_foo_h_files`")

Question:

My solution works as expected, but I am unsure w.r.t. it's generic enought to work in all OSes supported by autotools.

1. Is there a better, more generic approach, in particular for explicit ls .../*.h and basename -a ... and echo^A)?

2. In case my approach is the way to go, why must EXTRA_HEADERS be set^B) in Makefile.am?

---

^A) The echo is needed to get rid of newlines in ac_include_foo_h_files.

^B) Otherwise I am getting an error.

Answer 1

In a Makefile.am, there is the following rule that mentions all *.h files in that specific folder (several hundreds of them):

foo_HEADERS = file1.h file2.h ...

As the project evolves, new header files are being added to that folder, and I am looking for a method that picks up all .h files automatically.

The canonical and most appropriate way to do this is to list all the files explicitly. There being hundreds of them is unusual and says something (I don't know what) about the project, but does not change the fact. It ought to be the build system that is authoritative for what gets built and (in this case) installed. Automake does not define any other supported way. And it's not that hard.

Respective Makefile.am is one of the AC_CONFIG_FILES in configure.ac

Are you sure? The corresponding Makefile should be in AC_CONFIG_FILES, but not Makefile.am (nor Makefile.in). Supposing that you meant it is the Makefile that is listed in AC_CONFIG_FILES, this in Makefile.am ...

foo_HEADERS = @FOO_INCLUDE_HFILES@

EXTRA_HEADERS =

... is relying on Automake to pass the @FOO_INCLUDE_HFILES@ through to the Makefile.in that it generates, where it will be meaningful to configure. This also assumes that Automake does not rely on any of its own analysis of the variable's value, which I think is true but cannot confirm.

Add AC_SUBST for FOO_INCLUDE_HFILES:

# Build a list of all .h files in  $(srcdir)/include/foo  as using
# make variables like $(wildcard) won't pan out as desired.
ac_include_foo_h_files=`ls $srcdir/include/foo/*.h`
ac_include_foo_h_files=`basename -a $ac_include_foo_h_files`
AC_SUBST(FOO_INCLUDE_HFILES, "`echo $ac_include_foo_h_files`")

If you want to hack on configure.ac then you really ought to get a better grasp on shell programming. Use modern $() instead of `` for command substitution. Do not interpose ls where your glob by itself already provides the file list you want. And expanding a variable as an argument to echo, just to (re-)capture that? You have newlines in the variable's value only because you are unwisely capturing the output of ls in the first place, but even having done that, the echo trick is a weird way to solve the problem.

Also $(srcdir) is a command substitution, which will probably fail. You seem to want a parameter expansion instead. And POSIX basename does not document any option arguments, just filenames.

And do not create or use your own variables with the ac_ name prefix. Autoconf reserves that prefix for its own internal use.

Is there a better, more generic approach, in particular for explicit ls .../*.h and basename -a ... and echo?

If I were writing that (though I wouldn't; see above), it might look more like this:

FOO_INCLUDE_HFILES=
for header in "${srcdir}"/include/foo/*.h; do
  FOO_INCLUDE_HFILES="${FOO_INCLUDE_HFILES} ${header##*/}"
done

# Mark FOO_INCLUDE_HFILES as an output variable
AC_SUBST([FOO_INCLUDE_HFILES])

That assumes only features of the POSIX shell, btw; if I were writing for Bash then I'd probably use an array instead of a loop, and I might use pushd / popd to avoid the need for basename.

In case my approach is the way to go, why must EXTRA_HEADERS be set in Makefile.am?

Your approach is not the way to go. It does not conform to Automake's documented requirements, as already discussed. I can't explain exactly why it makes a difference to define an empty EXTRA_HEADERS variable in this case, but I take it as a result of your misuse of Automake.

Answer 2

If the list of foo/*.h files becomes to large to be obvious for a handwritten list of files after foo_HEADERS =, and since Automake insists on a fixed list of files, I would go the following route to keep the list of files added to foo_HEADERS and the list of files in foo/*.h consistent:

1. Use a Makefile.am include file to list all the files. Let's call it foo-headers.mk.

2. Have Makefile.am define the variable and include the include file:

   foodir = $(includedir)/foo
   foo_HEADERS =
   include foo-headers.mk
   

3. Now foo-headers.mk will consist of exactly one line per header file which all look like

   foo_HEADERS += foo/moo.h
   

   This is a very long list to maintain manually, but we can have the computer help us maintain the foo-headers.mk file by checking and pointing out any inconsistencies.

   foo-headers.mk will be version controlled alongside configure.ac and Makefile.am.

4. To prevent any inconsistencies between the file list $(srcdir)/foo/*.h and the content of foo-headers.mk going unnoticed, we add a rule to Makefile.am which checks for that consistency, prints a useful error message and aborts any build if the two are inconsistent.

   all-local:
       @set -e; \
       LC_ALL="C"; \
       cd "$(srcdir)"; \
       oldmk="$(abs_srcdir)/foo-headers.mk"; \
       newmk="$(abs_builddir)/foo-headers.mk.$$$$"; \
       for hfile in foo/*.h; do \
         printf 'foo_HEADERS += %s\n' "$$hfile"; \
       done | sort > "$(abs_builddir)/header-list.mk.new$$$$"; \
       if cmp "$$oldmk" "$$newmk" > /dev/null 2>&1; then \
         echo "foo-headers.mk is up to date, nothing to do."; \
         rm -f "$$newmk"; \
       else \
         printf "Error: foo-headers.mk and the list of foo/*.h are inconsistent.\n       Update $$oldmk\n       from   $$newmk?\n"; \
         exit 1; \
       fi
   

   This code will run every time you run "make" or "make all" or "make distcheck", so you will never build a release tarball which contains a foo/*.h file list inconsistent with the content of foo-headers.mk.

Now you still need to maintain the list of foo/*.h files inside foo-headers.mk every time the list of foo/*.h files changes, but every time the two diverge, your build will fail, and you have a file which would be the foo-headers.mk for the current list of foo/*.h files ready for use (cp, diff -u, whatever) in case that helps.

I have put a complete minimum working example at https://github.com/ndim/so76847289.

You can improve on the Makefile rule by using less portable commands which produce nicer output like diff -u instead of cmp after checking that diff and diff -u exist and work as expected in configure.ac, but that is not strictly necessary to demonstrate the principle of doing automatic consistency checks, and so I have left that out.

You could also choose to just update the foo-headers.mk file if it is found to not reflect the foo/*.h file list, by replacing exit 1 with mv -f "$$newmk" "$$oldmk". This means that building can update the $srcdir, though, and will require a subsequent commit to version control. You might or might not want a simple build to be able to trigger that.