make(1): The Magic touch(1) or Directories As Targets


Here is the problem: you need to re-make a target when anything changes in a given directory: a file is modified, added, or removed. In my case, I had a directory with more than 30,000 files in it (no, I did not invent that), there were several file formats, all of the files were the source of a single binary. Listing all of them as pre-requisites, even programmatically, is, of course, a case of madness (the size of such a rule could exceed half a megabyte). This approach would not work too because files can be added or removed from that directory, thus forcing the user to edit (or re-generate) the make(1) script.

The key to the solution is remembering that, logically, a directory is a file that contains a list of files. When a file is added to a directory or a file is removed from a directory the modification time of that directory is updated—as with a regular file whose contents is modified. There is absolutely no undocumented trickery in the below, the solution relies on this fact alone.

The Simple Case: A Flat Directory

For starters, let us suppose that there is a directory named dir that has nothing but regular files in it. Also, there is the target all that must be re-made when dir changes (‘changes’ in the sense that I used above: a file in dir is modified, a file is added to dir, or a file is removed from dir):

all: dir
	touch all # Simulate building ‘all’

dir: dir/*
	touch dir # The magic touch(1)

It works as follows (I am skipping details that are not important for understanding): all will be re-made only if dir becomes newer than all. There are two cases when this can happen:

The problem is solved. If all depends only on some files in dir, the user can replace dir/* above with an appropriate list of prerequisities:

all: dir
	touch all # Simulate building ‘all’

dir: dir/*.c dir/*.h dir/README
	touch dir # The magic touch(1)

The (Somewhat) Convoluted Case: A Tree

Let us suppose that dir contains an arbitrary directory structure, i.e. there are sub-directories under dir with files in them. The problem remains the same: all must be re-built if a file is altered, added, or removed under dir—at any level of its structure. The solution becomes:

all: dir
	touch all # Simulate building ‘all’

DIR_FILES != find dir | grep -v '^dir$$'
dir: $(DIR_FILES)
        find dir -type d -print0 \
		| xargs -0 touch -t $$(date +%Y%m%d%H%M.%S) # The magic touch(1)

Both GNU Make and BSD make(1) perform the != assignement to DIR_FILES when they read the script by expanding the right-hand value of the statement and passing it to the shell. (In BSD make(1) there exists an optimisation: the semantically identical !!= assignement is performed on the first substitution of the variable, but it is not portable. On OpenBSD, any newlines in the assigned value are also replaced with spaces.)

Using find(1), DIR_FILES is assigned a list of all files without exception (regular files, directories, etc.) found under dir, at any level. Filtering out dir itself from the list using grep(1) is required for avoiding a circular dependency, i.e. the situation when dir depends on itself: GNU Make is very touchy in this regard. make(1) will consider the target dir to be out-of-date if any of these files is newer than the directory dir itself. If that happens, make(1) will execute the recipe for bringing dir up to date by passing the only line of the recipe to the shell using system(3). The shell will

  1. Execute date(1) once to capture the current system time in the format that is accepted by the standard -t option of touch(1).
  2. Form a list of all directories found under dir, including dir itself, using find(1) (note -type d passed to find(1)).
  3. Pass the list of directories and the captured system time to touch(1) using xargs(1). This guarantees that the modification time of all directories under dir, as well as of dir itself, will be changed to exactly the same current value. (This may require an additional note: find(1) lists directories starting from dir and then descends to leaves of the tree. Since it is not specified which time exactly touch(1) uses when it is given more than one file argument, it is possible that an implementation may set a different modification time to some of them (this may happen if touch(1) reads the system clock every time before ‘touching’ a file) thus dir may become older than one of its sub-directories, which is undesirable.)

If all depends only on some files in dir, the user can replace the assignement statement above, for instance:

all: dir
	touch all # Simulate building ‘all’

DIR_FILES != find dir -type d -o -name '*.c' -o -name '*.h' -o -name README \
	| grep -v '^dir$$'
dir: $(DIR_FILES)
        find dir -type d -print0 \
		| xargs -0 touch -t $$(date +%Y%m%d%H%M.%S) # The magic touch(1)

In this case, however, addition of any file to dir or removal of a file from it, at any level, (not necessarily a file that all depends on) will trigger bringing all up to the date.

Caveats and Nitpicking

The examples above work both with GNU Make and with BSD variants of PMake with one important difference: GNU Make insists that the expansion of prerequisites of dir must be non-empty (in plain English: there must exist at least one file that dir truly depends on). With any implementation, dir itself must exist (otherwise GNU Make will fail while a BSD make(1) may create an empty regular file, which is not desirable). It is easy to fulfil both requirements and prevent those ‘corner cases’.

Beware: evaluation of dependencies does take time. In the example that I mentioned in the beginning of this article, where I had more than 30,000 files in a directory, any invocation of make(1) (including those when all targets where up-to-date) took at least 24 seconds to evaluate on an old AMD Athlon 64 X2 Dual Core 3800+.

make(1), as it is currently specified by the The Open Group, neither permits wildcards in names of targets or prerequisites nor provides the != assignement operator, therefore nothing in the above is ‘standard’ in any way whatsoever. At the same time, these features are supported by all implementations of make(1) that are in widespread use on UNIX derivatives today, therefore this solution is de-facto portable.

Vadim Penzin, June 23rd, 2019


I hereby place this article along with the accompanying source code into the public domain.
I publish this information in the hope that it will be useful, but without ANY WARRANTY.
You are responsible for any and all consequences that may arise as the result of using this information.