Here is the problem: you need to re-make a target when anything changes in a given directory: a file is modified, added, or removed. In my case, I had a directory with more than 30,000 files in it (no, I did not invent that), there were several file formats, all of the files were the source of a single binary. Listing all of them as pre-requisites, even programmatically, is, of course, a case of madness (the size of such a rule could exceed half a megabyte). This approach would not work too because files can be added or removed from that directory, thus forcing the user to edit (or re-generate) the make(1) script.
The key to the solution is remembering that, logically, a directory is a file that contains a list of files. When a file is added to a directory or a file is removed from a directory the modification time of that directory is updated—as with a regular file whose contents is modified. There is absolutely no undocumented trickery in the below, the solution relies on this fact alone.
For starters, let us suppose that there is a directory named dir that has nothing but regular files in it. Also, there is the target all that must be re-made when dir changes (‘changes’ in the sense that I used above: a file in dir is modified, a file is added to dir, or a file is removed from dir):
all: dir touch all # Simulate building ‘all’ dir: dir/* touch dir # The magic touch(1)
It works as follows (I am skipping details that are not important for understanding): all will be re-made only if dir becomes newer than all. There are two cases when this can happen:
The problem is solved. If all depends only on some files in dir, the user can replace dir/* above with an appropriate list of prerequisities:
all: dir touch all # Simulate building ‘all’ dir: dir/*.c dir/*.h dir/README touch dir # The magic touch(1)
Let us suppose that dir contains an arbitrary directory structure, i.e. there are sub-directories under dir with files in them. The problem remains the same: all must be re-built if a file is altered, added, or removed under dir—at any level of its structure. The solution becomes:
all: dir touch all # Simulate building ‘all’ DIR_FILES != find dir | grep -v '^dir$$' dir: $(DIR_FILES) find dir -type d -print0 \ | xargs -0 touch -t $$(date +%Y%m%d%H%M.%S) # The magic touch(1)
Both GNU Make and BSD make(1) perform the != assignement to DIR_FILES when they read the script by expanding the right-hand value of the statement and passing it to the shell. (In BSD make(1) there exists an optimisation: the semantically identical !!= assignement is performed on the first substitution of the variable, but it is not portable. On OpenBSD, any newlines in the assigned value are also replaced with spaces.)
Using find(1), DIR_FILES is assigned a list of all files without exception (regular files, directories, etc.) found under dir, at any level. Filtering out dir itself from the list using grep(1) is required for avoiding a circular dependency, i.e. the situation when dir depends on itself: GNU Make is very touchy in this regard. make(1) will consider the target dir to be out-of-date if any of these files is newer than the directory dir itself. If that happens, make(1) will execute the recipe for bringing dir up to date by passing the only line of the recipe to the shell using system(3). The shell will
If all depends only on some files in dir, the user can replace the assignement statement above, for instance:
all: dir touch all # Simulate building ‘all’ DIR_FILES != find dir -type d -o -name '*.c' -o -name '*.h' -o -name README \ | grep -v '^dir$$' dir: $(DIR_FILES) find dir -type d -print0 \ | xargs -0 touch -t $$(date +%Y%m%d%H%M.%S) # The magic touch(1)
In this case, however, addition of any file to dir or removal of a file from it, at any level, (not necessarily a file that all depends on) will trigger bringing all up to the date.
The examples above work both with GNU Make and with BSD variants of PMake with one important difference: GNU Make insists that the expansion of prerequisites of dir must be non-empty (in plain English: there must exist at least one file that dir truly depends on). With any implementation, dir itself must exist (otherwise GNU Make will fail while a BSD make(1) may create an empty regular file, which is not desirable). It is easy to fulfil both requirements and prevent those ‘corner cases’.
Beware: evaluation of dependencies does take time. In the example that I mentioned in the beginning of this article, where I had more than 30,000 files in a directory, any invocation of make(1) (including those when all targets where up-to-date) took at least 24 seconds to evaluate on an old AMD Athlon 64 X2 Dual Core 3800+.
make(1), as it is currently specified by the The Open Group, neither permits wildcards in names of targets or prerequisites nor provides the != assignement operator, therefore nothing in the above is ‘standard’ in any way whatsoever. At the same time, these features are supported by all implementations of make(1) that are in widespread use on UNIX derivatives today, therefore this solution is de-facto portable.
Vadim Penzin, June 23rd, 2019
I hereby place this article along with the accompanying source code into the public domain.
I publish this information in the hope that it will be useful, but without ANY WARRANTY.
You are responsible for any and all consequences that may arise as the result of using this information.