Processing many files in a directory

Processing all files in a directory at once

We often have many files to be processed by the same program. The command interpreters of Unix-like operating systems are called "shells", and they have features called "globbing" or "wild card". Command interpreters of MS-DOS and Windows also have similar features though less versatile than those of shells. In a Unix-like OS, it would be possible to process all files in a directory by an Awk script, "process.awk" for example, like this:

% awk -f process.awk *

But, by this way, "process.awk" itself is included in the set of files to be processed. Usually it is not desired. To avoid it, "process.awk" should be put out of the directory. For example, let us assume that it is put in a directory called "program" that is parallel to the current directory (the directory that has the same parent directory as the current one). Then, we can do the job like this:

% awk -f ../program/process.awk *

There is no absolute limit of the number of files to be processed at once. But, practically, the length of a list of files accepted by the Awk interpreter and/or the length of a command line accepted by the shell (command interpreter) are limited by the condition given when the interpreters were compiled. The above command line is likely to fail if there are thousands of files in the directory. If the length of the list of files may be too long for the Awk interpreter but not too long for the shell and the "cat" command, the task can (probably) be preformed by:

% cat * | awk -f ../program/process.awk

The functional difference is that in the former way the Awk interpreter recognizes names of and boundaries between the input files, but in the latter way it does not.

Making a disposable shell script

Often it is necessary to handle files one by one (not just contents of all files concatenated as in the previous example). Also, it may be the case that "process.awk" is written to process just one file at one time and we must invoke it one by one, e.g.

% awk -f process.awk tokyo
% awk -f process.awk nagoya
% awk -f process.awk kyoto

If you need to process all files in a directory, we can use the wild-card feature of shells. For example, the following script would work with "tcsh":

%cat doprocess.tcsh
foreach x (*)
  awk -f ../program/process.awk $x
end
%tcsh doprocess.tcsh

But do we really want to process ALL files? The directory may still contain programs even if we have carefully removed "process.awk" itself. It may also contain a "core" file made when a program abnormally ended. If the content of the directory is observational data, it may contain lists of stations as well as files containing observational records. So we had better inspect the list of files in the directory before processing them. The list can be made by this command:

% ls -1 > ls1

Note that the character for the option is the numeral "one", not the lower-case L. Then the list may look like this:

% cat ls1
a.out
convert.awk
core
kyoto
list_of_stations
ls1
nagoya
preprocess.f
tokyo

Then I manually edit "ls1" so that it becomes:

% cat ls1
kyoto
nagoya
tokyo

Note that the original list included "ls1" itself and I removed it as well as other files that should not be processed.

% unalias ls
% ls -1 > ls1

% /bin/ls -1 > ls1

Once it is sure that "ls1" contains just names of files that should be processed,

% awk '{print "awk -f ../program/process.awk " $1 }' ls1 > doprocess.sh
% sh doprocess.sh

% awk '{print "awk -f convert.awk " $1 "> ../output/" $1 }' ls1 > doconvert.sh
% mkdir ../output
% sh doconvert.sh

{
  fname = $1
  while(getline < fname) {
#   ... to process $0 or $1, $2 etc. ...
  }
  close(fname)
}

% awk -f process_all.awk ls1