Sometimes I use Awk awkwardly -- An introduction

Academically speaking, I study the global environment. And technically speaking, my specialty is data processing. I collect observational records from many sources, and try to make synthesis of them. I need computers to do that, but my task cannot be fully automated. I split my task to small pieces, and visually check whether each part of processing by the computer has been successful. Processing observational data makes surprises very often, though they are rarely discoveries of really interesting thing but usually of mistakes of myself or of someone else. My "data processing" includes following tasks, among others:

1. To convert the format of data (without changing the information content). The format of observation records (produced by either the institution which conducts observations or the "data center" which compiles the data) are not always convenient for me to view or to further process. Also, observation records come in different formats from different sources. It is necessary to convert the data into "my" format. Sometimes the content of one file is separated to many files, or vice versa.
2. Quality check. Data records often contain erroneous values due to various causes. There is no automatic way to check them out. The best way is man-machine interaction involving visualization of the data values. Once a likely cause of errors has been found, maybe it is possible to automate checking of errors caused that way. A related task is to find extreme but real values.
3. Visualization. In order to see what is going on in the world as well as to check the quality of data, we need to visualize the data. There are good tools for visualization, such as GMT (Generic Mapping Tools by Paul Wessel and W.H.F. Smith, see http://gmt.soest.hawaii.edu/). We need to extract the part of data which we would like to plot, and convert them into the format which the visualization tools accept.
4. Statistical summary. We often need to compute monthly sums or averages from daily values. Sometimes we also want other statistics such as variances, or frequency of occurrence in various classes of values.
5. Spatial interpolation. The locations where we want to know the value of environmental variables are not always the same as the locations of observations. We need to estimate the values at the target locations from those at the locations of observations.

To do many of these tasks, I find that the programming language Awk is convenient. Awk was designed by three computer scientists Aho, Kernighan and Weinberger and named after their names. It has co-evolved with the Unix operating system, but is available on other operating systems as well.

I find that sometimes Awk is useful but sometimes it is not. I can list three factors.

First, Awk was designed for "filter-type" data processing (in the broader sense of the word, [see another page of mine for examples]). I find that many of my tasks are filter-type, and they can be implemented with Awk more easily than in other languages. On the other hand, some of my tasks are not such filter-type as the design of Awk assumed. For example, the task collecting data from many files into one (but not just concatenating them one file after another) requires opening many input files, and cannot be implemented by assigning the input files as either command-line arguments or standard input to the Awk interpreter [see another page of mine for examples].

For such tasks that are not simple filter-type, sometimes I give up using Awk and write a program in Fortran or C, but sometimes I still use Awk. We can write an Awk program which do everything within the "BEGIN"-section. By this way we can use Awk as an ordinary instruction-based (also called procedure-oriented) programming language. But still the Awk language interpreter requires either an input file or standard input. We can silence it by giving explicit nothing -- "/dev/null" in Unix-like operating systems.

This is obviously an awkward usage of Awk. (Here I use the word "awkward" in the ordinary meaning of the word, not meaning "in the style of the Awk language".) I do not recommend you to use Awk this way. Rather I recommend you to learn another script language whose design is not limited by "filter"-type applications. I guess that currently the best language for such purpose is Ruby (by Y. Matsumoto, see http://www.ruby-lang.org/) . But when I want to read a new series of data I always do not have enough time to master Ruby, and resort to awkward Awk scripts.

Second, Awk was designed for processing text files. Numbers should be expressed in strings of digits. The actual Awk language interpreters can somehow work with binary files (anything except text files), but it is difficult to master the way they work. Instead I handle binary files with Fortran or C. [If you read Japanese, see my short note about data handling in Fortran or C.] Sometimes I convert the data into text files and further process with Awk. But sometimes I find it easier to process them with Fortran or C from the source to the target.

Third, I find that Awk is not very good at handling data with two or more spatial dimensions. It is still practical to handle spatial data with Awk as long as all the files refers to the same set of locations. [If you read Japanese, see my examples.] But it becomes very inconvenient if the task involves spatial interpolation. In such cases I do not use Awk but mainly Fortran.