Awk is a programming language for filter-type operations. A filter-type operation is to read data from the standard input line by line and to write the result to the standard output. In processing numerical data obtained by earth observations, I often need to make many filter-type operations, so I often use Awk. The term "filter" used here has a very broad sense.
People who process time series data often use the term "filter" in a more restricted sense. In it, a time series is considered as a sum of many sinusoidal waves, and a filter may pass waves of some range of periods and it may stop others. I would like to discuss filters in this sense in another occasion.
Filter-type operations can be applied to any types of data,
but it is text files that are effectively handled by Awk.
In particular, I assume that the data are
formed into tables
and stored in space-separated text files.
In this section, it is also assumed that a data file expresses a time series.
As many (though not all) climatological data consist of monthly values of observations,
let us consider that the data has three columns,
1=year, 2=month, and 3=observed values.
As an example, a file "temp_tokyo" contains
monthly surface air temperature in Tokyo
observed by Japan Meteorological Agency.
Also another file "precip_tokyo" contains precipitation (rainfall and snowfall).
The simplest program in Awk just echoes the input. (The result will appear on the screen.)
% awk '{print}' temp_tokyo
We know that the data consists of three columns. So this also just echoes the input, though the format for numerical values may change.
% awk '{print $1, $2, $3}' temp_tokyo
We often need such records (lines) that satisfy a certain condition. For climatological data, we often want to have a time series of a certain month only (for example, April only). In our example the second column contains a number corresponding to the month (1=January, ..., 12=December). The records for April can thus be extracted. In this case, I save the standard output in a file, because I want to use it again.
% awk '$2 == 4 {print}' temp_tokyo > temp_april_tokyo
% awk '$2+0 == 4 {print}' temp_tokyo > temp_april_tokyo
To count the records and to calculate the average of the values are
also filtering operations, though there may be only one line of output
for many lines of input.
In this case, the program is a little too complicated to be included in the
command line.
So I first save the program in a file "ave.awk" and use it.
% cat ave.awk
BEGIN {
sum = 0
count = 0
}
# main
{
sum += $3
count ++
}
END {
if(count > 0) { sum /= count }
print count, sum
}
% awk -f ave.awk temp_april_tokyo > ave_temp_april_tokyo
The program consists of three parts.
The BEGIN part is executed before reading the data.
Actually all variables are initialized as zeroes in Awk, so the statements
written here are redundant.
Nevertheless I write them in order not to forget that they must be
initialized.
The second part is executed every time when a line is read.
There is no mark for such part, and I add a comment line "# main"
in order to make it visibly separated from the BEGIN part.
The END part is executed after reading all the data.
Though it is unlikely to apply the program to empty data intentionally,
it is possible to do so by mistake. So I made the division conditional.
When we look at time series, we often look at deviations of values from their long-term averages. For climate data, "climatological normals" are defined. They are 30-year averages with respect to a certain periods (e.g. 1961-1990). Here, however, I do not follow the practice, but calculate deviation from the average of all available input data. We must know the average value before we write the first output data. It requires scanning the input data twice, either by reading the input file twice or storing the values in memory.
We have already calculated the average.
If we use it, we need just another scan of the input file.
The problem is that the program to be written should have two input files.
Though it is possible to list two files on the command line as the input
and write the program to recognize the boundary, it is a little tricky.
In the following example, the main data file will be read normally,
and the file containing the average value will be read in the BEGIN
section. The name of the file for the average value is given as the
value of a variable from the command line by using "-v"
option of the "awk" command.
% cat dev.awk
BEGIN {
getline < avefile
ave = $2
}
# main
{
print $1, $2, $3-ave
}
% awk -v avefile=ave_temp_april_tokyo -f dev.awk temp_april_tokyo > dev_temp_april_tokyo
Another way is to calculate the average and the deviations by a
single program.
The main section of the program just reads the data,
and it may calculate their sum as in the example "ave.awk" above.
Calculation and output of deviations are done in the END section.
The actual code may depend on the range of possible values of the input
(especially the range of years in the case of climatological data here),
so it is not shown here.
When we graphically look at data, or when we make statistical analysis,
we make various transformation of data.
For example, we often take logarithms of input data
when they contain order-of-magnitude larger and smaller values.
Logarithms are defined for positive values only, however, so we need
to take care when the input data may contain zero and negative values.
The built-in function "log" of Awk returns the "natural" (i.e. base "e")
logarithm of the argument.
The "customary" (i.e. base 10) logarithm of x can be calculated
by "log(x)/log(10)".
As an example, let us compute the customary logarithm of monthly precipitation (in a certain unit as specified by the input),
and we choose the output for zero precipitation to be "-10".
It is reasonable that the output should also be "-10"
if the value is positive but less than 10-10.
(Such values are very unlikely in observational records,
but not rare in data files that have passed through many filter-type applications.)
% cat log10.awk
BEGIN {
log10 = log(10)
eps = 1.0e-10
}
# main
{
if($3+0 < eps) {
v = -10
else {
v = log($3)/log10
}
print $1, $2, v
}
% awk -f log10.awk precip_tokyo > log_precip_tokyo
Other transformations such as cubic root, square root, square, cube, exponential, etc. are also used. They are often employed to make the distribution of data values more symmetric.
Also, we are often interested only in the magnitude of deviations but not in their sign. In such cases we calculate the absolute values or the squares of the deviations. (As far as I know, there is no built-in function for absolute values, but it is easy to calculate them by a few short lines of Awk.)
We sometimes want to eliminate such values that is over (or under) a certain threshold, or replace them with some flag values just to indicate that those are out of range. These kinds of manipulation of data can be written concisely in Awk.
[To the index page of "an awkward awker"] [To the entrance page of K. Masuda in English]