[This is not a story of "reading files daily". I am not good at programming real-time or time-scheduled applications.]
We often need to read not just one input file, but many input files. I write about some general tricks in another page.
In meteorology and hydrology
(and probably also in other fields of practical applications),
we often need to read daily observation records.
If one file is created every day, it is likely that the file is named
regularly.
Suppose that the files are named like "YYYYMMDD.txt"
where YYYY is the 4-digit year,
MM is the 2-digit month and DD is the 2-digit day.
A series of files for the year 1998 begins with "19980101.txt"
and ends with "19981231.txt".
This notation is straightforward for Japanese (probably also Chinese and Korean) people.
It may not be so for Western people, but they will probably recognize that
this is convenient for sorting files: both ASCII and numerical sorting
result in the correct order of data.
It is redundant to generate a list of such file names.
So all the data will be read by "getline" function in the
BEGIN section.
(This is obviously an awkward usage of Awk.
See my remarks about such usage.)
My example [readdaily.awk]
reads files named in the form "YYYYMMDD.txt"
in the "../indata" directory,
and reports the number of lines in each file.
It is invoked like this.
awk -v dir=../indata -v year=1998 -f readdaily.awk /dev/null
One problem is that our (Gregorian) calendar is not very regular.
Every month has different number of days.
Numbers of days in a month for the normal (non-leap) year are given
in an array "ndaymo[]".
Decision whether the year is a leap year is based on simple "modulo 4"
calculation.
It happens to be valid for years between 1901 and 2099 and it is enough for
myself, but not so for some of my colleagues who may handle data of the years
1900, 1800, 1700 etc., or 2100, 2200 etc.
Months and days are calculated as 1, 2, 3,..., but
they must be expressed as "01", "02", "03", ... in file names.
To achieve this,
automatic conversion between numbers and character strings
by the Awk language is intentionally used in the function "twochar()".
The loop "while(getline < fname)" is very convenient,
but it has a problem that it waits indefinitely
if the specified file does not exist.
Such occasions that some of data files are missing or corrupt are
not rare at all, and they should not stop the job.
To prevent this trouble,
a variable "ERRNO" built in the GNU Awk intepreter is used.
I am not sure whether other interpreters have the same variable.
The program warns if the specified file does not exist, like this.
error [No such file or directory] reading [../indata/19980102.txt]