An awkward awker

My "physical" standard form of data handled by Awk:

Tables in space-separated text files

When we use awk, a logical table is usually represented by a text file consists of lines, and one line in the file corresponds to one row of the table. In Unix-like operating systems, lines are separated by the "linefeed" (or "newline") control character (ASCII 0a hexadecimal).

In Awk, the fields of a line that correspond to the columns of the logical table are separated by the "field separator" defined by the value of the built-in variable "FS". If "FS" is not defined, both a space (ASCII 20 hexadecimal) and a tab (ASCII 09 hexadecimal) work as field separators.

Used as field separators, spaces work somewhat differently from other characters: a consecutive series of multiple spaces is equivalent to one space. I find this feature useful, for we can make the position of the field constant by inserting the appropriate number of spaces between columns despite of different lengths of items in the cells. I often want to examine the content of tables visually, especially to find and to identify outlier values. In doing this, the fields of rows belonging to the same column (actually the column plus the field separator) should contain the same number of characters. Then, if viewed on a console (a "terminal") with constant-width fonts, a column appears vertically straightly.

The downside of this choice is that we cannot include a null string as the content of a cell except in the rightmost parts of a row. Also the content should not contain spaces. When I put data into this format, I modify them. I usually replace spaces within strings with underscores ("_"). I replace null values with a string of underscores if it is to be read as a string, and with a certain number that represents null in the particular context (admittedly not universal, but to utilize built-in reading functions of Awk and Fortran) if it is to be read as a number.

This format may be seen as a kind of fixed-column format as well as space-separated format. I prefer this particularly because I also use Fortran as well as Awk. When Fortran reads data by "formatted read" it considers data as fixed-column ones (e.g. "READ(*,'(3F10.0)') var1, var2, var3"). On the other hand, when Fortran reads data by "list-directed read" (e.g. "READ(*,*) var1, var2, var3"), it considers data as space-separated ones. I want to make a data to be commonly read by either way. The way in which Fortran reads character strings by "list-directed read" seems to depend on particular compilers. As I experience, many compilers require that the data to be read should have been quoted (like "string" or 'string'). When I really need to read strings, I use "formatted read". Otherwise I arrange the table so that columns containing numbers appear in the left-hand part, and read only those columns by "list-directed read".

Why not tab-separated?

I admit that tab characters are useful, if it is very sure that field separators are always tabs and that there are no tabs elsewhere. When I handle data by myself, I am not so careful. With ordinary console (terminal) settings, a tab character are shown as a certain number of spaces depending its context. It is not easy to see whether there is actually a tab or a string of spaces. Also, since the standard tab interval is 8 characters and the contents of cells in my tables are sometimes shorter and sometimes longer than that, the tab mechanism does not make the columns shown vertically straightly. In addition, the way that Fortran reads tab characters depend on particular compilers. For all these reasons, I avoid using tab characters.


2004-Feb-21, revision 2005-Feb-05, 2015-06-13
MASUDA Kooiti

[To the index page of "an awkward awker"] [To the entrance page of K. Masuda in English]