• John W. Tukey, 1977: Exploratory Data Analysis. Reading MA USA: Addison-Wesley, 506 pp. ISBN 0-201-07616-0.

A5判よりすこしおおきめのハードカバー。表紙はオレンジ色というか、朱色というか。背表紙には「Tukey E D A」とだけ書いてある。わたしが持っている本は、1980年代に買ったものだ。1980年代の前半の大学院生のころに、「探索型データ解析」 (exploratory data analysis, EDA) というかんがえかたを、さまざまな情報源から知って、よいことだとおもった。1985年に大学教員 (助手) になって、データ解析をおしえることになった機会に、EDA の原典といわれる本を読んでおこうとおもったのだった。

読んでみると、かなりくせの強い本で、あまり読みすすめなかった。また、手法はわかったが、そのまままねをする気になれなかったところもある。1980年代、いくらかは計算機がつかえるようになっていたから、著者が手がきでやってきた方法はめんどうに感じられた。ところが、当時の計算機入出力技術ではそれよりうまいことができなかった。著者のしごとなかまがいた Bell 研究所で「S」というソフトウェアがつくられていて、そこでは本書で紹介された方法がコンピュータグラフィックスで実現されていた。しかし、Sは当時わたしがつかっていた大型 (メインフレーム) 計算機でも MS-DOS パソコンでもうごかなかった。Unix ではうごいたが、有料のソフトウェアで、個別の研究室で買うにはちょっと高かった。わたしは、散布図などは、コンピュータグラフィックスの部品からくみたててつくるようになったが、もうすこし複雑な技法については、20-00年代、Sとほぼ互換のフリーソフトウェア「R」が普及してきて、やっとためせるようになった。

そういうわけで、この本自体は「わたしをつくった本」ではない。しかし、さまざまな情報源をとおしてみにつけた、Tukey さんに由来する EDA のかんがえかたは、わたしにしみついていて、自分がもともともっていた発想と区別がつかなくなってしまっている。

1970年代の理科系の図書や論文にグラフをふくめるときには、たいてい、てまをかけて (ときには製図の専門家に依頼して) 製図して版下をつくった。(わたしはやらなかったが)「からすぐち」をつかって墨入れしたり、のりつきのシールのインスタントレタリングをつかったりすることがあった。ところが本書のグラフは、ていねいにかいてはあるが、グラフの曲線も軸もラベルの文字も、あきらかに手書きで、しろうとっぽい。本書は、いまの1冊本になるまえに、講習会テキストとしてつかわれた3冊本があったそうで、当時のその種類のものだったら、本文はタイプ打ち、図は手がきが当然だっただろう。本書は出版社からでる本になってもそのスタイルをひきついだのだ。

また、ふつうならば図は「figure」、表は「table」としてそれぞれ番号をつけるところだが、本書のばあいは区別なしに exhibit としている。本書のばあい、文字をならべて図のようなものをつくることもあるので、図と表の区別をなくしたのはもっともな気がする。しかしそれだけでなく、データをそのままプリントしたところ、計算の中間結果をプリントしたところ、練習問題など、およそ本文以外のものを exhibit にふくめている。

章の番号は算用数字だが、章の下の節はアルファベット大文字で、第2章の最初の節ならば「2A」のようになっている。本全体の目次はこの節のレベルまでで、章ごとにまた目次があって、もう1段こまかい小節の名 (記号はない) と、exhibit (番号だけだが) がわかる。(この読書メモの下につけた目次は、全体の目次のなかに小節まで書きこみ、exhibitは省略した。)

本書に代表されるEDAの態度をわたしなりに整理するとつぎのようになる。

• はじめから統計量を計算しようとおもわないで、データ自体をみる。とくに度数分布をみるヒストグラムなど、2変量ならばそのくみあわせの度数分布をみる散布図などの方法を重視する。
• データの集合には、おおきくはずれた値がふくまれていることがあり、それをそのままつかって平均値などの統計量をもとめても代表性がとぼしい。はずれ値は、まちがいであるかもしれないし、実際に異常なことがおきているのかもしれないが、それとしてとりあげるべきだ。他方、データ全体の代表値やばらつきの指標には、平均値や標準偏差よりも はずれ値につよい、中央値や四分位幅などの順位統計量をつかう。
• 度数分布が非対称なときは、変数変換して対称にちかづけたほうが、データ集合全体をよくみることができる。2つの変量の関係がみられるが直線にのらないときは、直線にちかくなるような変数変換をさぐる。このとき、順位統計量は、たとえば中央値の対数は対数の中央値でもあるので、つごうがよい。
• データがだいたい直線にのるならば、直線からの残差に注目し、拡大してみる。そうするとデータからさらに情報をよみとれることがある。

章の題名を書きだすと、つぎのようになっている。
1. Scratching down numbers (stem-and-leaf)
2. Schematic summaries (pictures and numbers)
3. Easy re-expression
4. Effective comparison (including well-chosen expression)
5. Plots of relationship
6. Straightening out plots (using three points)
7. Smoothing sequences
7+ Optional sections for Chapter 7
8. Parallel and wandering schematic plots
9. Delineations of batches of points
10. Using two-way analyses
11. Making two-way analyses
11+ Optional sections for Chapters 10 and 11
12. Advanced fits
13. Three-way fits
14. Looking in two or more ways at batches of points
15. Counted fractions
16. Better smoothing
17. Counts in BIN after BIN
18. Product-ratio plots
19. Shapes of distribution
20. Mathematical distributions
21. Postscript

第1章では、「幹葉表示」(stem-and-leaf dislay) が奨励されている。これは要するにヒストグラムなのだが、単なる棒にしないでそこに数字を書くことによって、ヒストグラムの区間よりも一段こまかいデータの値が見られるようにしたものだ。これは1980年代、とくに Tukey の EDA を話題にしないパソコンの応用の本でもとりあげられ、BASIC や Pascal によるプログラム例を見たおぼえがある。

第2章は、データ集合の情報を要約する方法で、おもに順位統計量をつかう。中央値、四分位値、最大・最小値の5つの値による要約がつかわれる。(ただしTukeyは四分位値のことを hinge といい、こまかい定義は他の教科書の四分位値とおなじでないらしい。) 図としては、上下の四分位値にわたる箱をかき、その上下に棒をのばす「箱ひげ図」 (box-and-whisker plot) が紹介される。本書で「schematic plot」というのは、Tukey の標準にしたがった箱ひげ図である。人によっては「ひげ」を最大値・最小値までのばすのだが、Tukey の箱ひげ図はそうでなく、四分位幅の1.5倍までひげをのばし、その外側にくる点は個別にかく。Tukey先生の経験のうちでは 四分位幅の1.5倍よりも外側にくる点は「はずれ値」とみなすのがふさわしいことがおおかったのだろうが、じゅうぶんな説明はない。【わたしにはこれは、「頭ごなし」と感じられ、まねをする気がおきず、理屈があきらかな最大・最小までひげをのばす箱ひげ図をそのようにことわってつかっていた。いまでは R の箱ひげ図を おそらくTukey流のデフォールト値のまま つかってしまうこともあるが。】
第4章、第8章などで、複数のデータセットを比較するときや、度数分布の時間変化をおいかけるとき、(ヒストグラムをならべたのではかさばって不便なので) 箱ひげ図をならべてみることが奨励されている。

第3章は、変数変換である。ヒストグラムが対称でないとき、平方根、逆数、対数などに変換してやると対称にちかづけられることがある。このような一連の変換を、入力値が1 のときの出力値が 0、傾きが1になるように整理してかさねるとわかりやすい (第3章の exhibit 18, 19)。x と -1/x のあいだに log x がくる。さらに、x と log x のあいだに x の平方根がくる。
第6章で、この一連の変数変換が、2変量の関係を直線にちかづけるために選択されてつかわれる。

第5章では、時系列データをグラフにして情報をよみとる。点がほぼ直線にのっているのならば、その直線からの残差をとったほうがよりよく情報をとりだせる。

(わたしが議論をおいかけたのはこのあたりまでで、あとはところどころ拾い読みした。)

折りかえしのあとに目次をつける。

===== 目次 =====
Frontpapers
– 1. Break table for two-decimal logs (chap. 3 exhibit 2)
– 2. Break table for (square) roots (chap. 3 exhibit 7)
– 3. Main break table — digits of negative reciprocals (chap. 3 exhibit 8)
Preface
– – Examples, NOT case histories
– – Confirmation
– – Exploration AND confirmation
– – Relation to the preliminary edition
– – About the problems
– – Precision
– – Acknowledgments
To the Student or Teacher
Contents
1. Scratching down numbers (stem-and-leaf)
– Comments about the index page
– 1A Quantitative detective work
– – review questions
– 1B Practical arithmetic
– – precision of arithmetic
– – rounding
– – cutting
– – units and decimal points
– – h for “and a half”
– – “*” for place filler
– – “#” for count
– – review questions
– 1C Scratching down numbers
– – batch
– – bold figures
– – review questions
– 1D Doing better with stem-and-leaf
– – stem-and-leaf displays
– – checking
– – review questions
– 1E Using the right number of stems
– – stretched stem-and-leaf
– – squeezed stem-and-leaf
– – using mixed leaves
– – review questions
– 1F How to count by tallying
– – used cars again
– – review questions
– 1G What dows it mean to “feel what the data are like” ?
– – review questions
– 1H How far have we come?
– – what have we learned to do?
– – where do we stand?
– 1K How to use stem-and-leaf to pick up additional information (optional technique)
– – pairs of numbers
– 1P Additional problems
2. Schematic summaries (pictures and numbers)
– General comment
– – review quesions
– 2A Extremes and median
– – extremes
– – median
– – counting-in
– – rank
– – rank down
– – rank up
– – depth
– – interpolated ranks
– – review questions
– 2B Hinges and 5-number summaries
– – hinges
– – 5-number summary
– – letter-value display
– – examples
– – further examples
– – review questions
– 2C Box-and-whisker plots
– – rules
– – basic ideas of plotting
– – tracing paper
– – scale values
– – plotting without graph paper
– – review questions
– 2D Fences, and outside values
– – H-spread
– – step
– – inner fences
– – outer fences
– – adjacent values
– – outside values
– – far out values
– – fenced letter display
– – ranges
– – range
– – trimeans
– – trimean
– – review questions
– 2E Schematic plots
– – rules for same
– – review questions
– 2F Pros and cons: the Rayleigh example
– – a better use
– – aim reiterated
– – choice of plots
– – review questions
– 2G Eights, sixteenths, etc.
– – eighths
– – sixteenths
– – E, D, C, etc.
– – 7-number summaries
– – 9-number summaries
– – letter values
– – review questions
– 2H How far have we come?
– – what have we learned to do?
– – where do we stand?
3. Easy re-expression
– – review questions
– 3A Logarithms = logs
– – review questions
– 3B Quick logs
– – beask tables
– – review questions
– 3C Comparisons of two batches
– – five-figure logs may not be enough
– – review questions
– 3D Quick roots and quick reciprocals
– – quick roots
– – quick reciprocals
– – use negative reciprocals
– – a volcano example
– – looking easier
– – reciprocal times
– – review questions
– 3E Looking quickly
– – combining evidence across batches
– – review questions
– 3F Counted data
– – review questions
– 3G Relation among powers and logs (optional)
– – trivial re-expressions
– – quick reciprocal roots
– – quick squares
– – review questions
– 3H How far have we come?
– 3K How to think about logs (background)
– 3P Additional problems
4. Effective comparison (including well-chosen expression)
– – review questions
– 4A Alternative forms of display of summaries
– – review questions
– 4B Comparing several batches (continued)
– – review questions
– 4C A more extensive example
– – a condensed approach
– – review questions
– 4D The meaning of comparison
– – review questions
– 4E Adjustments, rough and exact
– – rough adjustments
– – exact adjustment
– – review questions
– 4F Residuals
– – review questions
– 4H How far have we come?
– 4P Additional problems
5. Plots of relationship
– – a residual
– – a response
– – a factor
– – a circumstance
– – (factor, response)
– – review questoins
– 5A How to plot y against x
– – choice of ruling
– – choice of scale units
– – consequences of our purpose
– – kinds of grids
– – shape of plot
– – ticks and numbers along axes
– – review questions
– 5B Looking at subtraction
– – untilting
– – review questions
– 5C Subtracting straight lines
– – finding the line
– – an example
– – subtracting idfferent straight lines
– – sum of the two lines
– – back to the example
– – supplementary line
– – review questions
– 5D Plotting the population of the U.S.A.
– – the later decades
– – coming to details
– – the earlier details
– – review questions
– 5E Plotting the ratio of births to deaths
– – another try
– – going to the map
– – review questions
– 5F Untilting defines “tilt”
– – untilted
– – tilt
– – review questions
– 5H How far have we come?
– 5P Additional problems
6. Straightening out plots (using three points)
– – review questions
– 6A Looking at three points
– – review questions
– 6B Re-expressing y alone
– – U.S. population again
– – fitting lines to three points
– – review questions
– 6C Re-expressing x alone
– – back to the population of the U.S.
– – a caveat
– – review questions
– 6D A braking example
– – using our knowledge
– – review questions
– 6E The vapor pressure of H₂O
– – review questions
– 6F Re-expressing the second variable
– – another try
– – review questions
– 6G Wise change of origin as a preliminary
– – a radioactive decay example
– – review questions
– 6H How far have we come?
– 6P Additional problems
7. Smoothing sequences
– – rainfall, wheat, and gold
– – smoothing
– – smoothing gold, wheat, and rainfall
– – what approach?
– – sequence
– – review quesitons
– 7A Medians of 3
– – running medians
– – repeated medians
– – notation
– – a comment
– – bituminous coal; suspended payments
– – review questions
– 7B Eye resmoothing
– – judgment = choices
– – review questions
– 7C Looking ahead
– – a comparison of four pictures
– – blurring, advance notice
– – agenda
– – review questions
– 7D Copying-on — and more, usually
– – end-value smoothing
– – for those who prefer verbal arithmetic
– – 3R or 3R’
– – a comment
– – review questions
– 7E Blurring the smooth — and setting the fences
– – modified hinges, medians, sizes, etc.
– – exact zero
– – *-letter values
– – proper laziness
– – outside values
– – smooth supported by the rough
– – review questions
– 7F Splitting peaks and valleys
– – smoothing short subsequences
– – going farther (optional)
– – a comment
– – review questions
– 7G Hanning
– – skip means
– – comments
– – going a little farther
– – going a lot farther
– – what of the rough?
– – review quesions
– 7H How far have we come?
7+ Optional sections for Chapter 7
– 7I Breaking a smooth
– – presidential New Hampshire
– – back to suspended deposits
– – review questions
– 7J Choice of expression
– – first 3R
– – second, end values
– – third, residuals as a basis of choice
– – comparing expressions
– – fourth, picturing the smooth
– – counted data
– – suspended bank counts
– – review questions
– 7K A two-section example
– – review questions
– 7M How much more may we have learned?
8. Parallel and wandering schematic plots
– – review questions
– 8A Parallel schematic plots
– – letter cuts
– – review questions
– 8B Smoothing the cross-medians
– – broken median
– – cross-medians
– – middle trace
– – median trace
– – review questions
– 8C Smoothing broken hinges
– – cross-hinges
– – review questions
– 8D Dealing with the two questions
– – cross hinges
– – smoothing the separations
– – leter-value differences
– – hinge traces
– – review questions
– 8E Wandering schematic plots
– – adjacent polygon
– – wandering schematic plot
– – interpretation
– – review questions
– 8F A more demanding example: Governor’s salary and bank deposits
– – first analysis
– – untitled analysis
– – combined analysis
– – review questions
– 8G Further questions/analysis in the example
– – time changes
– – vertical width
– – slopes and levels
– – consolidated summary
– – moral
– – review questions
– 8H How far have we come?
– 8I The need to smooth both coordinates (optional)
– – review questions
9. Delineations of batches of points
– – review questions
– 9A E-traces and D-traces
– – review questions
– 9B Simple delineation – -Twin Rivers again
– – delineation
– – an engineering example
– – review questions
– 9C Reduced and schematic delineations
– – reduced delination
– – schematic (x, y)-plots
– – two-way schematic plot
– – review questions
– 9D What our schematic plots and delineations have missed
– – patterns of salaries
– – review questions
– 9E Three variables at once — or more
– – handling more variables
– – index cards
– – how many data sets?
– 9H How far have we come?
10. Using two-way analyses
– – review questions
– 10A Two-way residuals: row-PLUS-column analysis
– – single and double lines
– – double lines between things that add up
– – effects, common, eff, all
– – row-PLUS-column
– – a comment
– – review questions
– 10B The row-PLUS-column fit
– – have we learned more?
– – AN analysis — not THE analysis
– – review questions
– 10C Some points of technique
– – use of bolding
– – zeros and decimal points
– – review questions
– 10D Row-TIMES-column analysis
– – row-TIMES-column
– – review questions
– 10E Looking at row-PLUS-column fits and their residuals
– – row effect PLUS column fit
– – the two-way polt
– – a two-way plot
– – TIMES fits
– – looking at redisuals
– – two-way plot of residuals
– – other postponements
– – review questions
– 10F Fitting one more constant
– – comparison values
– – diagnostic plot
– – row-PLUS-col-PLUS-one fit
– – review questions
– 10G Converting Plus to Times: re-expression
– – consumption again
– – there could be others
– – review questions
– 10H How far have we come?
11. Making two-way analyses
– – review questions
– 11A Taking medians out
– – controlling errors
– – subtraction
– – back to the example
– – a rule of thumb
– – four steps
– – the fit
– – review questions
– 11B Alternative organizations of the arithmetic
– – another pattern
– – review questions
– 11C Making the core of a two-way plot
– – review questions
– 11D Going on with the residuals
– – the residuals
– – review questions
– 11E Coding residuals: condensing fits and residuals
– – condensing
– – review questions
– 11F We can combine!
– – presidential Connecticut
– – a further opportunity
– – review questions
– 11G Guidance for expression
– – amounts and counts
– – diagnostic plot
– – balances
– – counted fractions
– – a helpful thought
– – review questions
– 11H How far have we come?
11+ Optional sections for Chapters 10 and 11
– 11I exploring beyond PLUS-one (extends Chapter 10)
– – review questions
– 11J Taking out any summary
– – repeated midextreme removal
– – the general case
– – repeated mean removal
– – review questions
– 11K An example of re-expression — city killings
– – review questions
– 11L An unusual fit
– – coastwise distances
– – review questions
– 11M How much more may we have learned?
12. Advanced fits
– – review questions
– 12A Plus-one fits
– – row-TIMES-column-PLUS-one-fit
– – PLUS-one fit
– – the special response
– – review questions
– 12B Pictures for “-Plus-one” fits
– – another example
– – PLUS-one plot
– – -PLUS-one plot of residuals
– – review questions
– 12C Making those pictures
– – review questions
– 12D Sometimes we can have parallel-line plots, still
– – but not always
– – review questions
– 12E More extended fits
– – telephones by continent
– – extended row-PLUS-column fits
– – interpretation of the example
– – an interpretation
– – review questions
– 12F Simplification is sometimes possible
– – other possibilities
– – the moral
– – review questions
– 12H How far have we come?
13. Three-way fits
– – review questions
– 13A Three- and more-way analyses: Arrangement and tagging
– – a psychological experiment
– – a tag example
– – review questions
– 13B An analysis of the psychological example
– – review questions
– 13C Making three-way analyses
– – review questions
– 13D Three-way re-expression
– – review questions
– 13E More about the example
– – re-formulated responses
– – log (time of detection)
– – review questions
– 13H How far have we come?
14. Looking in two or more ways at batches of points
– – review questions
– 14A Coordinates and level traces
– – level lines
– – level curves
– – trace
– – level traces
– – review questions
– 14B Different middle traces for the same slices
– – changing the traced coordinate
– – review questions
– 14C An explanation
– – curved traces
– – review questions
– 14D Changing the slicing coordinate
– – review questions
– 14E What matters?
– – some morals
– – review questions
– 14F Rematching and strength of ralationship
– – dependency ratios
– – dependency ratio
– – dependency trace
– – reconstituted traces
– – a comment
– – actual middle
– – review questions
– 14H How far have we come?
– 14I The ubiquity of medians (optional section)
– – odd count
– – even count
– – review questions
15. Counted fractions
– – review questions
– 15A Started counts and counted fractions
– – split count
– – ss-count
– – ss-fractions
– – review questions
– 15B Three matched scales for counted fractions
– – review questions
– 15C Quicker calculation
– – review questions
– 15D Examples where careful expression clealy pays off
– – a 2 x 2 opinion example
– – two-way tables of counts
– – review questions
– 15E Double folding — the 2 x 2 case
– – a double flog
– – fflog
– – review questions
– 15F Double folding — larger cases
– – review questions
– 15G Easy froots and flogs with a slide rule (optional)
– – review questions
– 15H How far have we come?
16. Better smoothing
– – review questions
– 16A Reroughing
– – reroughing
– – twicing
– – review questions
– 16B Some examples
– – comparison of smooths
– – review questions
– 16C If we want things still smoother
– – review questions
– 16D Further possibilities
– – new components
– – heavier smoothers
– – problems
– – review questions
– 16H How far have we come?
17. Counts in BIN after BIN
– – review questions
– 17A Root smooth and root rough — bins of equal width
– – lengths of forearms
– – symmetry
– – concentrations of gold
– – review questions
– 17B Counts of basic counts
– – corn borers in field corn
– – octaves for logs
– – breeding pairs of birds
– – more problems
– – review questions
– 17C Fitting to smoothed roots
– – finding the peak
– – choosing an expression
– – making the fit
– – review questions
– 17D Corn borers, wheat prices, and Student’s simulations
– – Student’s simulations
– – wheat price changes
– – review questions
– 17E Bins of unequal width
– – Student’s t
– – fitting the smooth
– – review questions
– 17F Double roots
– – table
– – polonium schitillations
– – review questions
– 17G Cautionary examples
– – forearm lengths again
– – peak polishing (rest of section is optional)
– – polonium scintillations again
– – comment
– – review questions
– 17H How far have we come?
18. Product-ratio plots
– – review questions
– 18A Sizes and counts
– – “Zipf’s law”
– – c’rank
– – a possible approach
– – review questions
– 18B Product-ratio analysis
– – half-octave calculation
– – display
– – review questions
– 18C Forcing the unusual to be noticed
– – review questions
– 18D Comparison between collections
– – review questions
– 18E Looking at the smallest basic count
– – review questions
– 18F When zeros are counted
– – review questions
– 18G Under the microscope
– – review questions
– 18H How far have we come?
19. Shapes of distribution
– – review questions
– 19A Looking at shapes of distribution
– – Chevrolet prices
– – shapes
– – binned data
– – review questions
– 19B The Gaussian reference
– – review questions
– 19C Using letter values to look at shapes of distribution
– – pseudospreads
– – review questions
– 19D Pushback technique (optional section)
– – a half-octave version
– – review questions
– 19H How far have we come?
20. Mathematical distributions
– – review questions
– 20A Binnings vs. distributions
– – continuity and discreteness
– – fractions, densities, cumulatives
– – (mathematical) density
– – review questions
– 20B Densities for distributions vs. densities for binnings
– – correctino for curvature
– – average density
– – (mathematical) density
– – review questions
– 20C Tables and pictures comparing two sets of shapes of distribution
– – identification
– – further comment
– – review questions
– 20H How far have we come?
21. Postscript
– 21A Our relationship to the computer
– 21B What has been omitted?
– 21C How should the past chapters look different?
– 21D What have we been introduced to?
Glossary
Index to reference tables
Rearpapers
– 4. Pluralities, folded roots, folded logarithms (chap. 15 exhibit 2)
– 5. Values of log_e sqrt(count + 1/6) (chap. 15 exhibit 4)
– 6. Values of sqrt(count + 1/6) (chap. 15 exhibit 5)
==========