This is a copy of an article that Doron Witztum published as a PDF file. The Hebrew has been transliterated, using the Michigan-Clairmont encoding scheme. No claims of accuracy are made.
A reply to this article is available.
Does Tolstoy Really Love Brendan McKay?
Doron Witztum, 1997.
On July 31st, 1997, Dr. Brendan McKay of the Australian National
University presented an article on the internet, in which he claims
that the personal details of his life "amazingly" appear encoded at
minimum ELSs along with his name, in Tolstoy's War and Peace. He
claims the probability of this occurring by chance to be less than
one in five thousand.
In this response, we will show that Dr. McKay achieved his results
through a combination of systematic error and very partial reporting
of the number of trials conducted.
The two errors were employed by Dr. McKay with the aim of creating
a "counterfeit" effect of success. We will demonstrate that real
probability involved in this experiment corresponds exactly with
what is expected to take place randomly.
A. THE TYPE OF SAMPLE
In a previous version of his sample (in a letter sent to the Internet
TCODE group on May 5, 1997), Dr. McKay describes his experiment as
follows:
Forwarded message:
From: 0 bdm@cs.anu.edu.au (Brendan McKay)
Sender: 0 owner-tcode@express.ior.com
Reply-to: 0 tcode@express.ior.com
To: 0 tcode@express.ior.com
Date: 97-05-05 06:02:02 EDT
Just today I calculated my own birth date in the Jewish calendar.
It was 26th Tishri 5712. Then I tested it against my name BRNDN,
using the usual date forms and the year forms chosen for our new
experiment. (My surname is shorter than 5 letters.)
The results are amazing. The average distance over the three date
forms is 0.176, compared to 0.328 and 0.330 for the two lists in
the StatSci paper. The average distance over the five year forms
in the 5-8 length range is 0.261, though it would be only 0.138
except for a single large value. The geometric means (which are
a crude measure of how much contribution is made to the
P2 statistics) are 0.168 and 0.149, compared to 0.210 and 0.215
for the two StatSci lists. I would be quite a champion in the
StatSci lists.
Brendan
PS. Perhaps I should mention that the text in which I measured
my name was War and Peace.
In other words, he searched for "meetings" of the ELSs of the
name BRNDN (Brendan) with those of his birth date (day and month)
in three forms, and with the year of his birth written in six forms.
This type of sample, in which we measure the proximities between
ELSs of a single expression and those of a list of other expressions,
we termed a "heading sample". In our paper "Hidden Code in the Book
of Genesis", ("CPN XMWY BSPR BR)$YT" preprint 1996; originally
presented as a lecture before the Israeli National Academy of
Sciences, 19 March, 1996), we analyzed this type of sample, and we
explained how to carry out the randomization process for it.
In his final version, Dr. McKay selected another heading, DR MQYY
(Dr. McKay) and created another heading sample with the same dates;
the final product being the union (merger) of these two samples.
(He did one other experiment, searching for "Canberra". This will
be discussed in section F).
B. THE METHOD OF MEASUREMENT
As we explained in the above-mentioned paper, the proper method for
analyzing a heading sample has three stages:
1. We use the function c, ("the Corrected Distance" in the
Statistical Science paper), to measure proximity between the heading
and each expression w (from the list). However, the function has
the following alteration: the heading is only taken as ELSs, while
the expression w is taken as ELSs and as PLSs (perturbed letter
sequences). In other words, ELSs of w compete with the PLSs of w
over the more successful proximities to the ELSs of the heading.
If there are n expressions in the list, the values of P1 and P2
(the Overall Proximity Measures) are calculated based on the n
results.
2. Given a list with n expressions, a series of n permutations is
randomly drawn: permutation i changes the order of the letters in
expression i. In this way, n new expressions are derived and their
proximities with the heading are calculated (as explained above in
stage 1). The values of P'1 and P'2 are calculated for this series
of n permutations.
3. This process is carried out N times, and the rank order of P1
and P2 is calculated.
C. THE RESULTS FOR MCKAY'S SAMPLE:
When Dr. McKay's double "heading" sample is calculated in this manner,
the following results are obtained:
At stage 1: P1 = 0.082 and P2 = 0.031 At stage 3: The rank order of P1
for N = 100,000: 3955 The rank order of P2 for N = 100,000: 1199
i.e., the probability is p = 2 x 0.012 = 0.024, that is, only 1/40,
instead of 1/5000.
(Of course, 1/40 is the probability if the headings BRNDN and DR MQYY
were the only ones checked. But this is not the case. In section E, we
will show that the final probability is actually much higher than even
this.)
In what follows, we explain the source of the differences between
these findings and those of McKay:
D. MCKAY'S SYSTEMATIC ERROR
1. A few months ago, (in a posting called "Equidistant Letter Sequences
in Genesis- A Report. Feb. 22, 1997), Dr. McKay noted that some of the
names in the list of Rabbis included in the experiment published in
Statistical Science, had the following property: their ELSs appeared
more often than their PLSs. This advantage may allow for easier
"successful" proximities with ELSs of other expressions. Suppose that
expression x possesses this advantage. If we calculate the proximities
between its ELSs and those of the series of expressions, w1, w2,.. wn,
we will receive an overabundance of "successful" results. This will
happen whether the other expressions are related to x or not. This
is a systematic error because we exploit a certain property of the
expression more than once.
This possible advantage connected to the number of ELSs of a word, may
be only one possible advantage that an expression may have. Regardless
of the origin of the advantage, let us call expressions with these
types of advantages "charismatic" expressions.
Naturally, besides charismatic expressions, some expressions may also
be "anti-charismatic". For instance, if the ELSs of an expression
appear less often than its PLSs. Such expressions will tend to produce
"failures" in measuring proximities with other expressions. In the
sample of the Rabbis, there were many names and appellations. Some may
be charismatic, and others may be anti-charismatic. One would expect
that these effects will balance each other out.
The permutations test proposed by Professor Persi Diaconis, as
described in the Statistical Science paper, is supposed to eliminate
any such residual effects of charisma for symmetry considerations. If
we assume that the success of the proximities of name x is due to its
charisma, then it should be just as successful with unrelated dates
to which it is paired as to related dates. Thus, the permutation test
should cancel the "charisma" (advantage) of individual terms.
However, Dr. McKay (in his Report) argued that the permutations test
may not have succeeded in canceling this effect. We thought Dr. McKay
was wrong, and that Professor Diaconis' permutation test was a good
one. Nonetheless, it occurred to us that to demonstrate to Dr. McKay
that he is incorrect, one could apply the test described in B1
(described in the above mentioned paper). In a response sent to him
(A Preliminary Analysis and Comments on the Report of New ELS Tests,
June 19, 1997. Posted on the internet), Professor Eliyahu Rips proposed
the following test (Stage 1 in section B above):
To repeat the experiment described in Statistical Science, with the
single variation that the names will be taken only as ELSs, while the
dates as ELSs and PLSs. This simple procedure cancels all possible
effects due to charismatic names.
When we carried out this experiment, it turned out that the results
improved. Whereas the ranking of the most successful statistic P4 was
originally four in a million, it now improved to one in a million. In
other words, the success of the Statistical Science experiment was
clearly not caused by this effect.
2. When one is conducting an experiment that involves a heading sample,
the need for caution cannot be overstated. If the expression being used
as a heading happens to be charismatic, then it will "succeed" with
many expressions, whether they are related or not.
It appears that Dr. McKay neglected to take into account his own
observation: he happens to have used charismatic expressions for
the heading in his sample, and seems not to have realized that the
"success" of his results is entirely artificial. Moreover, the
randomization technique Dr. McKay performed, instead of offsetting
the charisma, ignored it altogether. Indeed, as per the discussion
in section B above, the use of only ELSs for the headings and the
appropriate randomization technique, is sufficient to make the
artificial result of 1 in 5,000 into a real one of 1 in 40.
3 . In my opinion, there is an even more basic approach to the
prevention of errors of this sort, and it touches on the question of
the exact phenomenon being traced. Here is not the place to discuss
this subject at length. It will be covered in detail in my forthcoming
book.
One more remark: in the summer of 1985, we set out to define a
"distance" between two expressions, w and w'(a measure of proximity).
In general, each expression will be represented by several ELSs.
First, we define a "distance" between a particular ELS of w and a
particular ELS of w'. We then have two options: a) To sum all the
"distances" between ELSs of w and ELSs of w'. We call this option TOT.
b) To take only the best "distance" of all these distances. We call
this option BEST.
At the time, Professor Rips suggested that TOT would provide a more
stable measure, so we used it for all our experiments, including the
Rabbis experiments. This is the function described in our 1994
Statistical Science paper, from which function c is derived.
Last year, I conducted several experiments (also described in detail
in my forthcoming book), using the BEST measure. I believe that BEST
avoids the problem of charisma, arising from ELSs appearing more
often than expected. It must be stressed, however, that BEST should
be considered only as an additional or complementary measure of
proximity, not as a replacement for TOT.
If we measure Dr. McKay's sample using BEST, after randomization
(section B, stages 2 and 3) the resulting p value is reduced even
further to p = 0.019 x 2 = 0.038, or roughly 1/25. In other words,
using BEST, there is no success for Dr. McKay's list.
E. SELECTION OF THE HEADINGS:
Until now, we did not take into account the number of headings used by
Dr. McKay. Basically, there are two possible ways of transcribing the
name "McKay." There is a transcription according to the written
characters and a phonetic transcription (which Dr. McKay claims he
prefers). There are also two ways to spell the name, either with a
single "Y" at the end or with a double "YY". It is a mistake to write
McKay with a double "YY". Nonetheless, since he used it, we will go
along with his choice.
1. Transcription according to the written characters: Dr. McKay uses
the Encyclopedia Hebraica for transcription. For example, he directs
us to copy the name QNBRH (Canberra) from there. In this encyclopedia,
in volume XXII, p. 965, (eighth line from the end of the page), one
finds two transcriptions beside each other: MQQLWQ for McCulloch and
MQY for Mackay. According to the encyclopedia, Dr. McKay should
properly spell his name: MQQY.
Moreover, Dr. McKay himself, in a paper describing Chanukah codes in
War and Peace, used the transcription MKQY (with a single "Y").
Dr. McKay also provides us with an explanation why he ignored the
very encyclopedia he used in his present article and why he is not
consistent in his usage of MKQY as before. He states that the only
time his name was mentioned in the Israeli press (the Israeli
newspaper Maariv), it was spelled MQYY. In truth, we think that the
transcription MQYY is a mistake altogether. A newspaper should not
be considered an authoritative source for spellings of transliterated
words. Nonetheless, since he used it, we will consider its use.
However, Dr. McKay's claim that his name was never otherwise mentioned
in the Israeli press is simply not true. In an article in the magazine
Mishpacha, published on the July 3, 1997, (page 13, Column 1, Line 14),
his name is spelled MQQYY. And if Dr. McKay should claim that he did
not know of this publication, it would be a strange claim the
dedication at the head of his own article: LA.L. M$T"P )MYC (M HAMT
is based on that same piece in Mishpacha.
In summary, using the transcription according to written characters,
in view of previous publications by Dr. McKay, the sources he has
variously cited himself, and the fact that he sometimes uses one "Y"
and sometimes two, we have with the following eight variations:
MKQY, MKQYY, MQQY and MQQYY, and with the D"R (Dr.) as a prefix:
DR MKQY, DR MKQYY, DR MQQY and DR MQQYY.
Of these, there are 5 expressions of 5-8 letters each, which appear
as ELSs in War and Peace: D"R MKQY, D"R MKQYY, MKQYY, D"R MQQY, MQQYY.
It should be remembered that according to the criteria established
in his email (page 1 above), Dr. McKay only wanted to use his surname,
but in fact, he used both his surname and the D"R prefix.
2. The phonetic transcription: According to the above-mentioned
Encyclopedia, the name should be spelled MQY with a single "Y".
This usage of one "Y" is how he himself spelled it in his paper
on Chanukah. Here he freely adopted MQYY with a double "YY".
By adding the prefix D"R, we have 2 expressions of 5-8 letters each:
D"R MQY and D"R MQYY.
In all, we have found 5+2=7 different expressions which may be
found as ELSs in War and Peace.
3. The transcription of his first name: Dr. McKay spells the name
Brendan: BRNDN, but in the same article of the Maariv newspaper
which is the source of his spelling of MQYY, indeed, at the same
place in the article, the name is written BRNDWN rather than BRNDN.
In summary, Dr. McKay thus appears to have tried at least 16 different
headings. According to his letter, he tried BRNDN separately and
according to his article he tried combinations. In other words, he
could have readily tried BRNDN separately and also BRNDN plus
each of the above 7 possibilities, i.e., a total of 8 headings. The
same goes for BRNDWN, which translates into a total of 16 different
headings. Please note that all of the above calculations are based on
Dr. McKay's own data and selections, and we have made no references to
other possibilities which may have been taken into consideration.
F. THE SELECTION OF THE EXPRESSIONS IN THE LIST:
Once Dr. McKay added QNBRH (Canberra, which is not the place of
birth) to the list, he revealed that he had an inestimable space of
biographical details to work with. Not only is it prohibited to include
QNBRH on the list, it also casts a dark shadow on the remainder of
the list.
Indeed, it is clear that Dr. McKay was interested in both the date
and place of his birth. Thus, his list of names and dates is itself
a partial list of the biographical details of his birth: it contains
only the date (day and month), and the year, but does not include
the place. If he takes a partial group of the details, we have to
calculate how many sub-groups he could have taken, searching for
that cluster which offers the most impressive proximity. Dr. McKay
had three options available:
1. Dates stating only the day and month (K"W T$RY, etc.).
2. Dates stating only the year.
3. Options 1 and 2 combined.
G. SUMMARY:
If we compile all the selections Dr. McKay could have made for the
headings and for the list, it follows that even the most conservative
calculation leads to at least 16x3=48 different possibilities. In
fact, in his email, Dr. McKay reports the use of additional two
statistical analyses beyond P1 and P2, so the final probability
obtained in section C (p=0.024) should be multiplied by at least
2x48=96.
In other words, the statistical significance of Dr. McKay's results
does not deviate at all from that expected for a random event.
Note: All the above calculations are based on the assumption that
the data supplied by Dr. McKay is accurate. We have reason to be
skeptical of its accuracy given the serious flaws in the data of his
earlier reports.
We have shown that Dr. McKay's success was in fact artificial.
For the sake of comparison, we will shortly publish an authentic
example of a heading sample.