June 29, 99

Concerning the Statistical Test that was Published
in our Paper in Statistical Science

by Doron Witztum


1. The Accusations of McKay et al:

In their article, "Solving the Bible Code Puzzle" which is scheduled to appear in the May 99 issue of Statistical Science, McKay, Bar-Natan, Bar-Hillel, and Kalai (MBBK) write the following about the statistical test that Witztum, Rips and Rosenberg (WRR) utilized:

"To correct the error in treating P_1-4 (that is, P₁, P₂, P₃ and P₄) as probabilities, Diaconis proposed a method that involved permuting the columns of a 32X32 matrix, whose (i,j)th entry was a single value representing some sort of aggregate distance between all the appellations of rabbi i and all the dates of rabbi j. This proposal was apparently first made in a letter of May, 1990 to the Academy member handling the paper, Robert Aumann, though a related proposal had been made by Diaconis in 1988. The same design was again described by Diaconis in September (Diaconis, 1990), and there appeared to be an agreement on the matter. However, unnoticed by Diaconis, WRR performed the different permutation test described in Section 2." (Section 3)

Further on (Section 10), they name the test that was published in our article "the test invented by WRR," as opposed to " the permutation test of Diaconis."

Even before this, Dr. McKay had written the following letter, which was published in "Galileo" (issue #27, March-April 98):

"A Grave Mistake:
My intention is to correct a grave mistake in Doron Witztum's letter to the editor ('A Refutation Refuted', Galileo #26, p. 75). He claims that the experiment involving the bible code that he published in Statistical Science utilized a test that was devised by Prof. Diaconis. This assertion is false and extremely misleading. Documents, copies of which I have in my possession, demonstrate that he overlooked Diaconis' test and set up another test, the result of which was hundreds of times better. The data had already been in his possession for three years. All of this has already been confirmed by individuals who are knowledgeable in this matter, among them his colleague in the research and the article. Mr. Witztum, however, continues to tell the same lie. This testifies to his integrity." (translated from the Hebrew.).

Dr. McKay's letter (which, by the way, is typical of his style) should surprise us for its impudence. Dr. McKay can assume that the readers of Galileo have no idea what the experiment was that was agreed upon with Prof. Diaconis. He should not, however, allow himself to assume that the readers of Galileo have faulty memories. Here is a quote of my exact words that were published in that issue of Galileo:

"After a great success in measurement for the second list as well, Prof. Diaconis suggested that we use a new method of measurement, and try it out on the second list. This is what we did, and the surprising results of the experiment...." (translated from the Hebrew).

Now let us compare my statement to what Dr. McKay himself wrote (together with Dr. Bar Natan and Prof. Bar Hillel, in their article in Galileo #25, p. 53):

"Prof. Persi Diaconis, world renowned mathematician and statistician... suggested another method to them, which they used in the paper that was published in Statistical Science." (My emphasis. This quote was also translated from the Hebrew.)

Amazing? - But this isn't all. We will now unfold before the reader with entire course of events, and clarify what in truth the documents in the possession of MBBK prove.

2. The truth about the statistical test that was published in our paper in Statistical Science

Our paper, "Equidistant Letter Sequences in the Book of Genesis," was submitted for publication to PNAS by Prof. Robert J. Aumann, professor of mathematics at the Hebrew University of Jerusalem and member of the American National Academy of Sciences. Within the framework of this attempt, he had discussions and correspondence with the referee, Prof. Persi Diaconis. In September 90, the stage at which they concluded their discussions, they were both at Stanford University, and they exchanged letters as follows:

Prof. Persi Diaconis sent the following letter to Prof. Aumann on September 5, 90:

Professor Robert Aumann
Department of Economics
Mail Code 6072
Stanford University
Stanford, CA 94305

Dear Bob:
I am glad to report we are in agreement about the appropriate testing procedure for the paper by Rips et al. A permutation test is to be performed. There are four basic sets of data/test statistics, I will call them additive, multiplicative, with and without Rabbi. For each there is a 32X32 table of distances. It is my understanding that for each such table, one million permutations will be performed. For each permutation SIGMA _i=1³²t_ip(i) will be computed. This gives one million numbers/table. Again for each the number SIGMA t_i will be located. If it is within 1/4000 of the smallest table sums, that test is judged a success. If one of the four tests is successful, the whole experiment is.
         In case of ties, the interval of ties will be broken at random. If half the proportion of such breaks amount to better than 1/4000, that table is successful. Otherwise not.
I hope that the authors agree to make their findings public no matter what the outcomes. Please let me know when you need from input from me.
Sincerely,
Persi Diaconis

A number of things are not clear in this letter. For example: 1) It isn't clear what the intention is in the adjective "additive," vis a vis one of the statistics. 2) It isn't clear what the "distances" are that make up each table. 3) It seems that he is referring to four different tables, and it isn't clear at all what he is referring to. 4) It isn't even clear which list of names/dates is to be used in the suggested experiment.

In order to clarify all of these, Prof. Aumann wrote the following letter to Prof. Diaconis on September 7, 90:

Professor Persi Diaconis
Department of Statistics
Stanford University
Stanford, CA 94305

Dear Persi,
Thanks for your good letter of September 5, about the paper submitted by Rips et al. to the PNAS.
Since it's important to clarify the precise rules of a statistical test before performing it, allow me to set down here a few points of clarification.
The same 1,000,000 permutations may be used for each of the four basic tests. The million will consist of the identity permutation plus 999,999 others. All million will be different from each other.
The sample to be examined is that of their "second experiment" (Table 3 of their submission). For each of the four basic tests, the exact same procedures as reported on in their paper (Tables 5 and 7) will be done for each of the 1,000,000 permutations. (Incidentally, "bunching" or "twenty percent" might be a more suggestive name for the test you call "additive").
The precise tie-breaking rule (agreed on by phone today) is this: Out of the million permutations, let there be s that are ranked smaller than the identity, and t with which it is tied (excluding itself). Then the test is successful if and only if s+(t/2) < 250.
Again, with many many thanks for all your help on this,
Sincerely,
Bob Aumann

Bellow his name, Prof. Aumann added in handwriting:

"given to Persi by hand in Sequoia hall, September 9, 1990, 2:50 PM. He looked it over and approved."

That is, the details in Prof. Aumann's letter were approved by Prof. Diaconis.

When Prof. Aumann came to Jerusalem, he presented the above agreement to us.
We thereupon wrote our paper anew, exactly according to the details of the agreed upon experiment, in which "question marks" replaced "results". During the course of 1990-1991, Prof. Aumann sent this new paper to Prof. Diaconis and to other referees (all of them members of the American National Academy of Sciences). They were all asked to comment on the described experiment, and to establish a success threshold. Prof. Diaconis didn't claim that we "overlooked his test and set up another test". The experiment itself was performed at the end of '91 after Prof. Aumann established, with the help of Prof. Diaconis and two other referees, the statistical seed needed to run the permutation test. The results were now incorporated into the paper in the place of the question marks. All of this was reported to the referees, who were asked to write a referee's report.

It is thus clear that the exact description of the experiment, as detailed in the Statistical Science paper, was inspected by Prof. Diaconis and the other referees before the experiment was performed.

To conclude this chapter, I will quote from a letter that Prof. Aumann later wrote to Prof. Bar Hillel on January 17, 97, in which he describes the chronology of our research in its various stages. At what he calls Stages J and K, he writes:

"J.        The details of a formal test are agreed between Diaconis and Aumann (I'm trying to avoid pronouns, because they often lead to confusion).
K.        The formal test turns out significant at a level of 16 out of a million. (That is, the best result of the four statistics is 4 out of a million, and then Bonferoni.)"


3. Have MBBK seen the documents presented above?

Yes! On September 9, 93, Prof. Bar-Hillel received from Prof. Aumann "all of my correspondences with Prof. Persi Diaconis concerning the work of Rips et al" (Prof. Aumann's words on the accompanying letter).

If so, why are MBBK hiding the relevant information, and in its stead, raising accusations that have no basis? It seems that they have no trust whatsoever in their other claims. At any rate, if we might use Dr. McKay's own words quoted above from his letter to Galileo, "This attests to their lack of integrity."