Dr. Dror Bar-Natan, Hebrew University
Dr. Brendan McKay, Australian National University
Prof. Shlomo Sternberg, Harvard University
A sequel to the Statistical Science paper has been circulating in preprint form for several years. Called "Equidistant Letter Sequences in the Book of Genesis: II. The Relation to the Text", it reports on three new experiments conducted by Witztum, Rips, and Rosenberg. One of those experiments concerns the list of 70 nations that appears in Chapter 10 of Genesis. A significance level of 4 parts in a billion is claimed as the result.
The data used by WRR in their experiment consists of a considerable number of pairs of the form (N,X), where N is the name of a nation and X is some related word or phrase. The pairs can be divided into regular pairs and irregular pairs. The regular pairs use the phrases "nation of N", "country of N", "language of N" and "script of N". Use of these attributes is explained by reference to a commentary written by the Vilna Gaon. The irregular pairs use the phrase "people of N", but only sometimes, and new names for the nation or country of N which are drawn from an Aramaic translation of Genesis called Targum Pseudojonatan.
The analysis method used by WRR is essentially the same as in the Statistical Science paper, except that the nation names are sought at skip +1 and -1 only. The related phrases are still sought as ELSs.
Our paper demonstrates in great detail how the data shows every sign of having been carefully adjusted with the result in mind, which of course means that the experiment is invalid and the result meaningless.
The first cause for alarm comes from realising that the Hebrew forms of the regular pairs do not follow from the Vilna Gaon's words. He wrote of the attribute "name", not the attribute "nation", but the obvious phrases "name of N" and "N" fail completely, as does the word for "nation" used by the Torah. Instead, WRR use a different word that performs very well. Similarly, the word for "language" used by WRR is different from the one used by the Vilna Gaon, and again the change was greatly in their favour.
To investigate this situation, we made an a-priori list of 136 phrases of comparable form, such as "person of N", "leader of N", "army of N". When we computed the P2 scores for each attribute, we obtained a reasonably uniform spread of values between 0 and 1, consistent with chance. Amazingly, three of the four forms used by WRR were the best three in our list of 136! To show this observation in perspective, we construct several similar experiments that perform well in War and Peace. One of them, supported by a commentary of the Ramban (Nachmanides), performs 100 times as well in War and Peace as the phrases used by WRR perform in Genesis, even though, unlike them, we used the exact spellings given by our source.
Turning to the irregular related phrases used by WRR, we find them to be the result of a considerable number of arbitrary decisions, many of them highly doubtful. In every case, we find that making alternate decisions causes the result to get worse.
The use made of Targum Pseudojonatan is especially revealing. Not only do WRR use an edition that is well known to be replete with errors, but they modify the data in their source in ways often not defined in their paper. Undoing those modifications, or taking the data directly from more accurate editions of the Targum, completely eliminates the benefit to the WRR result from the irregular part of the data. This observation strongly confirms our conclusion that the choices made in preparing the data were not made a-priori.
Finally, we show by simulation experiments that the permutation test exhibits anomalous behaviour for data of this nature, and provide a partial theoretical explanation. Thus, even if the data was prepared properly, the result is meaningless.