Distances between identical symbols in information strings (biological, language, computer programs (*.exe files) are described with a different precision with four distributions: Exponential, Weibull, lognormal and negative binomial. The correlations are sometimes highly significant. Here are analyzed distances between signs in the novel of A. C. Doyle. Some distance tests revealed specific formal features of the text.

This is a continuation study of statistical properties of distances between identical symbols in information strings (1, 2, 3). The Doyle's novel was obtained in the form of RTF. Using MS Word, the text was transformed into the plain *.txt. Some formatting, as headlines, remained unchanged. Then the file has 238430 bytes. It contains 230112 signs including spaces, 189183 signs without spaces in 4159 lines and 42549 words. It means that the mean length of a word is 4.446 signs (including apostrophes and punctuation marks). At some letters, the list of distances were split into more equal (approximately) parts, since the used statistical software Statgraphics (in version, I work with) does not work with too long lists.

After these formal corrections, the distances were determined by a program elaborated by Rádl. The string is at first indexed with the position index i (i going from 1 to m) of each individual symbol in the string, and then the differences of these position indexes are determined. The differences are considered to be the topological distances between the same symbols. The sets of these values were evaluated by different statistical tests. The program counting distances counts all signs, including spacebar, return, and punctuation marks.

From all available implemented distributions, only four distributions gave significant results, the exponential distribution, the Weibull distribution, the lognormal distribution, and the negative binomial distribution, as before.

The actual values (mean, standard deviation, skewness, kurtosis, distribution parameters etc.) are of little interest, since they differ considerably.

The distribution is of Weibull type, a = 1.75916, b = 108.163.

Chisquare = 10.4418 over 52, significance level was 0.72929 with 14 degree of freedom. There exists a shortage of points between distances 68 - 82 (237 occurrences against 265 expected). This alone makes 28.4 % of the chi-square test value.

The other punctuation mark, the semicolon (108 occurences), is used as follows:

Chisquare Test

Chisquare = 0.945022 with 5 d.f. Sig. level = 0.966877

This is an example of the almost perfect lognormal distribution.

There are many commas. Therefore, the file was split into 6 parts. The distribution of distances was exponential with different fitting, as follows in tabulated form

There are no immediately repeated commas, as ",," which contributes 21.2 - 50.78 % of the chi-square test value.

The two way sample analysis shows how the parts of the lower case differ:

Note: The asterisk shows the statistically significant difference between tested parts.

The commas are used without too great differences.

The distances between consecutive spacebars greater than 1 determine the number of words of the length corresponding to this distance minus one. There exists 40931 spacebars without corrections. Some of them are used as formatting tools. The results are tabulated as follows. Cumulating frequencies of shorter distances, improved in some cases the fit, since bellow it the counts are scattered, and differences can balance themselves.

The distribution of length of words seem to have the lognormal shape, but this guess was not tested.

Notes to some results:

The distribution of one letter words is correlated by the lognormal distribution. There are two peaks between distances 70-81 and 128-138 (36 (9) occurrences against 28.4 (5.6) expected). Each makes about 14.7 % of the chi-square test value. There exists a shortage of distances 13-23 (375 occurrences against 406.9 expected). Each contributes about 17.9 % to the chi-square test value. The distribution is shorter than expected (4 occurrences against 9.1 expected. This makes about 20.7 % of the chi-square test value.

The two way sample analysis shows that only the first and second parts are similar:

The distribution of three letter words was divided into four parts, too. The parts are correlated by different distributions:

The two way sample analysis shows that only the second and fourth parts are similar:

The distribution of four letter words was divided into four parts. These words are following each other more often than corresponds to the shape of the exponential distribution but not too much, at most 28.2 % of the chi-square test value in the first part. The parts are correlated as follows:

The two way sample analysis shows that only the second and third parts and the third and fourth ones are similar:

The distribution of five letter words was divided into three parts. These words are following each other slightly often than corresponds to the shape of the exponential distribution. The parts are correlated as follows:

The two way sample analysis shows that the parts are similar:

The negative binomial distribution of six letter words is fair, worsened by short distances (the chi-square test value is 0.681 over 7, 0.810 over 8, 0.420 over 9).

The distribution of seven letter words is described by the negative binomial distribution (the chi-square test value = 0.0529. It is somewhat improved over 4 to 0.2304). The tail is longer than expected (18 occurrences against 10.1 expected above 82). This makes 24.9 % of the chi-square test value.

The distribution of eight letter words is also the negative binomial one (the chi-square test value = 0.137). It is somewhat improved over 4 to 0.455). The tail is again longer (16 occurrences against 9.8 expected over 1.7). This makes 15.6 % of the chi-square test value.

The distribution of nine letter words is also the negative binomial one (the chi-square test value = 0.126 is improved over 6 to 0.871). The shortage of these words within distances 88-96 (7 occurrences against 13.9 expected) contributes 16.0 % of the chi-square test value.

The distribution of ten letter words is fairly correlated by the negative binomial distribution (the chi-square test value = 0.072 is improved over 25 to 0.605). The shortage of these words within distances 19-28 (107 occurrences against 132.4 expected) contributes 17.0 % of the chi-square test value. There are more distances 145-172 than expected (23 occurrences against 10.1 expected). This makes 31.9 % of the chi-square test value.

The distribution of eleven letter words is fairly correlated by the exponential distribution (the chi-square test value = 0.358). The shortage of these words within distances 160-186 (11 occurrences against 16.5 expected) contributes 18.3 % of the chi-square test value. There are more distances 28-54 than expected (183 occurrences against 159.0 expected). This makes 36.7 % of the chi-square test value.

The distributions of longer words are well correlated with the Weibull distribution. As an example, the results with the 16 letter words are given:

Chisquare Test

Chisquare = 2.20513 with 3 d.f. Sig. level = 0.530938

The results for all letters are presented in the form of the table, where the frequencies of all symbols are given and the significance of the performed chi-square tests. Then the commentaries to all symbols of the alphabet are given. The values in the square brackets show the corresponding values of the combined lower and upper cases.

Notes:

EX = exponential distribution

WE = Weibull distribution

L N = lognormal distribution

NB = negative binomial distribution

* = the test was not made, since not enough of data

Statistic = XX, the chi-square test

At the upper case, the Weibull distribution is the best one in the case of 16 letters. The lognormal distribution correlates 2 cases, only, the exponential distribution is the best in the 3 performed tests, and the negative binomial distribution in no case.

At the lower case, the Weibull distribution is the best one in the case of 8 letters. The lognormal distribution correlates 1 case, only, the exponential distribution is the best in the 13 performed tests, and the negative binomial distribution in 4 cases. At combined cases, the Weibull distribution is the best one in the case of 10 letters. The lognormal distribution correlates no case, the exponential distribution is the best in the 12 performed tests, and the negative binomial distribution in no case. Sometimes, the distinction between the fit is small and more than one distribution is applicable. The chi-square values sometimes are practically zero, and only adjusting the lowest possible value to greater distances by pooling these shorter distances increases the significance of the chi-square tests. Now, the commentaries to the individual letters follow.

A

The capital case A frequency allowed the separate test. The fair result was obtained with the exponential distribution (the chi-square test value 0.378). The excellent fit with the Weibulll distribution (the chi-square test value 0.913) is worsened by too many repeating within distances 2376 till 2850 (12 occurrences against 7.4 expected) which makes 26.0 % of the chi-square test value.

The distribution of distances between the lower case a is exponential, except that there are practically no repeating aa. This fact contributes 58.4 -77.3 to the chi-square value. The lower case a repeats too often within distances 6 - 14 (1.185 - 1.336 of expected values).

The two way sample analysis shows how the parts of the lower case differ:

Note: The asterisk shows the significant difference between tested parts.

The first sixth differs significantly from the second till fifth ones. The fifth and sixth are different, too.

The most important disturbances from the shape of the distribution in all parts are tabulated:

The lower case a repeats too often within one till three words.

The distances between both case (a + A) are fitted poorly by different distributions. Again, there are practically no repeating Aa. This fact contributes 58 - 77.6 to the chi-square value.

The first sixth of a fits well with the negative binomial distribution with pooled distances to 16 (the chi-square value = 0.592. Other parts give much worse fits, and other distributions (the exponential distribution and the negative binomial distribution) give a better fit.

The two way sample analysis of both cases (a + A) gives worser results as the lower case a:

The first sixth differs significantly from the three parts but its consistency with other parts is low, too. The most important disturbances in all parts are tabulated, again:

B

The distribution of distances between upper case B is Weibull. The distribution of distances over 20 between lower case b is exponential, the chi-square test value is then 0.614. There are too few b within distances 129-150 (106 occurrences against 128 expected), which contributes 19.3 % of the chi-square test value. Contrary, there are too many b within distances 282-324 (63 occurrences against 46 expected), which contributes 34.3 % of the chi-square test value. The distribution of distances over 20 between (b +B) is exponential, the chi-square test value is then 0.921. But here the Weibull distribution gives even a better chi-square test value 0.927. The fit

is worsened by too many (b + B) within distances 295-316 (31 occurrences against 20.7 expected), which contributes 40.9 % of the chi-square test value. There are too few (b + B) within distances 422-442 (1 occurrence against 5.4 expected), which contributes 28.5 % of the chi-square test value.

Including B improved the fit, the disturbances lessened and shifted to longer distances.

C

The distribution of distances between upper case C is the Weibull one (the chi-square test value is wery good, 0.932).

The distribution of distances of the lower case of this letter (and c + C) is described well by three distributions, exponential, negative binomial and Weibull.

The distances between lower case c were split into 3 parts. The results are tabulated:

The parts are rather different, as two way sample analysis shows:

The distances between (c = C) were split into 3 parts, too. The results are tabulated as follows:

The parts are rather different, as two way sample analysis shows:

Combining both cases worsened the fit. It is difficult to choose between the exponential distribution and the negative binomial distribution, both give practically the identical results.

D

Here the exponential distribution and the negative binomial are applicable. The chi-square test values are as follows:

The capital case D frequency allowed the separate test. The lognormal distribution correlates poorly, the chi-square value is only 0.048 since there are too many repeating within distances 1274 till 1909 (20 occurrences against 12.6 expected) which makes 34.3 % of the chi-square test value. The tail is shorter than expected, only 1 occurrence against 5.1 expected, which contributes another 25.9 % of the chi-square test value.

There are too few repeating dd (Dd). This fact contributes 42.4 - 70.5 % (32.3-63.8 %) to the high chi-square values given in the table above.

The two way sample analysis shows that the parts of the lower case d are different:

The third sixth differs significantly from the first part. Only the third and the fourth parts are similar.

There are always less doubled dd then corresponding to the exponential form which makes 42-70.5 % of the chi-square test value.

The combined [d + D] gives somewhat different results. The two way sample analysis shows that the parts of [d + D] are different, too:

The third and fourth parts differ significantly from the first part. Only the third and fourth parts are close.

There are always less doubled Dd then corresponding to the exponential form (0-10 occurrences against 23.5-37 expected) which makes 32.3 - 63.8 % of the chi-square test value.

E

There are relatively few E comparing with the great number of e. The distribution of distances between lower case e and both case (e + E) is mostly the negative binomial, some parts fit better the lognormal or exponential distributions:

The two way sample analysis failed due to too large samples.

F

The distribution of the capital F, of the lover case f, and of [f + F], is correlated with the Weilbull distribution. The set of the lover case f, and of [f + F], were divided into two parts, which both are rather different (the two way sample analysis results 0.0002 and 0.0048, respectively.

The distribution of this letter is distorted by too few double ff [Ff] (e. g. 96 occurrences against 28.7 expected). This makes 84.9 % of the total very high chi-square test value.

G

The distribution of the capital G is correlated with the Weibull distribution rather well. It effects the distribution of the lover case g, divided into two parts, in both parts differently:

The most important distortions:

H

The distribution of the capital H is correlated with the Weibull distribution rather well:

Chisquare Test

Chisquare = 8.67292 with 8 d.f. Sig. level = 0.370635

The surplus of distances 1057-1267 is followed by the shortage of longer distances.

The frequency of h made necessary to split the set for the evaluation into four parts which correlated badly with the negative binomial distribution (1. part has the chisquare value 0.315 over 30) but they were still too long for the two way sample analysis. [g + G] was split for the evaluation into six parts which correlated badly with the negative binomial distribution (e. g. 3. part has the chisquare value 0.115 over 27)

The two way sample analysis shows how the parts are different:

I

The distribution of the capital I is correlated poorly with the lognormal distribution. The greatest disturbance is a shortage of counts within distances 305-607 (102 occurrences against 125 expected) which contributes 39,1 % of the chi-square test value. The tail is longer over distances 1516 (17 occurrences against 10,2 expected) which contributes another 49,5 % of the chi-square test value.

The frequency of the lower case i made necessary the splitting. The parts are poorly correlated with the exponential distribution, as the best the 5. part (the chi-square test value 0.701 over 5), and they pass the two way sample analysis, as follows:

Only the first part differs significantly from the others, since the result with the fourth part is only slightly above the limit of rejection. There are no repeating ii. This makes 55.6-78.2 % of the chi-square test value. The distribution is more skewed, there exists always a surplus of intermediate distances:

The including of I changed the results of the two way sample analysis as follows:

In most cases, the similarity is worse. Only the fourth and fifth parts are less different. There are no repeating Ii. This makes 60.5-70.1 % of the chi-square test value. The distribution is more skewed, there exists always a surplus of intermediate distances:

J

The distribution of the letter is the Weibull one. The Weilbull distribution of the lower case j is better correlated than both cases [j + J]. There are too many distances 874-1310 (22 occurrences against 16.7 expected). This makes 25.0 % of the chi-square test value. Contrary, there are too few distances 2619-3055 (10 occurrences against 6.4 expected). This makes 31.2 % of the chi-square test value. ombining both cases worsened the fit. There are too many distances 963-1284 (52 occurrences against 40.4 expected). This makes 44.2 % of the chi-square test value.

K

The Weilbull distribution of this letter is bad. There are no repeating kk [Kk]. This makes 18.5 [19.3] % of the chi-square test value.

L

The occurrences of capital L is correlated by the exponential distribution. There are too many distances 2653-3173 (14 occurrences against 11.3 expected). This makes 54.1 % of the chi-square test value.

The frequency of l and [l + L] made necessary the splitting.

The parts are correlated with the Weilbull distribution. It is distorted by many double ll [Ll]. This makes 74.8-79.9 % [74.8-81.2 %] of the total chi-square test value. The parts fit over different distances rather well, see table:

The parts of l pass the two way sample analysis, as follows:

The fourth part differs significantly from the first and second ones, the first from the fifth one.

The including of I changed the results only slightly, see table:

M

The upper case M is correlated well using the Weilbull distribution (the chi-square test value is 0.458).

The lower case m, divided into 3 parts, is correlated as best with the exponential distribution (1. part the chi-square test value over 44 is 0.798, 2. part the chi-square test value = 0.443, 3. part the chi-square test value = 0.137). The doubled mm fit excellently only in the second part, in other parts, the repeating mm is more scarce than expected.

The parts of the distribution of m are different. The two way sample analysis shows following results:

The upper case m, divided into 3 parts, is correlated as best with the exponential distribution (1. part the chi-square test value over 14 is 0.849, 2. part the chi-square test value = 0.096, 3. part the chi-square test value = 0.082). The doubled Mm fit only in the second part, in other parts, they are more scarce than expected.

The parts of the distribution of [m + M] are different, too. The two way sample analysis shows following results:

N

The upper case N is correlated using the Weibull distribution (the chi-square test value is 0.072). There are too few distances 1600-2000 (7 occurrences against 13.3 expected). This makes 25.6 % of the chi-square test value.

The distribution of n and (n + N) was divided into seven parts.

The distribution of this letter is distorted by too few double nn [Nn] (e. g. 10 occurrences against 93 expected). This makes 44.0-67.9 % of the total very high chi-square test value. In some parts are rather great disturbances:

The two way sample analysis shows following results:

[n + N]:

In some parts are rather great disturbances:

The disturbances have slightly less weight than at the lower case n.

The two way sample analysis shows for [n + N] the following results:

Both sets are alike, the including of N did not changed the results of the two way sample analysis dramatically. The second part differs signficantly from the firts and fifth ones.

O

The distribution of O can be correlated also with the Weibull distribution (the chi-square test value 0.077). There are too many distances 3572-4714 (14 occurrences against 7.3 expected). This makes 35.1 % of the chi-square test value.

The distribution of o and (o + O) was divided into seven parts, which correlated poorly with the exponential distribution.

The distribution of this letter is distorted by too few double oo [Oo] only slightly,

(at most the first part of o, 72 occurrences against 119 expected). This makes 40.6% of the total very high chi-square test value. Here are the greatest disturbances:

In four parts, an excess of distances 14-22 occurs.

The two way sample analysis shows following results:

The first part differs significantly from the fifth one.

[o + O]:

In some parts are rather great disturbances:

The excess of distances 14-22 occurs again. The disturbances have slightly less weight than at the lower case o.

The two way sample analysis shows for o + O the following results:

There is no significant difference between parts.

P

The upper case P correlated also using the Weilbull distribution (the chi-square test value = 0.356). The Weilbull distribution of p and [p +P] is distorted by too many pp [Pp] (69.8-80.8 [72.2-73.6] % of the chi-square test value). The other disturbances give only a minor opportunities for commenting. The both sets were divided into 3 parts. The test did not revealed their dissimilarity.

Q

This letter correlated also using the Weilbull distribution (the chi-square test value = 0.367 at q, 0.389 at [q + Q]) but better fit gives the exponential distribution (the chi-square test value = 0.441 at q, 0.433 at [q + Q]).

R

The upper case R correlated with the Weilbull distribution (the chi-square test value = 0.109). The fit distribution is worsened by too many repeating within distances 1000 till 2000 (45 occurrences against 33.3 expected) which makes 40.4 % of the chi-square test value.

The distribution of r and [r + R] was divided into five parts. They fit with the exponential distribution.

The most important disturbances from the shape of the distribution in all parts of r are tabulated:

The two way sample analysis shows that the parts of the lower case r are rather different:

The third and fourth part differ significantly from all other parts.

The most important disturbances from the shape of the distribution in all parts of [r + R] are tabulated:

The two way sample analysis shows that the parts of [r + R] are rather different:

The third and fourth part differ significantly from all other parts. Combining r with R changed the distribution insignificantly.

S

The Weibull distribution of the capital S is distorted mostly by a walley within distances 1600-2000 (10 occurrences against 14.2 expected). This makes 50.8 % of the chi-square test value.

The distribution of the lower case s an d [s + S] was divided into five parts.

The most important disturbances from the shape of the distribution in all parts of s are tabulated:

The two way sample analysis shows that the parts of the lower case s are rather similar, except the fifth part which differs from the first and third parts:

The most important disturbances from the shape of the distribution in all parts of [s + S] are tabulated:

The two way sample analysis shows that the parts of [s + S] are rather similar:

The third and fourth part differ significantly from all other parts. Combining r with R changed the distribution insignificantly.

T

The distribution of the capital T has the Weilbull shape. There is more distances 544-679 (54 occurrences against 43.3 expected). This makes 26.2 % of the chi-square test value.

The distribution of the lower case t as well as the both [t + T] is divided into six parts.

The most important disturbances from the shape of the exponential distribution in all parts of s are tabulated:

There are too few repeated s in all parts. Moreover, the shape of the distribution is rather sharp in the range 5-20, except the third part.

The most important disturbances from the shape of the distribution in all parts of [s + S] are tabulated: