Taking Eder/Rybicki’s Stylometry Script for a test drive
One of the things I planned to do with this blog is to document my experience with various tools for computational text analysis. This is a way of letting other people know what these tools can do, and what I am currently working on and maybe hear their (your!) thoughts on it; it is also a way of forcing myself to document the various steps and settings in any such setup, more for myself than for anyone else. So, here goes.
Over the last weeks, I have been experimenting with Maciej Eder and Jan Rybicki’s stylometry script for the R statistical framework. This is a script implementing a number of recent stylometric algorithms or statistical measures of distance for use in author attribution studies, such as Classic Delta, Argamon’s Delta, or Euclidian Distance. Also, it is capable of displaying results in several forms, as a consensus tree, a cluster analysis tree, or a principal component analysis matrix, whatever is most appropriate. It has a simple GUI that allows to set a wide range of parameters, especially what range of “most frequent words” should be taken into account, what the lanugage of the text is, and quite a few more. Note that this is not a package for R, but a script which can be started from within R and uses some of its capabilities. (The script’s authors will offer a workshop and present their latest work with the script at Digital Humanities 2012 in Hamburg next week; this is also a way for me to prepare for the workshop.)
So, in order to try out the script and have some comparison of my results with previous work, I decided to try and work with a collection of plays by Molière, Corneille and Racine, and some other authors from the same period of French “théâtre classique”. There has been some quite controversial work on this corpus recently, with one side claiming that some of the plays traditionally attributed to Molière have actually been written by Corneille (this side is represented by the website “L’Affaire Corneille-Molière“), and the other side vigorously defending Molière against such claims (this side also has a website called “Moliere auteur des oeuvres de Molière“). The controversy itself has led both sides into a discussion of a fascinating array of biographical and stylistical evidence, but I am not really interested in this here. (I hope to cover some of the background of the controversy in a later post.) In the context of work about Molière and Corneille, Racine is also studied.
So, first of all I found a very nice collection of texts on the website of Paul Fièvre, called “Théâtre classique“; it gives access to 458 French plays from the seventeenth and eighteenth centuries, and provides them in HTML and plain text versions; recently, they had the brilliant idea of also providing the TEI/XML versions they use to create the other formats. This last fact made preprocessing for stylometric analysis a whole lot easier; my colleague Katrin Betz helped me write a little XSL transformation to extract the speeches of characters from the files and hence discard things like prefaces, cast lists, scene and act divisions, and the names of the speakers (which, because they come up so frequently, may distort the results). This means that in my current implementation, the stage directions have been excluded; they would be easy to include, although I am not sure of how much this would change anything. It is important to save the files in UTF-8, so that the script’s tokenizer is not mislead by the French accents on some vowels. Once the files are preprocessed and placed in the appropriate folders, the script can be run from within R.
With my first series of runs, I wanted to see how much of an impact the choice of a given range of most frequent words (MFW) had on the attribution results, and what other results may come up. This is an interesting issue, because a lot of authorial attribution work is done on a varying range of several hundreds of the most frequent tokens, and because Eder and Rybicky found that for different languages, different ranges of MFW produce the most reliable results.
This was my setup: I included 12 plays by Racine in my corpus, and made the following basic settings: language = French; culling = 60%; algorithm = Classic Delta. My variable parameter would be the “most frequent words” (MFW), which I decided to go through in steps of 100 without overlap. Basically, I had two questions: (1) Which parameter settings concerning the MFW would have the strongest “distinguishing power”? This I assumed can be assessed in terms of the number of plays attached to each other by a common node to the main axis of the graph; the less of these common nodes there are, the more the plays spread out on the main axis of the graph, and the better the method distinguishes between the plays. (2) Will the parameters with the strongest distinguishing power also create a chronological sequence? If there are exceptions to the chronology, I was hoping to be able to explain these exception by other factors.
The results showed that there are indeed quite strong differences in the distinguishing powers of different settings. Using only the 1-100 MFW range (or the 1-50 and 50-100 range which I added) has the weakest results, with a lot of plays clustering around a common node. This changes quite drastically when one moves into the 100-200 and 200-300 range, with a much stronger spread and a clear main axis forming. In the lower part of the graph, the plays start to spread out nicely along the axis, but the upper part of the graph is still less distinguished, with three pairs of plays attached to a common node, although two of the pairs with their own additional branch. In the even higher ranges, the situation stabilizes somewhat; for 500-600, 600-700, 700-800, 800-900 MFW alike, the graph splits into three sections, each with a relatively distinct main axis. The two larger ranges 1-500 and 1-1000 behave similar to these higher ranges, but the best result in terms of distinguishing power seems to be the overarching 100-1000 MFW range. This is the graph I include here and comment on a bit more in the following paragraph.
The second result concerns the chronological ordering of the plays; the result is actually very impressive, to my own surprise: from the lower left end of the graph, we start with the two latest Racine plays (1689 & 1691), at ex aequo positions; then four plays from 1677, 1674, 1673 and 1672 spread out in order; then there are two plays from 1671 and 1670, at ex aequo positions, then we find a play from 1668 which is equidistant to the group of play from 1671, 1670 and 1672, in interchangeable ex aequo positions, but closer to the play from 1666; these are followed by the plays from 1664 and 1669, respectively, the two being equidistant from each other and the furthest away from the latest plays. Basically, there is a clear overall chronological order, with some chronologically very close plays at equidistance to each others and to each other’s closest neighbours, and one exception: the play “Les Plaideurs” of 1669 is not in the right sequence.
What’s wrong with “Les Plaideurs”, I wondered. I double-checked doing a similar analysis with another output format, the cluster analysis. Again, the groupings are similar, but this time “Les Plaideurs” was in an even stranger position, very close to the two latest religious Racine plays. I suspected something was wrong with my text, and a quick look into the file confirmed that, while the text was alright, I had by mistake classified it as a tragedy, when in fact it was a comedy! I knew genre had a strong influence on these stylometric measures of distance, but this was really striking. Taking the play out of my set, and repeating the whole procedure, produced precisely the same results, only without this strange outlier.
To conclude, I think two things are remarkable or astonishing about these results: first, the way the plays group nicely according to their date of creation; second, the robustness of the method, which gives very similar results in a relatively wide range of settings, and which is best with a large range of MFW; only the small ranges of the very most frequent words seem to be less reliable or helpful. However, one big caveat occured when I tried to repeat the exercise with a set of tragedies by Pierre Corneille: although the plays also spread out quite nicely when I used the 100-1000 MFW range, they spread out even better in the 400-500 and the 600-800 MFW ranges; also, I was unable to repeat the wonder of the chronological ordering with these Corneille plays. These two matters, the MFW range that best sets the texts apart, and the chronological ordering of texts, will certainly be issues I would like to bring up in the stylometry workshop with Maciej Eder and Jan Rybicki in Hamburg next week. But there is a lot more I would like to try out with this powerful script.