Unsupervised relative distances, or supervised clear-cut decisions: more work (or play) with Eder/Rybicky’s stylometry script

The biggest Digital Humanities event of the year is over: DH2012 in Hamburg. And one of the major topics seemed to be, at least from my perspective, stylometry. Several panels and some separate papers were directly concerned with stylometry, during the main conference. I’ll have to postpone more thorough comment on the ones I heard and liked best to another post; among them were those by Maciej Eder, David Hoover, Joseph Rudman, Jan Rybicky, Tomoji Tabata, Ted Underwood, and Joris van Zundert and Karina van Dalen. I missed the papers by Matthew Jockers and by Ana Lucic and Catherine Blake.

I won’t even try to say anything right now about the panels concerned more generally with text mining, of which there were also several, like the ones on “Computational models of narrative structure“, “Topic modeling the past” and “Text Analysis meets Text Encoding“, or papers like “The Swallow Flies Swiftly Through: An Analysis of Humanist“, by Geoffrey Rockwell and Stéfan Sinclair, “Intertextuality and Influence in the Age of Enlightenment: Sequence Alignment Applications for Humanities Research” by Glenn Roe, or “Trees of Texts – Models and methods for an updated theory of medieval text stemmatology” by Tara Andrews and Caroline Macé.

What I want to focus on in this post is a great workshop on stylometry in which I participated before the conference proper started. It was taught by Maciej Eder and Jan Rybicky, who acquainted us with some basic features of their computational stylistics for R script, and let us test the very latest, new and improved version of the script. If you haven’t used the script yourself, you may want to look at my related post about my own first steps with the previous version of the tool and at the links provided there. One intuition I formulated in my initial post was confirmed by Maciej and Jan: When the consensus tree is mostly circular, with a lot of texts clustering around a common node, this means that the script is having a hard time, with the settings you choose, to make a distinction between the various texts clustered in that way; when the consensus tree is more stretched out along a linear axis, the distinction of texts works much better.

What is new in version 0.4.5 of the script is that there are actually two scripts now: one that is designed for various distance measures to create graphs over large sets of texts with known or unknown authorship (this is an augmented and streamlined version of the previous script). This script (called “stylo” script) basically works in an unsupervised fashion and outputs graphs visualizing relative distances, the meaning of which the researcher needs to establish on his or her own, using knowledge about the texts or about their contexts. (Particularly, whether a text usually thought to be by author A but close to other texts by author B was actually written by author B or is just similar to author B’s texts for some other reason, something which can be a tricky business). This script now allows you to base calculations on words, letters, word n-grams and letter n-grams of the size you choose (or the size your system and/or the script is able to handle). It is not necessary anymore to use two separate directories for the texts, but you just use one “corpus” folder for the texts to be compared. Below are two images of the script outputting the results of a multidimensional scaling analysis on the now classical case of Molière vs. Corneille plays.

Multidimensional Scaling Analysis on Molière and Corneille plays, 200 MFW

At 200 MFW (above), the script does not differentiate the texts clearly, while at 600 MFW and beyond (700 MFW shown below), it does a very good, and increasingly god, job. But, which one is the best graph, assuming we don’t know the established thruth beforehand?

Multivariate Distance Analysis on Corneille and Molière plays, at 700 MFW

The second script is more specific, moving some features of the previous version of the script into a dedicated script (called “classify” script). It is designed for supervised methods involving some degree of machine learning, and basically suggests attributions of specific texts to specific authors. What you need here is three things: a training set of texts by known author, a testing set with texts by known authors in order to check how well the settings used on the training set work, and one or several texts with unknown or disputed authorship, for which the script will propose an attribution to one of the authors in the training set. (Of course, instead of authors the attribition test could also be to other categories, like author gender, time period, or genre, provided the question at hand is a “one-of-a-few” classification problem.) Instead of the graphical output representing relative distances, what you get in this case are a measure of how well the attribution seems to work on the test set (say, 12 out of 14 texts were attributed correctly), and the script’s “guess” at which of the authors is the author of the disputed or unknown text (or to which of the predfined categories the disputed texts belong to). Again, of course, you have quite a wide range of settings for what features should be used and what algorithms should be applied, to name just two obvious things. During the workshop, we tried this out on a set of British novels from the nineteenth century.

<p>Results of the classification problem on a set of English Novels</p>

Obviously, the output is much less spectacular, being just a short plain text file. And, the answer is clear (in this case: obviously, Eliot is the author of Middlemarch), and you can simply trust it or not. If you don’t really trust the tool, it probably makes sense to do the whole thing many times over, changing settings and parameters to see how they affect the results.

Of course, I immediately wanted to try this out on my favorite test-case, the plays by Molière and Corneille. To prepare this, I did another run with the two playwrights’ plays (settings: Eder’s Delta, a range of MFW of 200-500, culling 80%). This produced the following consensus tree visualisation.

Consensus tree visualisation of the Molière and Corneille plays

From this visualisation, it becomes clear that despite the fact that the script detects no actual overlap between the two playwrights’ works, there is no big distance between the four or five plays which are the closest together. In fact, the distance between “Illusion comique” on the one hand, and the “Menteur” and “Suite du Menteur” plays, i.e. between the three Corneille plays closest to the Molière plays, just as the distance between “Amphytrion” and “Dom Garcie”, i.e. the two Molière plays closest to the Corneille plays, are roughly just as big or small as the distance between the Corneille and the Molière clusters. The line separating the Molière plays from the Corneille ones is therefore not very clear-cut or uncontronversial, from a purely statistical point of view.

This means that this is a fine case for the supervised method of the second of Eder and Rybicky’s scripts. My setup was as follows: I put all of the Molière and Corneille plays in the training set, except for six relatively uncontested plays forming the testing set (three for each author), and except for the five plays mentioned above, closest to the point of separation between the two playwright’s works. I renamed these five files to mark them as of unknown authorship. Then, I ran the “classify” script on these plays, with the following settings: three iterations on 600, 800 and 1000 MFW, using Classic Delta, and asking for one candidate as the script’s answer. The script ran like a charm and produced the following results:

Results of the classification problem for closely-related Molière and Corneille plays

Over each of the three iterations, the script attributed the five plays as expected, that is the three Corneille plays to Corneille and the two Molière plays to Molière, and this with high confidence: all of the testing plays were also correctly attributed.

Does this make the case any more clear? Yes, if we trust the method; but honestly, I don’t understand the effect of the various settings and parameters well enough to make any bold statement yet; I will have to repeat this setup, varying the various parameters carefully, and check what happens. Also, I don’t understand the statistical mechanisms well enough yet to make any strong statements as to the more detailed meaning of the results. For example, I don’t know how to interpret the Delta scores for the five plays to be attributed. All the more reason to participate in the 9-day workshop on stylometry with Maciej and Jan, taking place next week at the European Summer School for Digital Humanities in Leipzig. I’m really looking forward to that, not just because the script is great and we will be able to really get into the details, but also because Jan and Maciej are serious about their topic and fun to work with at the same time. Trying out the script further certainly raised a lot of important questions for me, which I think will become clearer over the next week or so. Be sure to check for updates!