Author or genre? Assessing the quality of cluster analysis graphs in two-dimensional classification problems

One of the very fundamental issues in stylometric classification tasks is that the data under scrutiny is usually messy in some way. And I don’t even mean dirty OCR here, which is a problem anyone even casually playing with Google’s N-Gram Viewer and interested in pre-1800s texts will have noticed (Mark Liberman, among others, wrote an interesting piece on this). What I mean, rather, is that the text collections we would like to classify according to one criterion, say by their author, their genre, or their author’s gender, usually containt texts which vary by all of the categories just mentioned, and usually some more.

How are we to know whether our beautiful distance measures and algorithms actually classify the texts according to the category we would like them to, or rather, how can we make sure they will? More likely than anything, the distance measures use distinguishing cues that actually pertain to the various categories all at once, in a necessarily intransparent fashion (if we knew what features distinguished authors rather than genres, we wouldn’t need stylometric tools in the first place). A typical text collection which we submit to stylometric classification tasks may vary by genre (comedy or tragedy), form (prose or verse), time (seventeenth or eighteenth-centrury), theme (plenty, and messy in themselves) and of course author (you name them) and maybe also author’s gender.  Any one of these categories may influence the classification we hope will be according to just one category.

One way to go about this is to try and come up with text collections which are less messy. That is certainly a good idea in many cases, and common practice: work only on early nineteenth-century novels, or only on seventeenth-century comedies, or only on twentieth-century crime fiction (research on the latter topic, by the way, is seriously hampered by copyright issues). But what if the classification problem you are interested in does not allow you to work with a more unified corpus? What if, for instance, the number (and/or length) of available texts becomes too reduced to allow reliable distance calculations?

One path which promises a way out of this would be to find the right parameters that focus in on the category you are interested in. First of all, you could choose the linguistic phenomena accordingly (words, letters or parts-of-speech, single or in n-grams); you could decide on the degree of preprocessing you need (lemmatize your wordlist or not, apply culling to it or not, to name just two options); also, you could choose wisely which part of your frequency list to use (only the 20 most frequent features, or the entire list, or anything in between); finally, you could select an appropriate statistical comparison method (vector-based distance measuring or Principal Component Analysis, for example) as well as choose the algorithm which best suits your situation (Burrows’, Argamon’s, or Eder’s Delta, for example).

But on what basis are we to make those wise, appropriate, suitable decisions? This is something I have been complaining about before, of course. Over the last couple of days I have been investigation one of many of those fronts, and have run into an interesting problem. For testing and fine-tuning purposes, I used a text collection which is more or less homogeneous as to time of writing (all texts are from the second half of the seventeenth century) and to form (all texts are plays written in verse). The texts vary with regard to two categories, authorship (Thomas Corneille or Pierre Corneille) and to sub-genre (tragedy or comedy). This text collection consists of 54 plays, with 29 plays by Pierre Corneille (9 comedies, 20 tragedies) and 25 plays by Thomas Corneille (8 comedies, 17 tragedies). As usual, the texts come from the exellent “Théâtre classique” site by Paul Fièvre. The question at hand was to know at which settings (notably at which range of words from the word frequency list) the classification would be dominated by the author category, and at which settings it would be dominated by the genre category.

I did a pretty large number of classification runs using, as usual, the ever-improving “stylo” script by Jan Rybicky and Maciej Eder. Keeping all other settings stable (Classic Delta, Cluster Analysis), and using always only 100 words from the word frequency list, I varied each run by the onset on the word frequency list, going all the way from an onset at the top of the list to an onset at the 2000th word. (I did this for single words and word tri-grams as a first experiment). Each time, the script produced a wonderful dendrogram, and now I finally get to the real issue of this post: with all of these dendrograms on my hand, how can I assess whether the clustering each dendrogram presents is rather according to authorship or according to genre? To tell the truth, and some exceptions apart, the dendrograms are all kind of messy, but how messy exactly? Which category dominates the classification, even if only slightly?

Dendrogram visualizing a cluster analysis on a Thomas and Pierre Corneille corpus (snippet)

Above is a fragment from one such dendrogram which may be useful to illustrate the problem. The cluster analysis groups two plays whenever it can, or else adds single plays to a pair according to its distance; it then it groups the pairs according to the same principle, until it reaches the top node (to the left). It is easy to see that some clusters are nice, and form pairs of plays written by the same author and presenting the same genre (in the middle, for instance, Le Festin de Pierre et Le Galant doublé are both by Thomas Corneille, and both are comedies). In other cases, the two plays in a pair match according to genre, but not according to author (Le Feint astrologue and Le Menteur, at the top of the graph, are both comedies, but one by Thomas, one by Pierre Corneille). What about the plays which are not in pairs, like La Suite du Menteur (third from the top)? Has it been grouped with plays which have the same characteristics? To which surrounding plays should it be compared when deciding this? What about the larger context? The Suite du Menteur is a play by Pierre Corneille, and there is another Pierre Corneille play close by, but the larger cluster is dominated by Thomas Corneille plays? How can all this be summarized in some convenient proportion of “author-matches” and “genre-matches”, simple enought to allow comparision of many dendrograms yet subtle enough to capture most nuances of these dendrograms?

I tried out two solutions: one was to indentify all low-level pairs that present a correct matching of both author and gender, of author only, of genre only, or no correct matching at all, treating single plays as pairing with their next neighbour, and leaving larger clusters aside. I then counted these types of pairs for each dendrogram and drew up a chart comparing all dendrograms.

Because this did not seem to do justice to the slightly more complex cases when isolated plays need to be matched with mixed pairs, I then devised a calculation scheme using the same kind of information but giving two numeric values between 0 and 1 to each play, for gender and author matching, according to the degree of matching found inside the pair or with the closest pair. This is slightly more subtle, and somewhat simplifies the graph, but still does not account for the larger context of each pairing.

Results from cluster analysis-based dendrograms: types of low-level pairing as a function of wordlist onset

Anyway, the graph above, which is based on the initial method, does show a few interesting tendencies (tendencies which, however, would need to be confirmed using a better way of assessing the graph): First of all, there is very little genre-based pairing in the dendrograms with a wordlist onset of 0 to 750, with the exception of 375 (I wish I knew what is going on there). This suggest that one way to avoid genre-based classification, if that is not what you are looking for, is to limit the range of words used for classification to the first 850 words or, even better, to the first 350 words.

This result confirms an experience others have made: that genre, most likely because it is strongly related to themes, is more likely to show up in parts of the wordlist beyond the function words. However, it is good to know that for French plays from the seventeenth century, this is also true, and that the critical area of the word frequency list starts only at around 850 words. That goes beyond the pure function words, of course, and gives plenty of room and plenty of data for good distance measure calculations. If you do want to classify your texts by genre and not by author, however, this graph is bad news: the author always somehow shows up along with genre.

Now, if any of you have any comments, or ideas on how to create a more systematic approach to this issue, on how to establish better counts of the various types of pairings in the dendrograms, and better still, or on how to further automate the whole procedure, then don’t hesitate to drop me line in the comments.