Dear fellow stylometrists, let’s drop the dendrogram and cherish the distance matrix

The dendrogram is a classic and beloved visualization in stylometry. One could even say that, ever since it was included in the inevitable and unmatched stylo package for R, it has become an icon and metonymy of stylometry itself. The example shown below (Figure 1) illustrates what such a dendrogram looks like, although given its ubiquity, there is almost no need to do so. Showing it, however, also serves to admit my own guilty indulgence in the practice of giving credence to the details of a stylometric dendrogram. Guilty, because I think we should drop the beloved dendrogram.  

Figure 1: Dendrogram from my 2014 paper on Corneille, Molière et les autres. (Click to enlarge.) 

If it is such a popular form of showing and interpreting the results of a stylometric analysis, from where then, you may legitimately ask, comes my temerity to suggest we should all drop it? Well, maybe not drop it entirely. But beware of the pitfalls and limitations it implies, and aware also of some alternatives that avoid said pitfalls and limitations. Of course, this is just a quick note and probably more provocation than methodological proposal, so take it with a grain of salt. Also, please note that two talks by Antonina Martynenko and Artjoms Šeļa at a recent workshop on stylometry held at Trier University inspired me to look more closely at the distance matrix again.   

To illustrate what I mean with the pitfalls and limitations of the dendrogram, here’s another one, this time based on an analysis of a subset of the English-language ELTeC collection that includes 10 sets of three novels by a different author each, for a total of 30 texts (Figure 2). The parameters for analysis almost don’t matter, because the results are quite robust, but for the record, the following parameters were used: word unigrams, 1500 most frequent features, Cosine Delta, no deletion of pronouns or culling applied. 

Figure 2: Dendrogram obtained using stylo for the authorship subset of ELTeC-eng. (Click to enlarge.) 

What this dendrogram shows is, first of all, that the method works with the reliability we have become accustomed to. All of the low-level clusters are made up entirely of novels by the same author. For a more detailed inspection, we can read off the distance between two particular novels from the dendrogram. If we project the meeting point of the Nesbit pair’s branches onto the horizontal axis, for instance, we can estimate that that score must be around 0.6 or so.

However, the two novels by Nesbit and Ouida that stick out a bit in their respective clusters, for instance, how similar are they to each other? Or which other novels do they resemble, if they are not quite as clearly clustered with the other two novels by the same author? The dendrogram makes it really difficult for us to clarify this question. Because it is the result of a clustering process that works on the basis of the distance matrix, which it forces into a reduced form that allows visualization as a dendrogram, a lot of information is either difficult to read off of the dendrogram, or is lost entirely. That’s reason number one for dropping the dendrogram.  

Only a look at the underlying distance matrix allows us to clarify these kinds of questions. There, we find the information we need, but it is somewhat cumbersome to go through all of these rows and columns of numbers. So, why not turn it into a heatmap and inspect the matrix visually? Also, while we’re at it, why not throw in a clustering algorithm on top of the heatmap to replace the alphabetic or random order of the novels by some meaningful structure. Python’s seaborn library has you covered (see Figure 3). But wait, you say, you’re introducing the clustering back into the equation after just disparaging it! True, but only to organize the distance matrix, without removing it. 

Figure 3: Distance matrix obtained from stylo for ELTeC-eng, visualized as a clustermap using seaborn. (Click to enlarge.) 

Here, we can recognize some of the clusters from the original dendrogram on the margins of the heatmap. We can also see a strong correlation between the clusters visible in the dendrogram and the squares of similar color forming alongside the diagonal in the heatmap. Depending on the clustering algorithm used, this can look more or less similar to the default ‘Ward’ algorithm used by stylo. A key difference of the clusters in the clustermap when compared to the dendrogram, however, is that the Nesbit novel that was only ‘sticking out’ a bit in the stylo dendrogram ended up in a different cluster altogether in the clustermap above (but for some reason, this didn’t happen to the other novels ‘sticking out’ just as much in the stylo dendrogram, e.g. by Yonge or Wells). Given that there are many possible ways to cluster a distance matrix, and that even implementations of the ‘Ward’ algorithm can vary, this is not surprising. It does point to the arbitrariness introduced by the clustering, however, which is another argument, to my mind, for distrust towards the dendrogram.  

Crucially, we can also get a much better sense of typical within-author (or within-cluster) distance scores in our dataset and compare those to typical between-author (or between-cluster) distance scores as well. For example, we see very clear clusters for Disraeli, Broughton and Yonge at the center, with rather low within-author scores in the range of 0.18 and 0.59. Dickens and Eliot, below, appear to be represented in this dataset with somewhat more diverse novels, with within-author scores in the range between 0.52 and 0.73. For Wells, scores go as high as 0.86. These latter scores would not be outlandish for between-author scores.  

Similarly, we can read off the distance scores between any two novels, including the Nesbit novel that was merged into a cluster of Trollope novels. Granted, stylo‘s Ward clustering algorithm did not make that particular mistake. But while the clustering makes them seem very similar here, the distance scores in the matrix are 0.88, 0.99 and 1.0. Only the 0.88 score comes close to the highest within-author scores we have seen above for Wells. And the distance to the two other Nesbit novels are 0.94 and 0.95, values that would indeed not be uncommon for between-author scores. 

Another thing we can do with the distance matrix, in order to get a better sense of typical intra-author and extra-author distances, is to plot the within-author and between-author distances (without the text by our ‘suspect’ author Nesbit, in this case; see Figure 4).

Figure 4: Dual density plot for score distributions of within-author and between-author pairs of texts. (Click to enlarge.) 

We can see that the modes of within-author and between-author distances are quite distinct (at 0.66 vs. 1.07), but also that the within-author scores are quite varied and that both distributions do overlap, with a small probability of encountering values of more than 1.0, even when our problematic author (Nesbit) is not included in the calculation. Typical values for within-author distances are 0.48–0.72 (inter-quartile range), while typical values for between-author distances are a more narrow 1.01–1.12 (inter-quartile range). This shows us that a distance score of 0.88 is indeed in between the typical scores of clear cases.

The degree of overlap between these two distributions tell us something about the difficulty of the dataset when it comes to authorship attribution. The more distinct they are, the easier the dataset should be. And we see that even in a relatively easy set like ours, where the clusters all look very clear and neat, there is some area of overlap. [Paragraph added Nov. 12, 2023.]

Be that as it may, even if we don’t throw the dendrogram over board entirely, let’s stop ignoring the distance matrix when interpreting stylometric results. It is as simple as checking the checkbox in stylo to save the distance matrix to disk and then opening it as a spreadsheet. In addition, the little Python script provided here (‘plot_distance-matrix-clustermap.py’) allows you to obtain the clustermap visualization of the distance matrix for a more visual inspection. You can do it! 

Citation suggestion: Christof Schöch, “Dear fellow stylometrists, let’s drop the dendrogram and cherish the distance matrix”, The Dragonfly’s Gaze, November 11, 2023. URL: https://dragonfly.hypotheses.org/1414.