Do Sentences in Novels Get Shorter over the Course of the Nineteenth Century?
The topic of this post is the evolution of average sentence length in literary texts and can be understood as a sequel, after a long time, to a post about sentence length in French literature, only this time the focus is on English-language fiction (with some forays into multiple European languages) and on a somewhat different time-frame (with a focus on the time period ca. 1840-1920).
The question: do sentences in novels gets shorter over the course of the nineteenth century?
The key question I would like to address concerns sentence length in fictional, narrative prose (primarily, but not exclusively novels) in English as well as in several other European languages. The specific claim that this post sets out to test is that the average sentence length that can be observed in fictional narrative texts decreases over the course of the 19th and the early 20th century. This hypothesis has been suggested to me by a colleague, who in turn found such an assumption formulated by German theorists of language and grammar in the nineteenth-century, in the context of a more general assumption of a gradual deterioration and simplification of languages since Antiquity. This colleague explained to me that prominent figures such as August Schleicher, Jakob Grimm or Franz Bopp) formulated such views, for example in Schleicher’s Die deutsche Sprache, first published in 1860, and that similar assumptions by others have been documented more recently in books like Konrad Koerner’s Practicing Linguistic Historiography or in the volume edited by Winfred Lehmann, A Reader in Nineteenth Century Historical Indo-European Linguistics. So, specifically and only with respect to sentence length, does the available data bear this out?
Note that there is of course a considerable amount of research on sentence length, as documented for example via a search on Semantic Scholar. In many cases, sentence length is used as a proxy for syntactic complexity. Some studies concern the issue of how to measure sentence length (e.g. Viera, Picoli & Mendes 2018, DOI: 10.1016/J.PHYSA.2018.04.104). I have opted for a simple ‘words per sentence’ definition, as the authors show that the six measures they tested all correlate strongly with each other. Several studies have tackled our question regarding the hypothesis of diminishing sentence length over time. In this context, Susan Conrad and Douglas Biber, for example, show that sentence length progressively grows shorter between the first half of the 18th century and the second half of the 20th century (Conrad & Biber 2009, 153). Their study, however, is based on very small samples (500 words) from just 17 novels spread out over over three centuries. Another study (Rudnicka 2018, DOI: 10.1075/scl.85.10rud) is highly relevant here because it uses a very large corpus that is different from the ones used here, the Corpus of Historical American English (COHA, DOI: 10.7910/DVN/8SRSYK, not freely available as far as I can see). Also, this colleague provides at least a partial explanation, at least for English, when relating the diminishing sentence length to evolving syntactic preferences. What I report on here is purely concerned with sentence length and does not delve into attempts to explain the observed trends.
The data: Project Gutenberg Corpus and European Literary Text Collection
The data that is readibly available to me comes from two main sources: The Project Gutenberg and the European Literary Text Collection (ELTeC).
Thanks to the Gutenberg Corpus project and code (see: Gerlach & Font-Clos 2018), the texts from Project Gutenberg are readibly available as a large corpus with limited but useful metadata. The Gutenberg Corpus code allows users to automatically download a current copy from Project Gutenberg, extract metadata and perform some very basic preprocessing of the texts as well. Doing this in November 2021 yielded a total of 63.208 texts. Based on the metadata table, I was able to filter out many items in order to keep just those that were textual by origin (not e.g. transcribed audio sources), that were written in English (whether translated or not) and were labeled as “fiction”. This yielded a total of 18.738 texts. (The list of texts in the filtered metadata list is somewhat longer than the list of files, probably because files skipped when downloading from Gutenberg.) From this subcorpus, which I will call Gutenberg Fiction from now on, a number of different, smaller samples were taken in the analysis steps.
The European Literary Text Collection (ELTeC) was created by the members of the COST Action Distant Reading for European Literary History. It is smaller in size than Gutenberg Fiction, of course, but has several advantages as well: Each complete, language-based collection in ELTeC contains 100 novels from the time period 1840-1920, with the entire period covered as equally as possible and with a wide variety of authors represented in each set. In addition, the metadata situation is considerably better than for the Gutenberg Corpus, but more on that below. Finally, several ELTeC corpora already come with linguistic annotation so that there is no need to perform the segmentation of the text into sentences using a proper method for each different language. As there is also an English-language ELTeC corpus, of course, this also provides a good reference to check the reliability of the method I used for determining the sentence length for the Gutenberg texts.
Preprocessing: Determining the date of first publication
The most important step in preparing the Gutenberg Fiction corpus for analysis was to come up with ways to determine the date of first publication for each text, a piece of information that is not included in the metadata after download. (This is not an issue for ELTeC, of course). For most texts, Gutenberg Fiction includes the year of birth and death of the author, however, so a very rough timeframe is known.
My first idea for retrieving publication years for the texts in Gutenberg Fiction was to look them up on Wikidata. This, however, turned out to be non-trivial, not only in terms of the required SPARQL-Query and issues related to timeouts. There is a Project Gutenberg ID property in Wikidata, but this is only used in a few rare cases and is basically useless at this point. As a consequence, querying needs to happen based on author names and titles which, however, are often longer than is useful for a catalogue search and need to allow for some degree of flexibility. After much experimenting, however, it turned out that in any case, coverage in Wikidata appears to be too weak to be useful at this point. (A clear avenue for improvement in the literary part of Wikidata!)
The second, and more successful strategy was to rely on Worldcat instead. Coverage is much better there and, although there is regrettably no API access for researchers, the catalogue can be searched automatically by generating appropriate search query URLs using the Python library requests and parsing out the results. The code for extracting keywords from titles that I wrote for the Wikidata tests came in handy and soon there was a lot of additional metadata on the probably year of first publication in the metadata table.
However, this data was of course imperfect as it stood. In some cases, my query would not find the right item in Worldcat. In others, the earliest edition that Worldcat knows of is not actually the first edition of the work. Finally, the first publication in English doesn’t need to be close to the first publication in the original language, of course, although that did not bother me much as I was going to analyse the English translation anyways. Finally, Worldcat has the strange habit of displaying some uncertain yers as e.g. “18–” in the catalogue, something that in the HTML source I extracted the years from was rendered as e.g. “1800”. This, of course, leads to issues with the assumed year of first publication. (Some of my plots from an earlier iteration showed larger clusters of texts around 1800 and 1900, certainly an artefact stemming from this issue that remains present, to some extent, despite some strategies I applied to alleviate this.)
In order to generate a probable year of first publication, the following heuristic was then used: If a year was found in Worldcat, and there are years of birth and death for the author, and if the Worldcat year lies in between the birth and death years of the author, then it is assumed that the Worldcat year is correct. If the Worldcat year lies outside the lifespan of the author, it is not used, but the middle year of the author’s lifespan instead. If only the year of death is known, and the Worldcat year lies after it, the year of death minus 20 years is assumed to be a suitable approximation of the year of publication. If only the year of birth is known, that year plus 20 years is used. If there is a Worldcat year and no other data, it is used. And if no year can be derived in this manner at all, then no year is given and the text is effectively excluded from any time-based sampling. Finally, some manual checking and corrections for authors with more than 3 texts published in one of the cluster years, that is 1800 and 1900, was performed. To sum it all up, this data is therefore far from perfect, but should nevertheless be useful in the aggregate. The dating issue does remain the most important caveat of this whole study, however, and obtaining reliable publication years for Project Gutenberg holdings would be high on my wishlist.
The method: use existing annotation or spaCy for sentence splitting and tokenization
Given these two datasets, how can the average sentence length per text be determined? In case of many ELTeC corpora, tokenization, lemmatization and part-of-speech tagging are already present, that is included in the XML-TEI compliant level 2-encoding of ELTeC. Whether there is explicit sentence segmentation using the
s element, or whether with an appropriate attribute value signalling sentence-boundary punctuation on the
w element, this is relatively easy to extract and count with confidence using XPath. As the level of granularity I need for this study does not extend beyond average sentence length per novel, it was sufficient to count all words in each novel and all sentences or sentence-boundary tokens to be able to calculate this value by a simple division (words / sentences).
For Gutenberg Fiction, no such annotation is provided (the tokenized version produced automatically when downloading the texts removes information relevant to sentence segmentation). However, the tasks of sentence splitting (to establish the number of sentences) and tokenization (to establish the number of words) can quite easily be performed using the spaCy library for Python. Comparing spaCy’s results on the plain text version of ELTeC-eng with the data derived from the level 2-encoded version confirmed that the two methods yield very similar results. This means that any data derived from the larger Gutenberg Fiction using spaCy should be comparable to results derived from ELTeC.
In order to see the distribution of values and the overall trend, the results for each dataset are visualized in several ways. The main visualization is a simple scatterplot that shows each novel with its (assumed) year of publication and its average sentence length. To this, a regression line is added (based on a third-order polynomial) to showcase the overall trend, including a visual indication of the margin of error. (In the Github repository, I also include an interactive scatterplot that makes exact values and some metadata available for those who would like to investigate results in detail, alon with, of course, the CSV file that has the numerical values for each text.)
In addition, in order to test whether there is indeed not just a decrease in values, but a statistically significant difference between the beginning and the end of the period covered by the corpus, I created a dual density distribution plot (using kernel density estimation in the Seaborn library) and performed a Mann-Whitney-U-Test (using the stats package in scipy) for the two extreme time periods. This is also shown here for each corpus or sample. The resulting plot also includes the median values for the two groups and the p-value from the test.
Note that all code, metadata and results are available in the relevant project repository on Github, see: https://github.com/christofs/sentence-length/ (DOI: 10.5281/zenodo.5783067). The ELTeC sets are included there as well, but for the Gutenberg fiction samples, only the metadata is provide due to repository size limitations of Github. The complete Gutenberg Fiction subset from which I have sampled here, along with the (somewhat) improved metadata, is available separately from Zenodo (DOI: 10.5281/zenodo.5783255).
Results: Average sentence lengths in various corpora and samples
There’s going to be a lot of plots now for samples from Gutenberg Fiction and for different ELTeC corpora, so bear with me.
ELTeC: English corpus (100 novels, 1840-1920)
First of all, the results from the analysis on ELTeC-eng. The dataset is rather small, with just 100 novels, but the texts are equally-distributed over the time period 1840-1920 and texts and metadata are highly reliable. What do we see?
Basically, the regression-line shows a relatively clear trend, from clearly over 25 words in 1850 to just slightly over 20 words in 1910, albeit with a rather wide margin of error, due to the limited amount of data points and the considerable variation in values.
In the case of ELTeC-eng, 20 texts were sampled from each of the two periods 1840-1859 and 1900-1919. The resulting plot clearly indicates a drop (from 23.6 to 18.9 words on average) and a statistically significant difference (p<0.0008).
Gutenberg Fiction, small sample (100 texts, 1840-1920; sample1)
This small sample from Gutenberg fiction is meant to replicate ELTeC-eng, so includes the same number of texts for the same time period. The values are a bit different, but the overall trend is the same or, if anything, slightly more pronounced.
This is borne out by the comparison plot.
Gutenberg Fiction, stratified sample (1157 texts, 1820-1940; sample2b)
This shows results for a second sample from Gutenberg, this time quite a bit larger at 1157 texts and with a somewhat wider temporal range, from 1820 to 1940 (beyond these limits, there are just not that many texts anymore). The particularity of this dataset is that the number of texts included from each decade is the same. This keeps things balanced, but of course in a somewhat artificial way. [Edit, Dec. 16, 2021: The time range for this sample was extended to 1820-1940, as advertised in the title.]
Again, the drop is clear and confirmed by the comparison plot and significance test. Note the clear difference in the shape of the two distributions, with the earlier phase showing a much larger variation in values than the later phase.
Gutenberg Fiction, large sample (4100 texts, 1820-1940; sample3)
Because we have a lot of data, let’s use it! This is a larger sample again, with more than 4000 texts from the period 1820-1940, but without the stratified sampling by decade. The temporal range remains wide and there are more data points, but it is clearly visible that there is a considerably diminished density before 1840 and even 1860 as well as, to some extent, after 1920. Most Gutenberg Fiction texts were published between 1875 and 1925.
The difference, when looking at 1830-1849 vs. 1910-1929 is very pronounced and statistically higly significant.
ELTeC-deu (German, 100 novels, 1840-1920)
English is just one among many languages, and ELTeC provides us corpora in many more languages, so let’s have a look at German. Same trend, at least for the period 1850-1900, with a slight upward bend towards the end.
Clear and statistically significant difference.
ELTeC-hun (Hungarian, 100 novels, 1840-1920)
Very similar results for Hungarian.
The difference is statistically significant, despite the rather wide margin above and the limited number of texts.
ELTeC-fra (French, 100 novels, 1840-1920)
Somewhat surprisingly, the consistent downward trend that was apparent in every sample or collection up to know is not present in the French corpus.
Despite the slight downward movement that is visible, there remains considerable overlap between the two distributions and their difference is not significant, at least with this small sample (p=0.17):
This can be brief:
- Yes, sentences do get shorter, overall, in English, German and Hungarian literature in the time period 1840-1920. (From about 25 words per sentence on average in around 1840-1850 to around or just below 20 words per sentence around 1910-1920).
- The same trend cannot, so far, be confirmed for French.
- All code, metadata, results is available on Github.
- Long live openly available data (thanks, Project Gutenberg!).
- We need more data for the time before ca. 1820 as well as open (e.g. as derived features, including punctuation) datasets for the time after 1920.
- We need better metadata! (What I produced with my crude heuristics is available as a starting point.)
Ideas for follow-ups [added after Dec. 15, 2021]
- Oleg Sobchuk suggested on Twitter that the increasing proportion of direct speech might be an explanation for the observed trend. Definitely worth checking by bringing together direct speech recognition and sentence length.
- A comparison with non-fiction would of course be interesting as well, where proportion of direct speech (citations in newstexts?) may or may not be an explanation for any downward trend.
- The French results suggest checking a chronologically larger window, which would of course be equally interesting for other languages. For French, I’ll certainly do a follow-up analysis that will combine several datasets for the period 1750 to 2000.
Christof Schöch (2021): “Do Sentences in Novels Get Shorter over the Course of the 19th-Century?” In: The Dragonfly’s Gaze. Marseille: Open Edition. URL: https://dragonfly.hypotheses.org/1152.