How good are our texts, really? Quality assurance for literary texts from various sources
by Ulrike Henny and Christof Schöch — this post originally appeared on the CLiGS blog.
What’s the problem, or: what is text quality?
The results: names, names, names, foreign words, …; also, genuine errors
The results for the Spanish-American texts were quite similar to the errors obtained for the French texts. Most of the words which were not recognized by the spellchecker were named entities. The abbreviation “Vd.” was classified as an error, so a list of acceptable abbreviations was added in addition to a text collection specific named entity list. Interestingly, some region specific and colloquial words stand out as a frequent error in single texts, e. g. “milico” (militiaman) in the novel El Chacho by Eduardo Gutiérrez. Further common errors in this collection are words with diminutive suffixes which are particularly widespread in Mexican texts.
Yet another important group of errors in this collection were historical spellings. Those are very source-specific errors. Texts from the Biblioteca Virtual Miguel de Cervantes, for example, have already been modernized. But in the case of other sources using first or early editions of the 19th century texts, the orthography has not always been updated. In addition to that, OCRed facsimile editions tend to have many historical spelling “errors”. Thus, this group of errors is very unevenly spread across the text collection, depending on where the texts come from and how they have been prepared.
Foreign words (French, Italian, English, Latin, …) occur, as well, but in most cases just as single or a few instances. To give a number, in the 24 Spanish-American novels which are part of the first textbox release, 5,197 single instance errors have been detected by the spellchecker (out of a total of 1,266,000 words, so 0.41%).
The consequences: corrections and transparency
- Alexander Geyken et al. discuss this issue for the “Deutsches Textarchiv”, where distributed proofreading is the strategy of choice. See Geyken et al., “TEI und Textkorpora: Fehlerklassifikation und Qualitätskontrolle vor, während und nach der Texterfassung im Deutschen Textarchiv”,