Busy times, and some light at the horizon

Lot’s of things have been going on over these last weeks. Some of them were related to text analysis, others were rather related to job interviews, grant writing, teaching, real work, and travel, and I was just too busy to write anything here. Instead of a “real” post, this is just a way for me to cover the ground and catch up.

One not entirely unrelated thing was that, while on a trip to Hong Kong in February, I met Donald Sturgeon of the Chinese Text Project. We had some fancy iced tea in a little restaurant and Donald told me all about his project. Basically, as the introductory text will tell you, the Chinese Text Project which Donald has been taking care of for a few years now, presents “ancient Chinese texts, particularly those relating to Chinese philosophy, in a well-structured and properly cross-referenced manner, making the most of the electronic medium to aid in the study and understanding of these texts.” And indeed, there are some really cool features there, such as the parallel passages search and display module, which I may or may not write about in more detail some day very soon.

Another event related to text analysis was a brief introductory workshop I taught at Freiburg university in late February. It was basically a quick round through full text digitisation (OCR), stylometry, and topic modeling. The lesson I took home from the workshop was that powerful as stylometry and topic modeling may be, there are two caveats: first, a large part of the texts people would like to work on are still not freely available in reliable full text versions (well, I knew this from my crime fiction deceptions); second, there are plenty of research questions for which intelligent semantic search is just as important, or even more important, than advanced statistical and probabilistic clustering or modeling techniques. Just as important as our efforts are to make the most out of stylometry and topic modeling for the research questions we are interested in, are the efforts we should make to link up current computational methods of text analysis with more traditional research questions.

Also, an important event of the past few weeks was that my paper proposal for the Digital Humanities Conference 2013, in Lincoln, Nebraska, was accepted. Very cool! Last week, I finally found the time to make the graphs more readable and make the text itself clearer and possibly more accurate before uploading everything to the conference system. Now I’m really looking forward to go, and hope to meet as many text analysis enthusiasts there as I met last year at DH2012 in Hamburg.

Finally, following a one-day workshop Annelen Brunner of the IDS in Mannheim held for us at Würzburg University, I finally got started in relative earnest with machine learning. We were working with WEKA, which Annelen recommended and which seems to be quite flexible and powerful at the same time. So, while revising my DH2013 paper, I came up with a new idea for solving my old author vs. genre signal issue. If I use machine learning techniques to classify my training corpus according to some specific target class (something which is not possible with normal stylometry), and then use the information gain technique to investigate which of my features are most relevant for the target category in question, then it should be possible to extract a list of a few hundred words most relevant to each of my target classes (say, author and genre). Comparing the lists would allow me to assess how much overlap there is between the two target categories, and if the overlap is not too extensive, it should be possible to reuse those lists as wordlists for stylometric analyses and get clusterings for either one of the target categories. I tried this out in a purely proof-of-workflow manner the day before yesterday (technically speaking, it works), and as soon as I find the time to try this out in earnest, I will report on it here. I promise!