Niels Ott - Automatic Mind

Named Entity list from German Wikipedia

nospam@example.com (Niels Ott) — Wed, 3 Feb 2010 11:11:00 +0100

In 2008, I developed an anagram solver that is still presented online in some German posts of my other blog. It actually is an anagram solver but it comes as a riddle solver for a stupid type of TV quizzes. To cut a long story short: do never ever call these numbers on the screen. If you just want to know the solution, use the linked pages and turn off your TV right after.

So what's interesting about anagram solvers and these riddles? Well, one needs a fairly large word list. I took the one from the Ispell dictionary, the one I have online on the page of BananaSplit. I soon found out that these riddles made use of celebrity names a lot. Yet my word list back then contained only a few named entities. I solved this issue by Wikipedia harvesting. What do all humans have in common? Right, they are born at some point. And of course, everybody has to die. Now Wikipedia has these births and deaths lists in the entry for each year of the calendar. Back in May 2008, I downloaded all pages from year 1000 to year 2008 and simply extracted all the persons that had ever been born or that had died. This gave me a list of 51.353 person names from German Wikipedia.

Recently a colleague asked me if he could have this list. So why not simply putting it online? Here we go.

Person name list from Wikipedia, CSV (bz2 compressed, 0,55MB) – downloaded in May 2008.
Perl-Script used for extracting the named entities, in case you want to do this yourself.

I hope this is useful for some people out there. Feel free to let me know what you're using it for or what improvements could be made. You can use the comment box below or write me an e-mail.

Ärger mit der Örgele

nospam@example.com (Niels Ott) — Tue, 26 Jan 2010 20:51:00 +0100

Apologies to my English speaking readers for the rest of this article being written in German. It simply seems odd to formulate complaints about a southern German newspaper article dealing with a southern German dialect in English. Stay tuned, more general stuff will come in English as usual in the future.

»Aus Lorch ereilt uns die überraschende Nachricht, dass es ein schwäbisches Wort für den Bohrfutterschlüssel gibt. Es lautet Ärgele.« So beginnt eine weitere Ausgabe der Kolumne Schwäbisch auf Anfrage von Henning Petershagen, abgedruckt wie üblich im samstäglichen Magazin der Südwest Presse, genauer am 9. Januar 2010. Leider gibt es den diskussionswürdigen Beitrag beim Verlag nicht online, weswegen ich das Risiko eingegangen bin, eine Kopie zur Verfügung zu stellen. In diesem Post geht es auch nicht darum, die Holzmedien anzugreifen, vielmehr wurde beim Lesen der Kolumne mein Stefanowitsch-Reflex getriggert: Wenn sich Leute in der Öffentlichkeit über Sprache äußern und dabei Grundlagen der Sprachwissenschaft übersehen, dann muss das doch von jemandem kommentiert werden. Und da bietet sich ein Blog-Post doch geradezu an. Doch der Reihe nach…

Zunächst berichtet die Kolumne von einem Leser – einem Nichtschwaben – der das Wort Ärgele entdeckt habe. Nun will er wissen, wo es denn herkomme und was es damit auf sich habe. Es ist natürlich verständlich, dass ein Nichtschwabe sich damit nicht so gut auskennt, und letztendlich findet Petershagen die richtige Erklärung: Beim Ärgele handelt es sich um einen Diminutiv von Orgel. Er fischt herum und findet andere Dinge, die ein Örgele sind: Der Schlüssel, der bei älteren Schlittschuhen zum Festziehen der Bindung benutzt wurde. Letztendlich landet er bei der Drehorgel. Das mag ja alles richtig sein, doch was überhaupt nicht passt ist die Unterscheidung zwischen Ärgele und Örgele. Dabei ist die Erklärung recht simpel, wenn man sich mit Phonetik und Phonologie (zweites Semester Sprachwissenschaft in Tübingen) mal befasst hat.

Ärgele und Örgele sind exakt das gleiche Wort. Dass Örgele für einige Sprecher die zusätzliche Bedeutung eines Bohrfutterschlüssels hat, das ist eine andere Geschichte. Man muss hier beachten, wie es sich mit der Zuordnung von schriftlicher und gesprochener Sprache verhält. Dazu kommt das Phoneminventar: Das Inventar ist eine Beschreibung der Phoneme, die eine Sprache kennt. Und hier muss man eben wissen, dass die standarddeutschen Laute für Ü ([ʏ] bzw. [yː]) und Ö ([œ] bzw. [øː]) im Schwäbischen nicht vorhanden sind. Für jüngere Sprecher könnte sich das derzeit natürlich ändern, es gilt aber auf jeden Fall für die NORMs (non-mobile older rural males, ein Begriff von Chambers und Trudgill, gecoined in deren Werk Dialectology, Cambridge Univ. Press 1998).

Was der Schwabe nun macht, wenn er in der Schriftsprache auf ein Ö stösst, das hängt von der Umgebung ab. Der Einfachkeit halber lasse ich nun auch mal die Transkriptionen weg und denke mir eine standarddeutsche Aussprache. Generell wird der Schwabe ein Ö in der Schriftsprache wie ein E aussprechen. Zum Beispiel beim Ortsname Mössingen, dieser wird zu Messenga (wobei das -a hinten als kurzer, Schwa-ähnlicher Laut zu verstehen ist). Bekannter und für Nichtschwaben anfangs erstaunlich ist sicherlich das einfache Wort Ehl, bei dem es sich um ganz normales Öl handelt.

Wieso wird nun also das Örgele in den Ohren eines Nichtschwaben (s.o.) nicht zu einem Ergele sondern zu einem Ärgele? Die Antwort darauf: Assimilation, also die gegenseitige Beeinflussung von Lauten in einer bestimmten Umgebung. Das kann durch ein minimales Paar von jedem selbst beobachtet werden: Spricht man die Worte Ehre und Ehe laut aus, so zeigt sich, dass im ersten Fall das E zu einem Ä tendiert, weil es vom nachfolgenden R beeinflusst wird. Das neutrale H in Ehe hingegen lässt das E wie es ist. Das R im Schwäbischen kommt in verschiedenen Varianten vor, in manchen Gegenden sogar als rollendes R. Häufig ist jedoch ein pharyngalisiertes R, man könnte sagen, der Schwabe ›schluckt das R hinunter‹ – es wird ganz hinten ausgesprochen. Dadurch wird ein zum E gewordenes Schriftsprachen-Ö noch viel mehr zum Ä, und schon sind wir beim Ärgele. (Siehe auch: Markus Hiller, Regressive Pharyngalisierung in Stuttgarter Schwäbischen als C-V-Interaktion, Linguistische Berichte (155), 1995)

Das Ärgele bleibt dennoch das Örgele, denn es ist kein anderes Wort sondern nur die natürliche schwäbische Aussprache ein und desselben Begriffs. Zu dieser Folgerung kommt dann auch der Informant von Petershagen, dem er im Zitat den letzten Absatz dieser Folge der Kolumne überlässt. Die Ausführungen über Dinge zum Kurbeln, die Örgele heißen können, sind interessant, tun bei der Erklärung der Aussprache aber leider nichts zur Sache.

Bildnachweis: Darryl Coe, fotografiert von dpup, CC BY-NC-SA.

Hohenheimer Verständlichkeitsindex

nospam@example.com (Niels Ott) — Sat, 26 Sep 2009 18:37:00 +0200

Intro

Tomorrow we are going to have elections to the Bundestag here in Germany. Democracies come in all kinds of flavors. In the German flavor, the Bundestag is the parliament superordinate to all federal states (Bundesländer). The Bundestag elections are most important ones in Germany. Traditionally, parties are presenting manifestos. Wikipedia does not miss the point by claiming that »in recent decades the status of electoral manifestos has diminished somewhat due to a significant tendency for winning parties to, following the election, either ignore, indefinitely delay, or even outright reject manifesto policies which were popular with the public.« Manifestos are promises that are seldom kept. Even worse, they do not even seem to be written for a public audience. At least this is what the results of an investigation based on content analysis and readability measures suggest. In this blog post I am trying to present how researchers from Hohenheim University and a private company in Ulm are using readability to assess the understandability of manifestos.

Electoral Manifestos, Quantitatively…

So first of all, what were these results mentioned in the intro? Let me quickly summarize what the researchers around Professor Frank Brettschneider from Hohenheim University are presenting online together with CommunicationLab, a company from Ulm. The results the media are fascinated about are primarily the ones shown in this figure:

Electoral Manifestos of German parties as scored by the Hohenheim Comprehensibility Index.
Source: Ergebnis-Präsentation Wahlprogramm-Check 2009, transferred to English by Niels Ott.

They defined a Hohenheim Comprehensibility Index (Hohenheimer Verständlichkeitsindex). The figure shows how well the electoral manifestos of the major German parties perform on that index. These figures are the outcome of the quantitative analysis conducted by researchers. The inherent dream of all readability research lies in these results: one single and simple figure for scoring a text on a scale. It is somewhat unfortunate that all manifestos are rather hard to understand. Especially the one of Die Linke (the left-wing party) is quite close to a doctoral dissertation in political science, which is certainly not the appropriate level of difficulty for reaching masses. Interestingly, the winner in terms of this index, Die Grünen (The Greens), presents a manifesto that is too long to read (about 52,000 words) while the loser Die Linke comes up with the shortest one with about 20,000 words.

Conducting such an analysis rewarded the researchers with a rather high attention in the media, with regard to the fact that this is only a single study. Various online sites of renowned newspapers and blogs added their interpretations depending on political opinions and tastes. (A small selection in German: Welt Online, Zeit Online, Focus Online, TV report by ZDF)

… and Qualitatively

The Hohenheim people and those from CommunicationLab are well aware that there is more to text difficulty than counting the occurrences of traditional indicators. Having conducted a manual examination by experts, they claim that they can sustain the quantitative results. Briefly summarized, the manifestos have been written using too much technical language as well as using uncommon foreign words without explanation. Some criteria exceed the scope of pure readability, such as the one they call Wording seem to punish texts for inappropriate wording or the consideration of good and bad text layout.

Yet another Approach to approaching Language

Communication studies as an academic field looks at language from a different angle than (computational) linguistics does. Yet in the end, language remains language. Until a few weeks ago I did not even know of the existence of this field. Interestingly enough, the methodology of content analysis comes in a flavor called computational content analysis that uses some of the same technologies as computational linguists use and develop. CommunicationLab in Ulm therefore seems to be a spin-off company from this field that just accidentally deals with things I am also doing as a computational linguist, namely readability and automatic assessment of text difficulty.

Computational content analysis will be a further topic for me to explore in the future. In the past I encountered a number of ways of dealing with language. The purist computer scientist may say that information is a sequence of symbols from a defined alphabet. In information retrieval, there is not overwhelmingly much linguistics behind the processing. In computational linguistics, some people make things so linguistically precise that nothing works out in the end. Where on the scale between naivety and sophistication concerning language will content analysis turn out to be placed?

So how does the Hohenheimer Verständlichkeitsindex work?

As reported in personal communication with Jan Kercher from Hohenheim University, their formula is still work in progress. They are in a phase of tuning and testing it, aiming to publish scientific work about it. For now they only reveal the ingredients and the general procedure. There is no full recipe for baking that cake yet. So here we go with their list of variables used in the Hohenheimer Verständlichkeitsindex:

Readability measures:
- Amstad Formula
- Wiener Sachtext Formula
- Simple Measure of Gobbledygook (SMOG)
- Lesbarhetsindex (LIX)
Other parameters:
- Average sentence length
- Average word length
- Proportion of words with more than 6 characters
- Proportion of embedded sentences
- Proportion of sentences containing more than 20 words

Apart from the syntax-based measures (embedded sentences), these other parameters are similar to the basic ingredients in any readability measure published since Rudolf Flesch made readability popular in the 1940ies. The researchers compiled a corpus of hard texts using the abstracts of dissertations in political sciences. Furthermore, they compiled a corpus of easy texts from the politics pages of the German tabloid Bild. Every single variable listed above was then scaled to a value between 0 (very bad) and 10 (very good). The averages of these values for both the readability measures and the other parameters (see above) were computed. The sum of these two average figures makes the actual comprehensibility index.

Now what's the difference to all those readability formulas out there, including the ones which I mentioned in previous posts and which are computed in the Information Retrieval for Language Learning prototype? For me the main difference seems to be the fact that the Hohneheimer Verständlichkeitsindex is designed exclusively for the use with German texts about politics. It is likely that it will perform quite well if used solely with texts from that genre. I am looking forward to read their publications.

Outro

A new readability measure for German texts on politics is emerging from communication research. It will be interesting to see the final product and its impact. The creators of the new formula from Hohenheim University and CommunicationLab made a smart move in applying their methods of content analysis to political manifestos since this assures them a remarkable amount of public interest. The plain and simple scores produced by a readability measure do their stuff once more: there is a significant media response. While this may make others researchers jealous, only the upcoming publications will show what exactly they did and and how to evaluate it.

Text Difficulty and Information Retrieval

nospam@example.com (Niels Ott) — Thu, 20 Aug 2009 11:21:00 +0200

Intro

It has been a long pause here on Automatic Mind. After finishing my Master's project and Thesis, it took me some time to adjust to my new situation as a researcher here at Tübingen University. Meanwhile some things went on in the readability corner. The tool for computing readability formulas that I demonstrated as a Java applet in an earlier post is now freely available as a Java library–including the applet and a standalone demo GUI. Some bugs have been squashed and all formulas have been cross-checked with the corresponding original publications. In this post I will focus on what one can do with those readability formulas in information retrieval. This is a brief summary of topics from my MA Thesis entitled Information Retrieval for Language Learning: An Exploration of Text Difficulty Measures. The practical part of my thesis continues living as the Information Retrieval fo Language Learning (IR4LL) project which also features an online demo and web site.

Yet another Search Engine?

»There is Google, so what do you want?« Apart from the fact that it is cool and nerdy to be able to say that one has developed one's own search engine, there's other stuff to it. Google and its competitors are very good at retrieving web pages of interest. But they can of course not guarantee that the texts on the returned pages are easy to read. However, it is crucial to language learners and teachers gathering readings to have material that is at a certain level of difficulty. Otherwise students will either be bored or overstrained. It is hard to say whether or not Google could do this if they were interested. They usually focus on statistical methods and it is unlikely that they would do deep natural language processing– it is simply too expensive in terms of processing power. So here we go with a new search engine that limits itsself to only a few web sites containing promising readings at several levels. At least this is the plan for the future, the current prototype shows that this is possible. Everything is work in progress.

Readability Measures and Beyond

A brief roundup on readability measures in general and on my previous post: readability measures or formulas try to compute a single value stating how difficult a text is to read. Many formulas are supposed to yield numbers in the scale of U.S. grade levels. Most formulas use the average sentence length and the average word length as variables. Word length is measured in syllables or in characters. While the formulas look like they were containing a lot of magic numbers, it is in fact the case that their constants are usually designed to match the variables to texts with a known difficulty level. To illustrate this, here is the Flesch-Kincaid formula which is supposed to yield levels on the grade level scale:

Flesch-Kincaid = -15.59 + 11.8 × AWL_s + 0.39 × ASL
where
AWL_s is the average word length counted in syllabes and
ASL is the average sentence length counted in words.

To play with readability measures, check out the Java Webstart of the Phantom Demo GUI.

Unfortunately, the world of language is not as simple as that. Sentence length depends on the domain. It probably produces a stronger classification for novels vs. technical manuals than for easy vs. hard texts. Word length also reveals some flaws: the general assumption that long words are harder (or even rarer, if we agree with Zipf's The Psycho-biology of Language) is questioned by frequent long words such as beautiful or absolutely. While the current IR4LL implementation strongly focuses on readability measures, there are things to be explored beyond those.

Vocabulary and Grammar

For future research, there are two basic strands which I want to follow with respect to Information Retrieval for Language Learning (IR4LL):

measures of text difficulty based on vocabulary lists, and
syntax-based measures of text difficulty.

Thinking back to one's own foreign language classes in school, vocab drill is what everybody had to do. There are different styles of vocabulary learning, but there is no doubt that new words must be learned actively. Which also means that missing vocabulary makes a text harder to read. Word lists are relatively easy to deal with for computational linguists. However, the contents of those lists must be well-chosen, so this might turn out as a challenge. Furthermore, vocabulary is again domain-specific. If a learner is a fantasy literature nerd, he or she will probably be able to read a novel of that genre that even native speakers will have a hard time with.

Syntactic complexity seems to be one of the more promising things to look at. Simple phrases linked with and or commas are probably much easier to understand than deeply nested subordinate clauses. There are a couple of existing approaches which I'm planning to integrate into the IR4LL prototype. A positive side-effect of that will be the possibility to directly query for linguistic forms. One could query for texts containing a lot of gerunds, or a lot of simple past, and so on. Since most tenses are taught separately, this could reveal nice real life reading materials for classroom use. Furthermore, the WERTi system (an automatic intelligent workbook) interacts with IR4LL already in its latest prototype version. A syntax-aware IR4LL system could greatly improve the usability of WERTi by finding better-suited readings.

At the time of writing, much of these thoughts are left to future research, which I will hope we will be able to conduct here at the linguistics department of Tübingen University.

Combining Measures into Simpler Difficulty Levels

There is yet another challenge to IR4LL: most users will not be able to specify their language proficiency level in a detailed way. A self-assessment such as »yeah, I'm rather weak at future in the past, but I don't have any trouble with deeply structured sentences« are unlikely. Therefore, IR4LL aims to combine several measures (text difficulty, vocab-based, syntax-based) into single templates. I call these query models because they are used as part of the search engine query. It would be great to have query models that actually classify texts into a well-known scale such as CEF levels or to a stages in a foreign language teaching curriculum.

Outro

With not too much related work being out there, my MA thesis project contributes to the small field of IR4LL. The thesis discusses ways of measuring text difficulty and the implementation part provides an extensible framework with a running prototype. Future plans include the integration of a web crawler and the refinement of the text categorization into well-established difficulty levels. Once this will have been successfully approached, the search engine will be of great use to language teachers and learners. Will boring school book texts finally be a thing of the past? If we manage and if there is enough easy text on the web, they will.

Graphics taken from Open Clip Art Library, modified by Niels Ott.

CL Blogs and a New Name

nospam@example.com (Niels Ott) — Mon, 16 Feb 2009 16:05:00 +0100

Marveling at Jason Adam's collection of computational linguistics blogs, I noticed that CL Blog is a rather dull name for a blog. It somehow felt like naming a newspaper Newspaper. Back then I decided that this blog needs a new name. I just renamed it to Automatic Mind. The term actually is related to Dual Process Theory and refers to the fact that we can simultaneously walk and talk, or perform other tasks of which one is conscious and the other one subconscious. Then again, it also refers to computational linguistics. The human mind can process language. The computer maybe can – a little bit. What we need or dream of is an automatic mind.

Currently I am consumed by working on my Master's Thesis so I rarely find time to read blogs, let alone writing serious posts. Please stay subscribed.

Graphics taken from Open Clip Art Library, modified by Niels Ott.

Simple Readability Formulas And Boring Preprocessing

nospam@example.com (Niels Ott) — Fri, 23 Jan 2009 14:57:00 +0100

Intro

Readability formulas date back to the 1920s. They come in countless shapes and flavors, all sharing one common dream of their makers: to have a simple mathematical means of determining the reading difficulty of a given text. Is this text suitable as a reading for 4th-graders? Just stuff it into the formula and you will know which grade-level it fits. Of course, people put up warning sings telling the naive users out there what to do and what not to do with these formulas. But don't these formulas resemble the big dream of all natural language processing (NLP)? After all, all we want to have is something smart and simple that does the job of dealing with real world language. In this blog post, I will give a basic introduction on readability measures and I will point out in some detail that ›boring‹ preprocessing steps such as tokenization and sentence splitting are often underestimated. An interactive demo for computing readability scores is included.

Two Example Formulas

One of the most popular formulas is the Flesch Reading Ease, introduced by Rudolf Flesch in 1948 in his article A New Readability Yardstick. A reading ease of 90.0 to 100.0 is to indicate that the given text is very easy to read. A value in this range is achieved for comic books. The range of 0 to 30 indicates texts which are very hard to read. According to Flesch, such a value is achieved in scientific publications. The formula by Flesch looks relatively plain and simple, apart from some funny magic numbers:

Reading Ease = 206.835 - .846 × WL - 1.015 × SL
where
WL = the number of syllables per 100 words (word length)
SL = the average sentence length

The formulation per 100 words indicates a fact common to many early formulas: as computations had to be done manually, many authors advised users to work on small samples such as 100 words. Some variants of this advice suggest to take a small sample each from the beginning, middle, and the end of a text. Kincaid and colleagues later on adapted the formula to yield grade levels of U.S. education. The general idea and the linguistic analysis remained the same, only the magic numbers were adapted.

A less popular measure is the Läsbarhetsindex (Readability Index, LIX) introduced by Carl-Hugo Björnsson in 1968. ›Less popular‹ here means less popular in the English speaking part of the world. For Swedish and Danish, LIX seems to be widely used. It is unclear where the formula given in the the Swedish Wikipedia article is taken from. The German translation Lesbarkeit mit LIX (Readability with LIX) mentions several versions of one formula with differing magic numbers adapted for German and Swedish each. I need to do further research on this. The formula is commonly given as follows:

LIX = W / P + (L × 100) / W
where
P = number of periods in the text or sample (lazy version of number of sentences)
W = number of words in the text or sample
L = number of long words (more than 6 characters) in the text or sample

The interpretation of LIX values ranges from 25 (very easy) to 65 (very difficult). But what is more important is that LIX does not require syllable counting. Syllable counting was found to be tedious in 1968, and the addressed audience being people involved in language teaching probably did neither have access to computer machinery nor the knowledge to operate it. Nowadays, the problem is not computational power but the lack of accurate analysis. Part of this issue is discussed below.

Most readability formulas look like the ones above. Of course, there are formulas that include more intelligent analyses such as word frequency lists, or even syntactic analyses (sentence structure). As it seems, the base of most readability formulas does not stand on solid grounds. But then again, who cares about these assumptions as long as these formulas actually work? Most authors carefully restrict the validity if their formulas to a certain language and even certain text types and profiles of the readers addressed. For example, the FORCAST formula introduced 1973 by Caylor and colleagues is to be used only for U.S. Army technical documents that are read by young adult male readers.

Readability Demo

If you have Java installed on your system, you can get a feeling for a number of readability measures by using the little program below. Some ideas for input: Texts for children, Normal English Wikipedia articles vs. Simple English Wikipedia articles, abstracts of scientific papers. (Use the copy-paste keyboard shortcuts in the text area of the program.)

(Apparently you do not have the Java plugin available in your web browser. If you do have Java installed, you can still run the demo application using Java WebStart. Your browser might ask you what to do with the file: tell it to open the file with javaws.)

(This program is an old, probably incorrect version.)

The program actually is a preview on my readability library called Phantom Readability Library which will be available from the software projects section of my web page soon. Be aware that some measures might be incorrect. I am currently gathering the publications behind those formulas in order to check each of them for correctness.

Tokenization, Counting Sentences and Syllables

In the beginning of my readability adventure, I planned to simply use Java Fathom, a port of Perl's Lingua::EN::Fathom available from CPAN. I played with some texts and a story for children computed to a Flesch-Kincaid grade-level of above 13. Clearly, something must have gone wrong. After a short journey into the code I found out, that the sentence counter could not deal with direct speech as used in that story. As mentioned above, most formulas punish texts for having long sentences. Having fixed the issue, the grade-level now computes to 2.6 for the same text (in the current version of the program mentioned above).

A former student colleague kindly provided me with her version of a Perl-based syllable counter, which I then ported to Java. She reports over 96% correct results on a large test data set, which is fairly good for a rule-based approach that operates directly on English spelling. Having that tool available, I compared my results to those of other tools, mostly those being available online. They gave different results on measures with syllable counting. I found out that an average syllables-per-word ratio of 1.5 vs 1.2 (my program) affects the readability measures a lot.

So what's the message? What I am trying to say here is that the preprocessing accuracy does matter. The example with the sentence counting shows that it even matters a lot. And this is what happens to me all the time: after having implemented any analysis component beyond tokenization and sentence splitting, I always find out that due to some errors during preprocessing, all later steps fail to a large extend. Preprocessing is the most underestimated thing in NLP! People tend to think about it as a solved problem, but in fact and in practice, it is still one of the biggest challenges we face.

Sometimes, as with the Fathom module, it is just people not being imaginative enough. Having [a-zA-Z] as the only legal characters of a word reveals an anglocentric computer scientist's view on a world without any other language than English and without foreign words being used.

The Use and Abuse of Readability Formulas

So what are people doing with these formulas? First of all, there are offers of commercial tools with remarkable prices. That does not say these products are actually selling, though. William H. DuBay writes in his accessible overview The Principles of Readability that the average adult citizen of the U.S. reads at the 7th grade level. A text written at the 10th grade level will not be understood by 80% of the U.S. population. In genres where a broad audience is addressed, such as manuals, health care, or government information, checking the texts with readability formulas can give hints on where to improve the communication. Apart from that, publishers may want to decrease the reading difficulty of their books or newspapers in order to reach a larger number of customers. However, they might scare away readers with higher reading proficiency as reading a lot of text below one's level can be boring or even exhausting.

Language teachers could in theory use readability formulas to judge whether or not a text is suitable for their students.

One issue concerning readability formulas is that one must not ›write to the formula‹. Quite naturally, one would trick the formula by e.g. using shorter words or shorter sentences. Which leads to texts with shorter sentences that are not necessarily easier. Readability formulas simply do not work the other way round.

A common misconception may be that these formulas are supposed to be an exact means of judging the reading difficulty of a text. However, they do not produce much more than ballpark figures. Their exactness heavily depends on the suitability of the text for the formula in use. Furthermore, the implementations in computer programs may differ widely, depending on the quality of preprocessing as discussed above and the linguistic conceptions of their makers. Christian Watson discusses the differences in the results of several online tools in his Smiley Cat Web Design Blog. Last but not least, there is the possibility to write difficult text with easy words and easy sentences, simply because they deal with a topic hardly anyone is familiar with. This issue is partly addressed by formulas using vocabulary frequency lists… another story to write a blog post about.

Outro

Readability formulas are one of the things in computational linguistics that work well under certain conditions without requiring complete and complex analyses of language. However, these are rather shallow heuristics that may fail for a large number of texts. If they are used in the wrong context, they will almost certainly fail. What makes them attractive is their simplicity – which at the same time bares the danger of being blinded by its beauty, leading to the overuse or abuse of the formulas. From the computational perspective, these formulas are fragile because they depend on preprocessing analyses such as sentence splitting and syllable counting. The success of any analysis component stands and falls with preprocessing. We as CL people should not forget about that simple fact. The helpfulness of readability measures in computer-aided language learning (CALL) will be subject of further analysis and discussion in my master's thesis.

Graphics taken from Open Clip Art Library, modified by Niels Ott.

The USES Issue

nospam@example.com (Niels Ott) — Tue, 2 Dec 2008 17:31:00 +0100

Intro

It is hard to term the phenomenon without offending someone. Good names would be Scienceware, or Guruware, or even better Scientistware. They are all taken by companies or other institutions that presumably all do a way too good job to provide a name for a negative aspect. So let me call it USES for Unsustainable Software Emerging from Science. This blog post shall shed some light onto the issues of USES and onto possible reasons.

What USES is All About

As a computational linguist, I am working with specialized software each and every day. May it be part-of-speech taggers, tools to explore corpora or treebanks, or simply software development tools such as compilers, or even integrated development environments. But there is one type of software standing out: Software that emerged from Science (USES). Common features of this type of program include:

Usually developed for a very special and highly sophisticated purpose that is only understood within the field.
Developed by a single person who is a major expert within this field.
The major expert developing the software often is not a major expert in neither software design nor software architecture.
Examples of file formats are rare and there is little to no documentation about the software.
The software reacts unexpectedly on certain types of input, e.g. it ignores syntax mistakes in grammar files and then malfunctions without telling users why.
The software is often not completely finished and and includes some missing bridges at the end of some roads without any warning signs.
The only person who knows how to work with it is the major expert in the field who is not the major expert in writing usable software.

Since I am not intending to offend anybody, let me give an anonymous example. In a paper about parsing noun phrases with a certain parser, it is written that a single day was spent on writing the slightly over 100 grammar rules in use. A number of source code examples garnish the publication and it seems to describe a mighty piece of software that performs well and is easy to operate. But this opinion can change rapidly once one starts using the software. The grammar parser is not safe enough to deal with missing semicolons. Sometimes it notices them, reporting an error 20 or 30 lines before or after, some other times it just ignores the issue and interprets something the grammar writer did not intend at all – without saying so.

I am sorry to say that this behavior – which is only one example of bad application behaviour – is shared by a number of applications I have been using so far. In all cases, documentation was sparse and I spent a number of days or weeks on trial and error procedures.

The Lack of Documentation

Why is there a lack of documentation of USES? Here are my hypotheses: science as a system rewards publications, may they be books or – probably even more important for most authors – papers accepted at conferences or by journal boards. In papers, people report about the great insights the gained. Of course, these insights were gained by employing USES. Which is alright. However, there are three things that are not rewarded by this system: 1) the free availability of the described software to other researchers, 2) the free availability of the data required for the described experiments, such as corpora, grammars, or other computational resources, 3) the existence and free availability of reasonably good documentation for the described software.

Authors should be encouraged to consider these three points. If they are not fulfilled, other researchers can neither confirm nor refute the results published in the corresponding papers. Which is, I hope not only to my opinion, what a large part of the science business should be about.

The Lack of Quality

With quality I refer to the engineering part of software: it must be stable, usable and not too complicated to install and maintain. I am not referring to the actual purpose of the software. The LaTeX typesetting system is a good example: its output is regarded to be some of the most properly typeset books and papers out there – but most people writing books and papers might find its programming-alike user interface simply not usable at all. Imagine Microsoft Word being »user-friendly« in the same way: it simply would not sell.

But why is this so hard? One possible answer: it takes time. Plenty of time. Designing a graphical user interface (GUI) is said to take 60% of a project's time and therefore costs. Now USES usually does not include GUIs, but every programmer knows how tedious it is to catch all errors and produce intelligent error messages to the user. Again, meaningful error messages and a good user interface are not rewarded in the world of publication-based science. They steal researchers' time, and so they better do without them.

The Lack of Design and Architecture

There is an even more important type of quality. The quality of the engine under the hood. The properties of those parts the car driver does not even know that they exist:

Knowledge in software design is a skill that many programmers in scientific business must do without. Even worse, it is regarded as a superfluos overhead of work. Its absence explains the lack of what software designers call the -ilities, denoting properties of code such as reusability, scalability, manageability, reliability, sustainability, …, Features one typically finds in enterprise software. Features that are not honored by the system of publications. An important side-effect of software design is that it allows the coordination of software development in a team. Without software design, this can be quite hard, depending on the size of the project and the team. Without a team, software becomes as idiosyncratic as USES tends to be.

One step further from software design, one finds software architecture. It is usually pattern-based. A pattern sketches a common problem and its solution. One could see it as a template solution or recipie to a given problem. These patterns are documented well. Using them can easy communication among developers. If a comment in the source code of a program reads »Using the Observer-Pattern here«, everybody with knowledge about software architecture does not need any further explanation on what is going on. This can simplify the development in a team or the takeover by a new maintainer of the project.

The Lack of Completeness and Maintainance

Most USES is either incomplete or many years old. If it has been written in rather machine-oriented programming languages such as C or C++, it is often hard or impossible to get USES running on an up-to-date operating system. Why is this? So far I bombarded the system of publications with criticism. But there is another issue: science often works project-based. A project proposal is written, hopefully a grant is given, and then the project is worked on. At some point, the project is over. This usually happens way before the USES product has reached a state of completeness. Researchers are then forced to move on to other projects and the old program lies there somewhere on the Web server, becomes old and grows gray hair and is rendered unusable by time bringing changes to computer platforms and file formats. Bugs are detected by users but they are not documented on a central bug tracking system and as the development period is over, nobody will ever fix them.

Outro

In this blog post I have describe the issues of software written by scientists. This is not to offend programmers out there, but the problem must be addressed. Good quality software is likely to quicken interest in your work in other researchers and students. It is likely to improve the gain of knowledge in computational scientific disciplines in general as it enables real reviews. Furthermore, good quality software has the potential of supporting good teaching instead of leaving students sitting madly frustrated in computer rooms.

One question remains: how can we reward people in science avoiding USES?

Graphics taken from Open Clip Art Library, modified by Niels Ott.

Addenda

2008-12-05: Jochen Leidner pointed me to a readable article that discusses the same issue with a lot more analytic expertise. Read Empiricism is Not a Matter of Faith by Ted Pedersen.

Explaining Linguistics with Physics

nospam@example.com (Niels Ott) — Wed, 12 Nov 2008 10:07:00 +0100

Intro

Recently, I was asked by a student of languages what linguistics is. She was a student on the MA level, yet in the old German Magister system and her major subject was – as far as I recall – German, which includes some obligatory courses on linguistics at our University here. A simple question but not so the answer. I have been struggling for years now to find a short and easy to understand and not too wrong explanation of what computational linguistics is about, now how about linguistics? I tried it with a physics explanation which I would like to present for discussion here.

The World of Models

In physics as the common man may imagine it, people put up models. These models describe how phenomena work in nature and how one can predict the outcome of a process or experiment. However, they do not describe what these phenomena actually are. Take light. In physics, one can use particle theory to describe the behavior of light. Or wave theory. Which one to choose? This depends on the scenario. For predicting the diffraction behavior of a light beam, one would use wave theory. For explaining how solar cells work, one needs a description of the photovoltaic effect and therefore particle theory. This perspective of wave-particle duality can be extended to all matter and energy. Still, it does not state what light, matter, or energy really are. It helps physicians to describe (and predict) phenomena in nature. These descriptions will always be incomplete.

Linguists are doing a very similar thing. They are trying to find models that describe the human language. In morphology, they have a model that describes how to build plural word forms from base forms (lemmas). In syntax, there are formal descriptions that model the combination of words in a way that produces well-formed (grammatical) sentences. Yet no claim is made about what language actually is. And yet again, there are several models explaining different facets of a given phenomenon. And again, all models will be incomplete. Some researchers are working on the connection of their models and the processing of language in the human brain. Others do not care about this connection as long as their models describe a certain phenomenon correctly.

Outro

Now I am neither a physicist nor a real linguist. For the first, taking science as a major in German Gymnasium (secondary school) is not at all a qualification, and for the second the basic courses I took during my BA studies are only a little piece of a qualification. What do people think about the above idea of an explanation? One point I immediately came up with is that most people do not know enough about physics to make any connections, leaving them with more questions than they had before. Simplifying the physics model to, say, gravity and the dropping of Newtons apple would not bring in the plurality of models for explaining one and the same phenomenon.

Why should we be concerned with this explanation business anyways? Well, I think we need it. May it be to explain to our friends why we are doing these crazy things, or may it be to explain to sponsors why they should give is money for it. And while we discuss an explanation of linguistics that has mass appeal, others might take the challenge on finding a similar thing for computational linguistics.

Graphics taken from Open Clip Art Library, modified by Niels Ott.

Retrieving CL Publications Quickly

nospam@example.com (Niels Ott) — Wed, 15 Oct 2008 13:18:00 +0200

There are plenty of journals, the library catalogue is huge, and time is short. In the 90ies one would have thought about a meta search engine. Now in 2008 we have Google doing it for us. How often did you google the title of a paper you just found cited in another paper? I did so quite often and it never gave me the desired paper as such. Until I created own Google Custom Search, Publications in Computational Linguistics.

This search engine looks at only a small but important part of all the web. I tried to include sites that host scientific papers as PDF. If you feel that an important site is missing, let me know. If you are using iGoogle, you can install this widget to have the specific search box on your start page.

The intended use of Publications in Computational Linguistics is that the user enters the full title of the desired paper into the query box. Give it a try right away:

For those recognizing that this search engine is an old story that has now been updated translated to English: of course, you are right.

Concerning other blogging activities right here in this blog: there are some things going on in the internal drafts section. I am yet undecided how formal all of this should be and how the targeted audience may be like. So stay tuned via RSS, it may be worth it.

An Overhaul and a Brand New Start

nospam@example.com (Niels Ott) — Fri, 12 Sep 2008 20:06:07 +0200

Welcome to my completely overhauled webpage. If you have been here before, some of the contents will be still very familiar. The design is entirely new and still work in progress. But what about this new subtitle, »Me and Myself and CL«? I decided to split activities and content on the web in private and something like professional. As a result, this very webpage is concerned with me and myself as a computational linguist. And as a result of this result, there is a brand new weblog which I simply call »CL Blog«. You are reading this blog right now.

The new blog will deal with CL issues only. My old German blog will remain the blather dustbin for my private activities – CL excluded. Here, I am planning to write much less frequently than on the other blog but much more focused on CL topics, mostly from my experience as a student, part time student assistant in the field, and prospective scientist (hopefully).

Feel free to subscribe using the RSS links at the very bottom of the page. Stay tuned for more computational linguistics!