Showing posts with label Wordle. Show all posts
Showing posts with label Wordle. Show all posts

2010-02-06

Comparison, TLG BC and AD: log-likelihood

Helma Dik left a comment on my post on comparing TLG AD and BC through Wordle, suggesting I use Dunning's Log-Likelihood measure of differential word frequencies in corpora, as Wordled by Martin Mueller. That lets you work out what the real shifts in frequency are, rather than trying to eyeball them through the aggregate word counts.

Here for instance is his comparison of the Iliad to the Odyssey—which words are more frequent in the one, or the other:
Wordle: Odyssey_plusCorrected
Wordle: Iliad_plus
I looked up Ted Dunning's paper, failed to understand it :-( , and used instead the walkthrough of the computation on the user manual of the Wordhoard corpus software package.

And this is the more statistically sound Wordle comparison. Words more frequent BC are in red, words more frequent AD are in black. I'm leaving in stop words this time, and not cleaning up the ambiguity, because this says some interesting things about the changes in Greek grammar between Classical and Late Greek. Do click:
Wordle: TLG AD vs BC comparison, using Dunning's Log-Likelihood metric

Here's my impressionistic notes, that haven't already been covered in the previous post (where I was working through rankings):
  • Both corpora talk about θεός God, but the big jump, of course, is Χριστός Christ. The second biggest jump is in ἅγιος holy, displacing ἱερός. (Was ἱερός too pagan-sounding?)
  • But the biggest discrepancy between BC and AD Greek is the avoidance of δέ but, on the other hand, followed by avoidance of μέν on the one hand. That tells you that AD Greek used different sentence structures, such as a lot more ἀλλά but. Tucked away, there's also more καί and (i.e. more coordinating constructions) and a lot less τε and (a very archaic phrase-second construction).
  • There are a lot more ἤγουν and τουτέστι that is, and a lot less ἐάν if and ἄρα therefore; I'm tempted to think that says something about changing rhetoric in the genres popular in the respective periods—less logic, more exemplification. It's foolhardy, but not impossible.
  • There is a lot more τίς who? being reported, and that's an error in ambiguity, but it's an illuminating error. τοῦ in Attic (though not Late Greek) is ambiguous between "whose?", and the genitive definite article. And there are a lot more definite articles in Late Greek, as you can see by the black ὁ. (My friend Io Manolessou actually wrote her PhD on that shift; nice to see it visually confirmed.)
  • There's also more ἵνα in order to, which suggests Late Greek was already moving towards more subjunctive constructions rather than participles and infinitives, even before Early Modern Greek made the switch completely.
  • Clearly less ὦ Ο!—A very Classical way of addressing people.
  • Some of the odder looking words more prevalent in BC Greek are there because there are a lot more geometric texts in the BC corpus: Ἄβ is actually mistakenly picking up the line ΑΒ, and you can also see in smaller print ΑΒΓ, ΒΔ, ΓΔ, ΕΖ, ΞΖ.

Hm. Yes, that was somewhat more illuminating. Thanks, Helma!
...Read more

2010-02-01

Comparison, TLG BC and AD

In the previous post, I used Wordle to illustrate stop words in Greek (and, by the by, the exponential distribution of function words following Zipf's Law). After getting rid of a whole bunch of stop words, I ended up with a Wordle of the lemmata of the TLG:

But I stopped short of making sense of the Wordle, because the TLG contains both Ancient and Mediaeval texts, and they talk about different things. I promised Wordles of the texts in the TLG from BC and AD, which will give at least a rough sense of the difference.

So here they are:





Images created by the Wordle.net web application are licensed under a Creative Commons Attribution 3.0 United States License.


The Wordle images are hyperlinked to the Wordle applets hosted there, so you can play with the applets by eliminating words. The stopwords are as before, but I also got rid of πολύς "much", which was crowding the BC texts a bit much.

A few things jump out quickly: there's a lot more God AD, as you'd expect (θεός), slightly more talk of "people" than of "men" (ἄνθρωπος, ἀνήρ), less talk of the City and more talk of power (πόλις, δύναμις).

But I'm not really a visual person, so I'm going to use more quantitative ways of working out the changes in vocabulary.

To begin with, the two Wordles show the 150 most frequent lemmata for each period, not counting stop words. These are the differences between the two—words in the top 150 of one period, but not the other.
Ancients talked more about...and less about...
Ἕλλην, Ἀθηναῖος, Ζεύς, ἀμφότερος, διαφέρω, ἑκάτερος, ἐλάσσων, εἶμι, εὐ, ἤλιος, ἡγέομαι, ἱερός, κεῖμαι, κύκλος, ναῦς, νέος, νομίζω, ὀρθός, οἶκος, πλέως, πλεῖστος, πλῆθος, πόλεμος, πολέμιος, ποταμός, θάλασσα, θεά, σημεῖον, ταχύς, ὔστερος, χρῆμα, χώρα, ζῷον Χριστός, ἅγιος, ἁπλόος, ἄξιος, ἀδελφός, ἀλήθεια, βασιλεία, δέχομαι, δηλόω, δόξα, ἐκκλησία, ἐνέργεια, εἶδος, φωνή, κίνησις, κόσμος, νόος, οἰκεῖος, οὐρανός, οὐσία, πάθος, πίστις, πνεῦμα, πρόσωπον, θάνατος, θεῖος, σάρξ, τέλος, τρίτος, χάρις, ζητέω, ζωή
Greek, Athenian, Zeus, both, to differ, either, less, go, good, sun, to lead, dawn, holy, to lie, circle, ship, new, to think, right, house, full, most, crowd, war, enemy, river, sea, goddess, point, fast, last, need, land, animal Christ, holy, simple, worthy, brother, truth, kingdom, to accept, to declare, glory, church, activity, form, voice, movement, world, mind, own, heaven, substance, passion, faith, spirit, face, death, divine, flesh, end, third, grace, to ask, life

The effect of Christianity on vocabulary use is pretty obvious. A few other changes are worth noting:
  • Byzantines nominalised a lot more than Ancients did. That's at last some of the reason for ἀλήθεια "truth" (instead of the more Attic τὸ ἀληθές "the true"), and it may relate to other nominalisations like κίνησις "movement" and ἐνέργεια "activity". (βασιλεία "kingdom" has a Biblical pedigree—but that is also because the Bible was not written in Attic.)
  • Many of the differences are a matter of language change, rather than different ideology. For all that most Byzantines did not write in the vernacular, their language was usually more akin to Koine than to Attic. That explains the absence of εἶμι, εὐ, ναῦς, πλέως, ἐλάσσων, ἱερός, πολέμιος (replaced by στέλλω, καλός, πλοῖον, πλήρης, μικρότερος/ὀλιγότερος, ἄγιος, ἐχθρός) "send, good, ship, full, less, holy, enemy", and presumably also the avoidance of ἀμφότερος and ἑκάτερος "both, either".


I've left out from those lists words that show up in the top 150 only because they're ambiguous with other legitimate words. (Yes, I should have pruned the Wordles.)
  • BC: δίκαιον, δοκεύς, ἠώς, θέα: rights, beam, dawn, view
  • AD: ἅγιον, βασίλειος, ἴδιον, κενόω, πρόσωπος, ζωός: sanctuary, royal, particularity, make void, face, alive

There's one further comparison I'll attempt: the words whose frequency changed the most between the two periods. To track this, I'm going to use the 2000 most frequent lemmata for each period—including both normal words and stop words; that constraint means we're only looking at words that are likely to matter. I'll go through the lemmata in those lists whose ranking changed by the greatest amount (e.g. from #1537 to #10342).

Because it's a pretty heterogeneous list—and different kinds of words tells us different things, I'll split them up into categories. (And I will do some silent suppressing of ill-recognised ambiguous words.)

These are the biggest shifts in proper names:
Ancients talked more about...Rank Shift
ἜφοροςEphorus-8530
ΠοσειδώνιοςPosidonus-8397
ΠελοποννήσιοςPeloponnesian-6655
ΑἰτωλόςAetolian-5399
ἙκαταῖοςHecataeus-5157
ΘεόπομποςTheopomus-5046
ἈπολλόδωροςApollodorus-4948
ΦωκεύςPhocian-4786
ΤυρρηνικόςTyrrhenian-4587
ΧρύσιπποςChrysippus-4043

Two things are going on here. First, some ancient authorities—primarily historians, if I read the names right—were of interest to several ancient writers, but of less interest to the Byzantines. They tend to be the historians whose texts didn't survive, which is related to them being of less interest to the Byzantines. (I don't know offhand whether that's cause or effect.)

Second, Greece was very important to Ancient Greeks, and so were the various regions of Greece. To the Byzantines though, Greece was a backwater, and the old regions did not survive into the Byzantine system of themes. So there was no reason to talk about Aetolia or Phocia outside of Ancient History; and less reason to talk about the Peloponnese than you might think, even while the name survived. The same goes for Tyrrhenians: it wasn't Etruscans that the Byzantines were having to deal with in Italy, but Lombards.

Ancients talked less about...Rank Shift
ΚύριλλοςCyril+214,509
ΚωνσταντινούπολιςConstantinople+214,399
ΓρηγόριοςGregory+214,391
ἈθανάσιοςAthanasius+214,154
ΓεώργιοςGeorge+85,856
ΚωνσταντῖνοςConstantine+47,064
ΠέτροςPeter+40,947
ΧριστιανόςChristian+36,162
ΒασίλειοςBasil+28,217
ΧριστόςChrist+23,988

The only surprise is that Christians turn in BC texts at all; there's only 5 instances though, and the dating of texts in the corpus is porous (late citations can appear as testimonia of earlier authors).

These are the biggest shifts in common nominals:
Ancients talked more about...Rank Shift
εὔδοξοςreputable-8805
κύλινδροςcylinder-6569
ἀσύμμετροςasymmetrical-5939
δημοκρατίαdemocracy-5389
πυραμίςpyramid-4714
ναυμαχίαsea battle-4274
κῶνοςcone-4205
παραλληλόγραμμοςparallelogram-3837
παρεμβολήinterpolation; encampment-3668
ψήφισμαdecree passed by vote-3194

If the AD texts have more theology, they clearly have a lot less geometry, and a lot less to do with representational systems of government. The drop in εὔδοξος is surprising, given it's in Plato; I wonder if the change of -δοξ- in compound from "reputations" to "glory" made the adjective confusing for later writers.

Ancients talked less about...Rank Shift
ἀποστολικόςapostolic+214,282
θεοτόκοςGod-bearing (Theotokos)+85,966
βάπτισμαbaptism+85,945
θεότηςdivinity+59,016
μόδιοςbushel+58,616
μοναστήριονmonastery+57,696
σεβάσμιοςreverend+57,691
αἱρετικόςheretic+35,602
χάρισμα(spiritual) gift+35,588
πατριάρχηςpatriarch+27,001

No surprises again; the only non-religious term is μόδιος "bushel", both as a vessel and a measure.

These are the biggest shifts in verbs:
Ancients talked more about...Rank Shift
διαπορεύωpass across-5939
βλώσκωgo-5710
εἰσοράωlook upon-4134
ἄημιblow (wind)-4113
κλύωhear-3392
ἐπιζεύγνυμιjoin to-3039
ἀμφισβητέωdoubt-2436
ἐφάπτωhang on-2122
ἱκνέομαιcome-2088
μεταπέμπωsend for-1974

Many of the missing verbs are poetic and/or dialectal, and would not have a natural place in Byzantine prose; that includes βλώσκω, εἰσοράω, ἄημι, κλύω, ἱκνέομαι. The surprise here is the vanishing of doubt in the Middle Ages.

... Yes, yes, the jokes just write themselves, I know...

Ancients talked less about...Rank Shift
ἐνάγωpersuade+10,366
βαπτίζωbaptise+8529
ψάλλωchant+5021
φανερόωreveal+3988
καταδικάζωcondemn+3483
φωτίζωilluminate+3308
περισπάωtake a circumflex+2948
βαστάζωcarry+2911
ἀνέρχομαιgo up+2809
προλαμβάνωanticipate+2769

I admit to being less sure about some the shifts here, such as ἐνάγω and προλαμβάνω. The Christian influence is clear in βαπτίζω, ψάλλω, φανερόω and φωτίζω. Language change accounts for βαστάζω and ἀνέρχομαι replacing φέρω and ἄνειμι, and I assume καταδικάζω for "condemn" replaced what came to look like more generic verbs, in καθαιρέω or καταγιγνώσκω. And unlike the Ancients, the Byzantines had to learn about polytonic orthography; so what word took a circumflex and what word took an acute was a matter much ink was spilled about.

Finally, these are the biggest shifts in function words:
Ancients talked more about...Rank Shift
τοτέat times-5796
αὖτεagain-4707
δισχίλιοιtwo thousand-4676
αἴalas-3334
πεντακόσιοιfive hundred-2859
ἠέor-2844
νή[I swear] by [deity]-2663
διακόσιοιtwo hundred-2597
μάyea-2470
πωyet, at all-2466

There is some Epic dialect here, in αὖτε and ἠέ; some strictly Attic rather than Koine words in τοτέ, πω, and δισχίλιοι; and a rather different approach to exclamations, with the old oaths by the Gods dispensed with, and the ai!'s of tragedy avoided in theological discourse. (There are 2100 instances AD of φεῦ "alas"; maybe αἴ was too specific to tragedy? *shrug*) Not sure why the written-out 500 and 200 were less popular. Maybe the armies just got bigger, so historians talked in the thousands instead of 300...

Ancients talked less about...Rank Shift
ἀμήνamen+19,195
νάto (Modern Greek)+18,984
ἀλλαχοῦelsewhere+7689
δηλαδήthat is+7587
ἤγουνthat is+6367
ιζ΄XVII+4541
καθόinsofar as+4524
ιϛʹXVI+4001
ιηʹXVIII+3727
ιδʹXIV+3196

It's obvious why amen is there; it's also obvious why να, the Modern Greek equivalent of the ancient infinitive inflection, is there. ἀλλαχοῦ for "elsewhere" is attested in Sophocles and Xenophon, but it became prevalent much later, and LSJ reports that Moeris proscribed it as vernacular, in favour of ἄλλοθι. The other conjunctions are run-in phrases, which Byzantine texts in general are rather more sympathetic to treating as single words than are ancient texts: δῆλα δή "so [they are] obvious", ἤ γε οὖν "or indeed then", καθ’ ὅ "according to what".

Finally, the numerals aren't there because the Byzantines were more numerate than the Ancients. After all, the Byzantines had given up on geometry, from what the counts tell us. (And that's a silly enough thing to conclude that you should not take much of this too seriously.) No, the reason there's a whole lot of XVII's and XIV's in the AD corpus is that there are a lot more chapter headings in the theologians...
...Read more

2010-01-31

Wordle and Greek stop words

Some of you may be familiar with Wordle, an online tool which displays the words in a text with different sizes, depending on their frequency. Wordle is a convenient tool for seeing what the frequently mentioned concepts are in a text, so it gets a fair amount of use in blogs. It's the same concept as Word Clouds; but done with much more typographical finesse. This, for instance, is Wordle run over the English text of Plato's Republic:


And courtesy of The Crazy Australian, this is the ESV New Testament:

(As The Crazy Australian noted, you can learn one thing immediately from that: the Third Person of the Trinity doesn't get as much stage presence as the Other Two in Holy Writ. Not really a surprise, but the point of Wordle is as much to visualise the obvious as it is to discover the not as obvious.)

Wordle works quite well with English, because most words don't have a lot of inflection, to multiply the instances of the concept you're looking for. In a language like Greek, on the other hand, lemmatisation—or as it's more often called in search engines, stemming—is essential. Otherwise, you get not one instance of "Jesus" or "state", but four or five, with no material difference.

Funnily enough, I do lemmatising. So what happens when you put the TLG through Wordle?


Images created by the Wordle.net web application are licensed under a Creative Commons Attribution 3.0 United States License.


Well, what you get is this:

I've highlighted the top seven verbs in green, and the top seven nouns in green. You can see the nouns, right?

Of course you can't, because there's a whopping great big ὁ and another rather outsize καί there, crowding everything else out. And being told that Greek texts have a whole lot of instances of the and and is unlikely to be what most people are curious to know.

What we have here is the notion of stop words: grammatical words that don't convey a lot of content, and which search engines traditionally ignore. Wordle also ignores them, which is why you don't see a lot of the and and in English-language Wordles. But Wordle doesn't happen to be configured for Classical Greek.

So what happens if we whittle away at the stop words? Let's do this slowly. We'll start by getting rid of ὁ and καί.

Woah. Where did all that come from? You can see something now: θεός, λόγος, and if you really squint, ἄνθρωπος. But that's still making life too difficult, because there are more stop words to dismiss. I've highlighted the next batch in red: τίς, δέ, αὐτός, εἰμί, who?, but, he/himself, be. Of these, τίς "who?" is inflated through ambiguity with τις "someone"; because the lemmatisation is not disambiguated by context, a few word counts are more sizeable than they should be.

With those four out of the way, we have:

An improvement; you can see ἄνθρωπος now, and maybe even πατήρ "father" next to θεός "god". But we still can do better. We have eight more stop words that we don't really need to hear about: ἐγώ "I", ὡς "as, that", ὅς "who, that", τις "someone", οὐ "not", γάρ "because", ἐν "in", and οὗτος "this".

With them left out, we have:

Still better: you can make out ἔχις "viper" now, at the bottom left hand edge. Not that Greeks spent a lot of time talking about vipers; they just spent a lot of time using the verb ἔχει "has", which happens to be ambiguous with the dative of ἔχις. It's automated lemmatisation, this kind of thing can happen.

We have sixteen more stop words, and as you may have worked out, the easiest criterion is to bundle up all function words—prepositions, adverbs, conjunctions, interjections, pronouns. With some of the ambiguity inherent in the venture—is πᾶς "every" a pronoun or an adjective?—but we can keep slicing nonetheless:

And again:

We're not making as much of a difference now; but notice that the screen is being crowded out by verbs: λέγω "say" (and "pick", as a synonym that used to be the same verb—just like "count" and "recount" in English); γίγνομαι "become", ἔχω "have". These are verbs, and are properly considered content words. But I already got rid of εἰμί "to be" (which as a copula is not a content word; and I'm happy to also throw out "have", "become" (close to a copula itself), and verbs for "say". (There is a lot of "he said she said" in the TLG, because there is a lot of narrative.)

If we get rid of those verbs?

And tidying up getting rid of the next hundred and fifty function words, which are a distraction as you squint for content:

You could argue there's still some guff there: ποιέω "do" doesn't tell you much more than ἔχω "have", and πολύς "much" doesn't really deserve its disproportionate size. But we have enough cleaned up that we can say now something about what the texts talk about. It's certainly a sight better than this:

So what do the TLG texts talk about? You may well be starting to come up with ideas if you can read Greek. But before you do, remember that there a whole lot of Christian texts in the TLG, and they quantitatively crowd the ancient texts out. The texts of John Chrysostom alone in the TLG are almost as sizeable as all surviving Ancient literature between Homer and Aristotle.

So yes, the TLG as a whole talks about God and logos a fair bit. But we'd expect that of John Chrysostom; it doesn't mean its what Plato or Homer talk about.

What'd be useful is to split up the corpus, say BC and AD, and see how they differ. Sounds like the next blog post to me...

Btw, I've been stamping out stop words, but stop words are of interest if you're looking at grammar; and Nikos Sarantakos did ask me to pony up the word counts that I was tossing out. So, for the TLG and the lemmatiser as of last night, these are the twenty five most frequent lemmata of Greek, with their textual frequency:
πᾶς534,845every
ἕ547,255he
ἀλλά548,203but
διά561,813for
ἐπί566,238on
πρός566,476towards
κατά643,767by
εἰς694,035to
τῷ732,938therefore (ambiguous with "to the")
μέν762,890on the one hand
ἐγώ767,104I
ὡς771,416as, that
ὅς801,401who, that
λέγω811,330say
τις834,155someone
οὐ926,059not
γάρ951,810because
ἐν1,128,716in
οὗτος1,228,627this
αὐτός1,646,014he, himself
εἰμί1,704,651be
δέ2,265,028but
τίς2,624,172who?
καί5,765,491and
ὁ14,335,717the

Of the lemmata we have not thrown out, θεός "god" is the 39th most frequent, with 388,933 instances.
...Read more