Jonathan Reeve: Computational Literary Analysis

Announcing Annotags: A Concept for a Decentralized Literary Annotation Protocol

2013-05-10

Have you ever wanted to livetweet a book? I often want to. Yet there doesn’t seem to be a standard way of doing that. You could tweet, for instance, “Oh Captain Wentworth! When will you propose? #JaneAustenPersuasion” but the hashtag isn’t standardized, so some people might be tweeting under #Persuasion and others under #AustenPersuasion. Furthermore, the hashtag isn’t specific enough to make it clear which part of the book you’re tweeting about. If you browse other tweets marked #JaneAustenPersuasion, you might find comments about other parts of the book, which may contain spoilers.

Enter Annotags, a concept for a literary annotation hashtag, or annotag. Whenever you want to tweet about part of a book, use this format:

Annotag diagram

The first letter is the book code type. Uppercase letters refer to raw codes and IDs, and lowercase letters refer to base 64-encoded codes. Here are some examples:

I = Raw ISBN, ex. I0393911535.
i = Base 64-encoded ISBN, ex. iXepzv
G = raw Project Gutenberg book ID, ex. G105.
g = Base 64-encoded Gutenberg book ID, ex. gBp
B = Google Books ID.
O = Raw OCLC Number. Ex. O29877721. These can be found using Worldcat.
o = Base 64-encoded OCLC number. Ex. oBx-XZ.

Any other sources can be added to this list. I’m working on a way to shorten DOIs, for instance.

The next letters and numbers before the colon are the book code, optionally encoded in base 64. What is base 64, you ask? It’s just a mathy way to take a long number and convert it to a short string of numbers and both uppercase and lowercase letters. That way, you can take an unwieldy ISBN—the ISBN for the Norton Critical Edition of Jane Austen’s Persuasion, for instance, is 0393911535—and write it as “Xepzv”. Both I0393911535 and iXepzv are valid annotags, but the latter saves five characters, which you can use to write “LOL!!” or whatever your heart desires. To try this out, grab an ISBN of a book from Amazon.com or equivalent, remove all the hyphens and paste it into a converter like the annotag calculator I wrote here, which will convert it into Base 64. Some book IDs won’t benefit much from encoding, of course. Many Project Gutenberg etexts IDs are short anyway, so you might as well keep them as-is. The Project Gutenberg ebook for Persuasion, for instance, is listed as “EBook #105,” which can be represented as the annotag #G105. No calculator required.

Next, there is a colon, followed by the location code. The location code is made up of chapter numbers, page numbers, or whatever you like. If you want to get super-specific—say, if you’re annotating a particular word or phrase—add a line number or word number. If your book doesn’t have page numbers (if you’re reading an etext, for instance), you can index your location any way you want—by chapter and paragraph number, or even chapter, paragraph, and word. This is simple, intuitive, and human-readable. If your book code is G105, and you want to tweet about something that happens in Chapter XXIII, you can tweet “Oh no you di’n’t, Captain Wentworth! #G105:c23”. Because the annotag is only eight characters, you have another 130 or so for your witty, insightful commentary.

List of Location Abbreviations:

d = part (“division”), ex. d2c5 is Part II, Chapter 5.
b = book, ex. D3c7 is Book III, Chapter 7. For those books that are divided into “books.”
c = chapter, ex. c34 or cXXXIV is Chapter 34.
p = page, ex. p54 is page 54.
a = act, ex. a1 is Act I. To be used for drama.
s = scene, ex. a1s3 is Act I, Scene 3.
P = paragraph or stanza, ex. c5P2 is the second paragraph of Chapter 5.
l = line number, ex. p76l20 is page 76, line 20.
w = word number, ex. p82l10w4 is the fourth word of the tenth line on page 82.

Examples:

Not to be, man. Not to be. Get over it. #Hamlet #G1524:a3s1P24
- The Gutenberg Ebook for Shakespeare’s play Hamlet, Act III, Scene 1, paragraph 24.
Septimus’s shell shock makes him unusually atune to nature. #Dalloway #i_ZOO:p22
- The Harcourt Annotated Edition of Virginia Woolf’s novel Mrs. Dalloway, page 22.
What circles does the blackbird mark, exactly? #WallaceStevens #iB8F4h:p94P2
- The Collected Poems of Wallace Stevens, “Thirteen Ways of Looking at a Blackbird,” page 94, second stanza.

Advantages of Annotagging:

It’s decentralized, so it’s not contingent on any one particular institution or website.
Annotations are owned by their writers, not by the website that hosts them.
Annotags are mostly human-readable. Once you know the code for your book, it’s easy to figure out what “p54” means (hint: it’s page 54).
Annotagging allows you to comment on both paper books and ebooks, and refer to specific editions when necessary.
You don’t even need a computer to annotweet. Just send an SMS to your twitter account.
No registration necessary, provided you already have an account on twitter or identi.ca.
If you use Annotags in your blog posts, you aren’t restricted to the 140-character limit of microblogging platforms. You can even Annotag your scholarly papers.

Future Applications

There are lots of ways that apps could interface with this type of protocol. Here are some ideas:

A webapp to generate Annotags, allowing the user to look up books by author, title, or ISBN, and generate Annotags from them.
A webapp to aggregate Annotweets and display them in the margins of an etext, so that users can read an etext online and see what people have tweeted about it. Tweets that are line-specific will appear right next to those lines in the etext.
A mobile webapp or native mobile app (i.e. an Android app) that can generate Annotags, and maybe even scan book barcodes using one’s smartphone camera.
A script that can expand Annotags to regular MLA-compliant bibliographic entries.
A browser extension (i.e. a Firefox plugin) that automatically generates Annotags for books when you visit a book page on Amazon or Worldcat.
Labels for paper books that include that book’s annotag, pre-calculated:

ISBN with an Annotag

Introducing the Macro-Etymological Analyzer

2013-11-01

Note: this post refers to an older version of macro-etym, a text analysis tool which is now a CLI utility, here.

Here’s a sneak preview of my latest project, a macro-etymological textual analyzer web app. It’s an abstract, complete with preliminary results, which I recently prepared to submit to DH2014. Here’s a PDF file version: Abstract: Macro-Etymological Analyzer (237K PDF), and the full text of the abstract is below the cut.

The English language has continually borrowed from foreign languages—close to 30% of modern English words are loanwords from French, and another 30% are borrowed from Latin. These words are often concentrated in semantic frames associated with their origin languages—legal vocabulary contains a preponderance of words of French origin, and the vocabulary of the natural sciences contains many words of Latin and Greek origin. The etymology of words in a text, therefore, may be suggestive of its context or its level of discourse. Should a writer choose the Latinate term “masticate” over the Anglo-Saxon term “chew,” for instance, one might assume a scientific context or a high level of discursive formality. By computing the proportion of origin languages for all the words of a given text, we may quantify stylistic properties that are potentially revealing about the text and its rhetorical modes.

The Macro-Etymological Analyzer is a computer program that I wrote for this purpose. Written in PHP on a LAMP stack, it is a web app accessible at http://jonreeve.com/etym, and is freely available for all to use, modify, and distribute under the GPLv3. It accepts as input a user-uploaded text file, and looks up each word in Gerard de Melo’s Etymological Wordnet database. These words are then counted by language of origin using two generations of language ancestry, and then categorized by language family. The results are displayed as a pie chart made with the Google Data Visualization API, along with a CSV log file which can be used for comparative analyses. Currently, the program accepts only English texts, but the database supports queries from any source language, and plans are in place to make the program fully multilingual.

Figure 1 shows the proportions of Latinate words—words descended from Latin or romance languages—for each of the 15 genres in the Brown Corpus. Learned texts and government documents show the highest proportions of Latinate words, whereas romance and adventure stories show the lowest. The same textual categories sorted by proportion of Hellenic words (words of ancient Greek origin) show changes in certain categories—religious language exhibits a higher rank, and that of mystery stories is ranked lower than in the Latinate scale. These data suggest that a high proportion of Hellenic words is correlated with religious language, among other genres, and that a high proportion of Latinate words is correlated with learned language. Once literary works are analyzed with this method, these hypothetical correlations become potentially useful as literary critical tools.

Figure 1: Brown Corpus Genres

In one such analysis, the chapters of A Portrait of the Artist as a Young Man were run through the Macro-Etymological Analyzer. This novel, James Joyce’s Bildungsroman, is known for its style—one that mimics each progressive age of its protagonist Stephen Dedalus. Early chapters, when he is young, are written with infantile language; later chapters are written with more elevated language. The program’s results quantify this stylistic mode, to some degree—Chapters 1 and 2 show low proportions of Latinate words, whereas later chapters show higher proportions, as shown here in Figure 2. The fact that the proportion of Latinate words begins to plateau starting with Chapter 3 might be used to argue that Stephen has at this young age already reached a precocious maturity of vocabulary, which may reflect his study of Latin.

A Portrait of the Artist as a Young Man

In another analysis, the extracted monologues of the seven narrators of Virginia Woolf’s novel The Waves were computed with this program. As shown in Figure 3, the two university-educated characters, Bernard and Neville, show the highest proportions of Latinate words, while the housewife Susan shows the lowest. In fact, the male characters rank higher for Latinate words than the female characters—this would be an interesting starting-point for a discussion of gender in The Waves, especially framed by Woolf’s much-discussed writings on gender politics.

The Macro-Etymological Analyzer was also used to chart variations between editions of a text. The seven revisions of Whitman’s Leaves of Grass made available by the Whitman Archive were analyzed with this program. The results show a gradual increase in Latinate words from the 1855 edition to that of 1891-2. This might be used to argue that Whitman inflated his style with each revision, introduced foreign loanwords as he gained a more international reputation, or used a greater breadth of words as his vocabulary increased.

The Waves Narrators

These experiments were not without their surprises, of course. An early test of selected books of the King James Bible seemed promising, as it revealed the gospels Matthew, Mark, Luke, and John to have much higher proportions of Hellenic words than other books (see Figure 4). Unlike the books of the Old Testament, which were mostly written in Hebrew, these books were translated from the Greek—a fact which might seem to explain the presence of Hellenic words. Upon closer examination, however, the program was discovered to be counting the etymology of frequently-mentioned names like “Jesus” among words of Hellenic origin, and it was these names that accounted for most of the Hellenic words. Although the language of the source text did not prove to be the determinant here, this discovery may yet be valuable for other reasons—the synoptic gospels of Matthew, Mark, and Luke show similar portions of Hellenic words, whereas that of John is 100% greater. This would seem to support the hypothesis that the synoptic gospels were adapted from a common source text, whereas that of John had an independent source.

KJV Bible

A number of other experiments were also conducted, and are described in this paper. Included among texts analyzed by the Macro-Etymological Analyzer were: selected Canterbury tales (in modern English translation), a series of early and late Henry James novels, a collection of Victorian novels compared with a collection of modernist novels, and groups of French and German novels in English translation. Questions to be explored include:

Do translated works show a larger-than-normal proportion of words with etymological origins in the language of the source text?
Given a large enough data set, can linguistic trends (such as a general decrease in the use of Latinate words) be detected with this program? Can macro-historical events such as the Scientific Revolution be detected?
Do male and female writers of the 19th century differ in the origin-types of words they use?
Can the semantic frames in which these etymological groups of words are concentrated be explained historically, such as through the habits of the French-speaking English aristocracy in the era following the Norman Conquest?

Finally, this paper will discuss how this new tool might contribute to the suite of computational stylistics tools already available, and how macro-etymology might constitute a new metric that could be used towards stylistic fingerprinting or authorial detection.

Hacking at the Open Syllabus Project: Collocations by Subject

2013-06-09

I was invited to hack around on the Open Syllabus project this past Saturday, which I was really excited to do. They’ve scraped the web and come up with around 1.5 million syllabi, and only just released their API to researchers this weekend. I wanted to run some computational analyses on these syllabi, to attempt questions like:

What were the most frequently assigned texts in freshman composition courses?
What disciplines exhibit the most variance between their syllabi? That’s to say, which subjects have the most similar syllabi, and which have the most divergent?
What disciplines have the longest syllabi?
In which disciplines do technological marker words like “blog” or “Twitter” appear most frequently?

To start with, I used a JSON file containing a subset of these syllabi–around 1000, and tagged them by subject, using their first lines and filenames as hints. (See the quick and dirty code here.) I then imported another 600 subject-tagged syllabi from Graham Sack’s corpus, resulting in a subject-tagged corpus of around 1,300 syllabi. From there, I sorted the results by subject and ran it through NLTK to find collocations–words that frequently occur next to each other.

There were some interesting findings. Some were predictable, like “mineral resources” for Geology, “corale room” for Music, or “Homeric Hymn” for Mythology. Others were revealing about required texts frequently assigned for these subjects—Michel Foucault is likely frequently assigned in Sociology courses, “Beyond Good” suggests that Nietzsche’s Beyond Good and Evil is required reading in Philosophy courses, and the truncated “nhead Wilson” suggests that Twain’s tragedy—although here with a mistaken word boundary—makes a frequent appearance in Literature courses.

Some of the collocations are a little surprising, however. It’s a little surprising that “Hebrew Bible” appears as the highest-rated collocation in Literature syllabi. It’s even more surprising that “Gospel According” appears in Political Science syllabi. (Then again, since many of these syllbi come from Emory University, these may be features consistent with a Bible Belt institution.)

There are some stylistic features that deserve note here, too. Syllabi in the natural sciences exhibit the most use of ALL CAPS by a wide margin. Could it be that scientists have less exposure to the style manuals—those so loved by humanists—that discourage this style? Or is it possible that there is something about the exacting nature of the hard sciences that demands this kind of emphasis?

Chemistry syllabi featured some of these all caps collocations, and also featured the highest proportions of bigrams characteristic of course policy paragraphs: “make sure,” “honor code,” “usual penalty,” and “found guilty.” What is it about Chemistry as a discipline, I feel like asking, that makes it so strict in its policies? This could, of course, be an instance of a department-wide requirement for all syllabi to include a paragraph of boilerplate course policy text (this was a requirement for the composition courses I taught at CUNY).

Faced with these preliminary results, here are a few questions I ended up asking, with some hints as to their answers:

Which disciplines care the most about dress code? Physical Education and Music.
Which language course is the most concerned with grammatical accuracy? German.
What is the most-mentioned historical period in (this small subset of) history courses? The Civil Rights Era.
What discipline is concerned the most with weather? Geology. (Owing, probably, to class field trips.)

Of course, since this data set is so severely limited, and since this analysis was so haphazard, none of these answers may be considered even halfway accurate. Still, I achieved an even more important goal in this process—to generate more questions.

You can read the original output of the command here in the shared Google Drive folder we used, run the script yourself from the code on github, or check out this heavily-curated selection of the findings, organized by subject:

Astronomy: astronomers know; investigating nature; natural sciences; ultimate fate.

Biology: LAB EXAM; Cellular respiration; procedures outlined; FINAL EXAMINATION; SPRING BREAK.

Chemistry: Honor Code; Honor Code.; liberal arts; College Honor; Make sure; honor code; academic honesty; HONOR CODE; highest standards; usual penalty; found guilty.

Computer science: selected material; new StringBuffer; UNIX operating; constructor call.

English: Rhetorical Analyses; unauthorized information.

Film Studies: reserve Screening; Citizen Kane; Experimental Film; Francis Ford; Jean-Luc Godard; Maltese Falcon; Orson Welles; Royal Tenenbaums.

Geology: SPRING BREAK; scientific method; unsure whether; Mineral Resources.

German: grammatical accuracy.

Hist: Civil Rights; World War; Luther King; Martin Luther; National Historic; University Press; Black Folk; Field Trip.

Literature: Hebrew Bible; Invisible Man; Kingsblood Royal; Black Folk; nhead Wilson; online lectures; Frederick Douglass; Human Stain; exegetical paper;

Music: aesthetic sense; Chorale Room; Old Church; Black dress; Chamber Ensemble;

Mythology: Classical Mythology; Homeric Hymn; Friedrich Nietzsche; Irrational Hero; Roman Appropriation; Sigmund Freud; Using Madness;

Philosophy: Nicomachean Ethics; Beyond Good; Categorical Syllogisms; Ordinary Language;

Physical Education: medical condition; body composition; dressed appropriately; include vigorous; full participation; physical fitness; Skill Test; Tai Chi;

Policial science: University Press; New York; Conflict Resolution; Communist Manifesto; Second Treatise; Gospel According

Psychology: Think Critically; mental processes; critical thinking; Psychological Disorders; Human Development; theoretical perspectives; system; appropriate documentation.

Public Relations: well organized; rewrite opportunities; Case Study; Social Media; Personal brand;

Religion: Native American; site visit; Reflective Analysis;

Sociology: Michel Foucault

Annotag Calculator 1.0

2014-09-28

Following my initial spec for Annotags, a decentralized literary annotation protocol, I’ve just posted the first rough version of a webapp designed to encode and decode the tags. Now you can annotweet books, plays, poems, and other texts with shortened ISBNs, thus saving precious microblogging characters.

For example, my first annotweet using this system annotates the Folger Library Shakespeare edition of Hamlet, in the famous soliloquy that appears on page 127, at line 64. The ISBN-10 for that book, as Amazon tells me, is 074347712X. I put this in the annotag calculator like so:

Annotag Calculator

This generated the annotag #iEtVGa:p127l64. I copied and pasted this into my annotweet like this:

Annotweet

Because Twitter doesn’t like colons in hashtags, only the first part of the annotag is highlighted as a hashtag. This works to keep more specific annotags locatable, since it connects you to the other annotweets from the book, but isn’t so specific that it limits you to the line you’re annotagging. Eventually, I’ll write an aggregator that can find annotags and linkify the page numbers and lines, where possible, to electronic versions of those lines and pages.

Happy annotweeting! If your annotweet has a few extra characters left at the end, give me a shoutout at @j0_0n.

A Comparative Macro-Etymology of Whitman Editions

2014-09-14

Walt Whitman, in addition to being a poet, was somewhat of an amateur etymologist. He calls the English language “an enormous treasure-house, or range of treasure houses, arsenals, granary, chock full with so many contributions … from Spaniards, Italians and the French” (quoted in LeMaster, 226). In the preface to the first edition of Leaves of Grass, he summarizes the history of English word borrowing: “On the tough stock of a race who through all change of circumstance was never without the idea of political liberty … [the English language] has attracted the terms of daintier and gayer and subtler and more elegant tongues” (PW, II, 456-57, quoted in Warren, 34). Here, the “tough stock” is Anglo-Saxon, and the “more elegant tongues” are French, Latin, and Greek (35). Elsewhere, he credits French as the power that “free[d] the nascent English speech from those useless and cumbersome forms with which the Anglo-Saxon was overloaded (Rambles, 273, quoted in Warren, 45).

In his poems, Whitman uses foreign words and expressions liberally, and a few critics have noted that his use of loanwords increases with every revision of his magnum opus Leaves of Grass (LeMaster 227). French loanwords, many of which he may have picked up from his travels in Louisiana, were some of his favorites. Whitman himself comments on his own use of French words in an article he published in the Brooklyn Times in the 1850s. In this article, he gives a long list of French words, calling them “a few Foreign Words, mostly French, put down suggestively” (Asselineau 368, n143). He continues, “some of these are tip-top words, much needed in English.” Indeed, many of these words, such as “insouciance,” “cache,” and “ennui,” have now entered the English langauge, but others, such as the titularly appropriate “feuillage,” have not. As LeMaster and Kummings note, Whitman anglicizes some French words, dropping some of the accents, and uses phonetic or creative spellings for others, rendering “rondeur” as “rondure,” for instance, a spelling which recalls “verdure” (227). While some of these French loanwords are probably malopropisms—“resumé” in “Night on the Prairies,” for instance (LeMaster 228), others bring new levels of meaning to the poems. One of these orthographic inventions, the title of his poem “Respondez!” is especially interesting. If we do not dismiss this as an archaic or erroneous spelling of the French “repondez,” we might read it as a mixture of the French word with the English “respond,” which itself is derived from the Old French “respondere.” The word is both more English-sounding (to the anglophone ear) than its modern French equivalent, and more archaic in tone than its modern equivalents in either language—the word straddles two languages, and five hundred years of language history.

Another interesting case is Whitman’s use of the French word amie, which appears twice in the 1855 “Song of Myself.” He uses the feminine form of the word, even though he uses it to refer to a man: “Picking out one here that shall be my amie, / Choosing to go with him on brotherly terms” (quoted in Warren, 700-1). Furthermore, as Warren points out, Whitman is well aware of the gendered forms of this word, for he defines both forms in his essay “America’s Mightiest Inheritance.” The definition he gives for both is “Dear Friend.” Here again we have an instance of linguistic divergence—in French, “ami(e)” simply means “friend”; Whitman has imported the word and given it a more specialized meaning.

The goal of this experiment is to test the hypothesis that Whitman increasingly uses words of Latinate origin in his revisions of Leaves of Grass. Thankfully, seven of these revisions were available in TEI XML from the Whitman Archive, so this test was relatively easy. A few lines of code were added to the code to strip out XML tags, and each of these TEI files were run through the program.

Latinate Words in Revisions of Leaves of Grass

The results of these test were as previously speculated—the proportion of Latinate words increases with each revision of Leaves of Grass, but with one minor exception—that of 1867. LeMaster and Kummings call this edition “the most chaotic of all six editions,” whose significance “lies in its intriguing raggedness, which is embedded in the social upheaval in the immediate aftermath of the Civil War” (365). Can the “images of a coherent union” and the “urgently accented democratic nationality” which they claim characterize this edition account for the slight drop in Latinate words, or slight increase in Germanic words? Do the six new poems of this edition contain an unusually high proportion of Germanic words? These are questions that demand further investigation.

Note

This post is an adapted and expanded excerpt from my 2013 Master’s thesis, “Macro-Etymological Textual Analysis: an Application of Langauge History to Literary Criticism.” The program described herein is the web app created for these experiments, the Macro-Etymological Analyzer. Read more about the program and related experiments in my introductory post, “Introducing the Macro-Etymological Analyzer.”

Works Cited

Asselineau, Roger. The Evolution of Walt Whitman. Iowa City: U Iowa P, 1999. Google Books. Web. 1 December 2013.

LeMaster, J.R. and Donald Kummings, eds. Walt Whitman: An Encylopedia New York: Routledge, 1998. Google Books. Web. 1 Dec. 2013.

Warren, James Perrin. Walt Whitman’s Language Experiment. University Park: Penn State P, 1990. Print.

A Macro-Etymological Analysis of Character Dialog in Virginia Woolf’s The Waves

2014-10-21

Virginia Woolf’s experimental novel The Waves is structured as a series of dialogues and monologues by six characters. Some of these speeches are as short as a line, but most are between a few paragraphs and a chapter in length. Their extremely regular structure takes this form:

‘I see a ring,’ said Bernard, ‘hanging above me. It quivers and hangs in a loop of light.’ (Woolf 9)

There is a short phrase, an attribution with the word “said,” and a continuation of the speech–this is true of all the speech in the novel. This regularity makes this novel an unusually good subject for computational study, since the text can so easily be extracted—see Stephen Ramsay’s tf/idf experiment, and Chris Forster’s subsequent blog post, wherein he suggests a possible regular expression to capture dialogue. Additionally, the fact that there are three male characters and three female characters make this an interesting study for the investigation of gendered speech, at least as it appears in Woolf.

For this experiment, text files containing each of the characters’ total speech were fed through the Macro-Etymological Analyzer. As the figure below shows, the macro-etymologies of the characters’ speech are distinct, and separate by gender, with the male characters showing the highest proportions of Latinate words, and the female characters showing the lowest.

The Waves by Narrator

Bernard and Neville are the two university-educated characters, and the ones with the highest proportions of Latinate words. Bernard identifies with Shakespeare and is, according to Fogel, “a clear surrogate for Virginia Woolf herself” (157, quoted in Hussey, 25). Neville “anticipates becoming a classical scholar, exploring the writing of Virgil, Lucretius and Catallus” (Hussey 181-2). Early in the novel, Neville imagines entering the library where he “shall explore the exactitude of the Latin language” (31). That exactitude is invoked by Bernard later–during one of his conversations with Neville, he praises him thus:

you wish to be a poet; and you wish to be a lover. But the splendid clarity of your intelligence, and the remorseless honesty of your intellect (these Latin words I owe you; these qualities of yours make me shift a little uneasily and see the faded patches, the thin strands in my own equipment) bring you to a halt. (85)

The “Latin words” to which Bernard refers are “splendid,” “clarity,” “intelligence,” and “remorseless,” all words descended from Latin which entered English through French. Later, Bernard describes Percival as one who thinks with “magnificent equanimity” (243). In parentheses, he justifies his own florid word choice here by saying “Latin words come naturally.” Is Woolf implying that, due to social circumstances, perhaps Latin words do not come as “naturally” to the female characters?

In contrast to Bernard and Neville, Jinny is “sensual, alert to vivid color and to the power of her own body to attract men” (Hussey 131). Susan is a “representation of the maternal ‘instinct.’ … closely identified with the natural world. She marries a farmer, lives in the country, and is fiercely protective of her children” (280). These are qualities typically associated with Anglo-Saxon words—visceral, sensual words; kinship terms, rural settings.

In Woolf’s “A Room of One’s Own,” she imagines a female Shakespeare, Shakespeare’s sister Judith. Woolf argues that Judith Shakespeare couldn’t have written her brother William’s works, not from lack of talent, but from lack of opportunities–“rooms of one’s own” for Elizabethan women writers during this period. “She had no chance of learning grammar and logic,” Woolf contends, “let alone of reading Horace and Virgil. She picked up a book now and then, one of her brother’s perhaps, and read a few pages. But then her parents came in and told her to mend the stockings or mind the stew and not moon about with books and papers” (47). Therefore, one imagines that Judith Shakespeare, like the female characters in The Waves, would have used fewer Latinate words than her male counterparts.

Works Cited

Hussey, Mark. Virginia Woolf A to Z. New York: Facts on File, 1995. Print.

Ramsay, Stephen. “Algorithmic Criticism.” A Companion to Digital Literary Studies. Oxford: Blackwell, 2008. Web. http://www.digitalhumanities.org/companion

Woolf, Virginia. A Room of One’s One. New York: Harcourt, 1929. Print.

—. The Waves. New York: Harcourt, 1931. Print.

Note

This post is an adapted and expanded excerpt from my 2013 Master’s thesis, “Macro-Etymological Textual Analysis: an Application of Language History to Literary Criticism.” The program described herein is the web app created for these experiments, the Macro-Etymological Analyzer. Read more about the program and related experiments in my introductory post, “Introducing the Macro-Etymological Analyzer.”

Detecting Literary Chiaroscuro in Eliot, Dickens, and other Victorian Novelists

2015-03-02

In an 1873 review in the Galaxy, Henry James criticizes George Eliot’s Middlemarch as he might a painting: “it abounds in fine shades, but it lacks, we think, the great dramatic chiaroscuro” (579). Although James is using the term metaphorically, to denote sweeping emotions and narrative flux, the same could be said of the visual mise-en-scène of the novel. The presence of literal light and darkness in a novel’s descriptive tableaux lays a visual foundation that often projects to the greater narrative structure. Imagery of intense light and stark shadows is correlated with similarly tumultuous brush strokes in the plot—chiaroscuro functions both poetically and narratologically. While this phenomenon is subtle in a novel like Middlemarch, other Victorian novels, especially Charles Dickens’s Bleak House, exemplify this shadow play. The following study synthesizes quantitative and qualitative methods to compare these two novels, to show how these writers employ optical effects to frame their theatrical dynamics. Where Dickens uses chiaroscuro to heighten dramatic intensity, Eliot uses it as a preliminary artistic maneuver towards a more deliberately subtle mezzotint.

A computational analysis of light and dark words in these novels suggests, to whatever small degree these tropes may be quantified, Henry James might have been right that Middlemarch lacks “chiaroscuro,” at least in comparison to a small number of other Victorian novels. To test this claim, a wordlist of hyponym lemmas was generated with the Princeton Wordnet, using the seed concepts “light” and “dark.” The wordlist was curated to remove irrelvant word senses, and the novels were then analyzed for the presence of these words, using the Python Natural Langauge Processing Toolkit. Adjusted for the total number of words in each novel, the combined score for light and dark words in Middlemarch was 3.28, while it was 4.31 for Bleak House. Middlemarch also scored lower than Henry James’s Turn of the Screw (4.18), a decidedly dark gothic novella, and even scored lower than Jane Austen’s largely domestic novel of manners, Pride and Prejudice, which exhibited a surprisingly high score of 4.74 (see Figure 1).

Proportions of Light and Dark Words in Selected Victorian Novels

Another surprising finding of this analysis was that Bleak House scored much lower for light and dark words than other Dickens works, despite the fact that it is widely regarded as the first of his “dark novels” (q.v. Stevenson). It scores lower than Oliver Twist, Nicholas Nickelby, Hard Times, Little Dorrit, and A Tale of Two Cities—all tested novels except The Pickwick Papers (see Figure 2). Of course, a simple word search cannot definitevely reveal the full breadth of lexical expression that convey a mood; a dark or light scene might be described more than adequately without ever using the words “dark” or “light,” or any of their hyponyms. One would expect the numbers of these words to be unusually high, for instance, in Ann Radcliffe’s archetypal gothic novel The Mysteries of Udolpho, but the score is a very low 1.27. One must conclude that a more sophisticated computational model is needed here. In the meantime, a qualitative textual analysis will help to fill that need.

Proportions of Light and Dark Words in Selected Dickens Novels

Very broadly, the traditional associations of light and darkness are those of good and evil, respectively. Norman Friedman’s 1975 treatment of “Sun and Shadow” in Bleak House, for instance, turns almost immediately into a discussion of morality and “the problem of evil” (363). An interpretation of chiaroscuro that stops at that level, however, misses much—light is also associated with knowledge and science, an association perhaps best represented as the periods that have been called the Enlightenment and the Dark Ages. Closely related are temporal connotations: light, especially gas light, in the Victorian imagination, could be said to be associated with the future and technological progress, whereas the past is increasingly shrouded in darkness. There is also a racial and social connotation, exemplified, for instance, in the title of Conrad’s Heart of Darkness, which adds a further dimension to this imagery. The multiplicity of meanings associated with chiaroscuro allows for an exploration of certain patterns of ambiguity, such as that in the binaries of the visual/tactile, emotional/epistemological, and technological/ontological.

The material conditions of lighting in the mid-19th century helped to shape their literary manifestations. Matthew Luckiesh describes this period as a transition between “mere” light and “more” light: “Until the middle of the nineteenth century mere light was available … Gradually mere light grew to more light and in the dawn of the twentieth century adequate light became available” (ix). It is even more true now that “In the present age of abundant artificial light, with its manifold light-sources … mankind does not realize the importance of this comfort” (5). Catherine LeGouis notes that “By mid-century, technological changes had brought about a different way of looking at reality using these new means of lighting, deeply affecting the realists’ use of the light/dark opposition” (424). Although LeGouis is primarily writing about Zola’s Paris (and although the shadows of Paris do appear in Bleak House), the same may be said of much of England. Bright gas streetlights had been installed throughout London in the early 19th century (Pool 198). George Sala, writing in Dickens’s Household Words, declares that “Not a bolt or bar, not a lock or fastening, not a household night-wanderer, not a homeless dog, shall escape that searching ray of light which the gas shall lend him, to see and to know” (341). The juxtaposition here, of “to see” and “to know,” is more than a reworking of the cliché “seeing is believing”—it associates visual experience with knowledge, hinting at an epistemological theory dependent on the experience of light.

Domestic lighting in early Victorian England would not have been nearly as bright as these gas lights, especially for the working class. Instead, candlelight or even rushlight would have been more common (Pool 198-9). The entry in the Imperial Dictionary of the English Language for “rush-light” calls it “any weak flickering light,” and even cites Dickens’s Pickwick Papers: “smoking and staring at the rush-light” (Ogilve 747). Dickens’s use of light, however, is more than just descriptive; it leaps out of the fireplaces and casts shadows on everything in the room, even (or perhaps especially) the psychological states of the characters.

John Ruskin, in a section of Modern Painters titled “The Truth of Chiaroscuro,” argues that “shadows are in reality, when the sun is shining, the most conspicuous thing in a landscape, next to the highest lights. All forms are understood and explained chiefly by their agency” (175). Later, in an 1863 letter to his father, Ruskin writes about his favorite scenes from Charles Dickens, all of which involve dramatic natural lighting: “The storm in which Steerforth is wrecked, in Copperfield; the sunset before Tigg is murdered by Jonas Chuzzlewit; and the French road from Dijon in Dombey and Son … are quite unrivalled in their way” (215). Perhaps what he finds appealing about Dickens is the same quality he celebrates in Turner. Although Ruskin does not mention Bleak House specifically, what he says of Copperfield certainly applies here.

Dickens has been rightly called “the most intensely visual of Victorian writers” (Andrews 97). In fact, he began his career as a novelist with images of light and darkness. The first sentence of his first novel begins:

The first ray of light which illuminates the gloom, and converts into a dazzling brilliancy that obscurity in which the earlier history of the public career of the immortal Pickwick would appear to be involved, is derived from the perusal of the following entry in the Transactions of the Pickwick Club, … (P 1)

The sentence continues for many more lines. The verbosity here, and also its intense, poetic light imagery, may have been what led critic Garrett Stewart to claim, of this sentence, that “After Dickens, no one could write that way again and be taken seriously” (136). The contrast between “gloom” and “dazzling brilliancy” is striking, especially for the opening passage of a novel. It is further striking to discover that this light is not, as one would imagine, a description of a landscape, but rather the figurative light of knowledge that illuminates Pickwick’s “public career” for the readers. The light imagery, therefore, has a threefold purpose: its literal associations paint a dramatic, Turnerian picture; its figurative meaning inaugurates an epistemological theme, and provides narrative tension by withholding information; and its association with the public places the light/dark dichotomy in dialog with the public/private.

This style is further intensified with Bleak House, which is often called the first of Dickens’s “dark” novels. Norman Page explains this term:

[Dickens] was no longer a young man, recently and happily married, with the world at his feet, but was middle-aged, unhappy in his personal life, and in deteriorated health. Beyond all this, though, the pessimism also seems a reaction to the problems of the age … For this reason the later novels have often been referred to as Dickens’s ‘dark’ novels … (2)

Here, “dark” is mostly being used to mean “pessimistic” or “unhappy,” but there is also the more literal connotation of “lacking light.” Dickens’s later novels deal with unpleasant social realities (social “darkness”), but also with darkness itself. This ambiguity is one that locates unhappiness in the visual realm, one that conflates emotion with visual experience.

The major settings of Bleak House—the High Court, Chesney Wold, Tom-all-Alone’s, and Bleak House itself—are described (one could nearly say “painted”) in terms of light and darkness. The High Court of Chancery, Chesney Wold, and Tom-all-Alone’s are dark; Bleak House, despite its name, is bright. The fog and darkness of the Court directly parallels a psychological darkness in the solicitors. The court itself is described as “dim, with wasting candles,” with “fog hang[ing] heavy … windows admit[ting] no light of day” (6). Similarly, the members of the bar are “mistily engaged in one of the ten thousand stages of an endless cause.” It is perhaps because of the fog and darkness that they are “tripping one another up on slippery precedents” and “groping knee-deep in technicalities.” In contrast, we are first introduced to Bleak House as “a light sparkling on the top of a hill” that becomes “a gush of light from the opened door” (59). Esther’s “first impressions” of the house remark about “its illuminated windows, softened here and there by shadows of curtains, shining out upon the starlight night” and its “light, and warmth, and comfort” (64). Tom-all-Alone’s, predictably, is “black,” and every so often a house collapses in a “cloud of dust” (197). The first time we see Mr. Krook’s “Rag and Bottle Warehouse,” it is “foggy and dark,” and whatever light there is, is “intercepted” by the shop; Mr. Tulkinghorn’s office is lit by “two candles in old-fashioned silver candlesticks that give a very insufficient light” (119).

The characters, too, are sketched with chiaroscuro. Esther Summerson’s surname, as Friedman points out, resonates with “summer sun” (366). When she is reconciled with Mr. Jarndyce, which is to say, relieved of her promise to marry him, he speaks “radiantly and beneficently, like the sunshine” (752). Conversely, Richard and Ada, when they come to terms with their inability to marry each other, are seen to pass from “the adjoining room, on which the sun was shining,” continue “lightly through the sunlight,” and finally disappear ominously “into the shadow” (163). The narrator laments, “It was only a burst of light that had been so radiant. The room darkened as they went out, and the sun was clouded over.”

The description of Sir Leicester’s clothes (ironically, in a chapter called “In Fashion”) is similarly monochromatic: “One peculiarity of his black clothes and of his black stockings, be they silk or worsted, is that they never shine. Mute, close, irresponsive to any glancing light, his dress is like himself” (14). Like his clothes, one might say that Sir Leicester is “unreflective,” both in the sense that he is lackluster, and in the sense that he doesn’t seem as thoughtful as the other figures in the novel. Here, light is revealed to be something more than an inherent quality—it is either reflected or absorbed by certain characters. Esther, in contrast, says that she “had never worn a black frock” (18).

Fire, as a source of light and shadow, is used heavily throughout Bleak House. Patricia Marks, writing about Barnaby Rudge, argues that fire, as man-made light “may be used to fulfill its original creative function or may be perverted into a destructive force,” and that “a domestic fire serves the function of providential light. At the Maypole, for example, the central fireplace not only vanquishes darkness but also, like the sun, causes reflections in everything that surrounds it” (74). When Mr. Tulkinghorn goes to find the dead Nemo, he is met with darkness: “He comes to the dark door … He knocks, receives no answer, opens it, and accidentally extinguishes his candle in doing so. The air of the room is almost bad enough to have extinguished it if he had not” (124). Dickens made a point to show that this candle did not extinguish itself by accident—that way, it is more obvious as an omen of the soon-to-be-discovered death. When Krook tries to light it later, he discovers that, in a similar image, “the dying ashes have no light to spare” (125).

Simultaneously, Mr. Krook’s spontaneous combustion might be read as an example of Mark’s “destructive force” which “serves the function of providential light” by “vanquishing” the “darkness” represented by Mr. Krook. The first time we meet him, his breath is, by way of subtle foreshadowing, “issuing in visible smoke from his mouth, as if he were on fire within” (49). Once he becomes a pile of ashes, the narrator insinuates that he was but one of many “lord chancellors” who make “false pretences” and do “injustice,” and that, as a result, he met with the same fate as they did—“Spontaneous Combustion” (403). Dickens’s implication, of course, is that all of these “lord chancellors,” if they are not already dead, are dying slowly, from the inside out, of some sort of moral decay. The fire, then, acts as a kind of judge for this immorality, separating the light from the shadow. What Marks says about Barnaby Rudge, therefore, that it “presents … a history of the cosmic alternation between life and death as represented by light and darkness” is not entirely true of Bleak House (76). Darkness is not always associated with death—during Jo’s death, for instance, “the light is come upon the dark benighted way” (572).

Darkness and shadow don’t always have a negative connotation in Bleak House. An early scene has Ada sitting at a piano, with Richard standing besider her. On the wall, “their shadows blended together,” foreshadowing, so to speak, their blended fates (68). Donald Ericksen compares this subtle note to similar visual themes in Victorian narrative paintings: “The reader is expected to puzzle out the meaning of the scene in terms of the present and the future, as Esther’s response leads us to do” (38). In Middlemarch there is a strikingly similar scene, which describes Dorothea and hints at her growing sexual awareness:

…there was nothing of an ascetic’s expression in her bright full eyes, as she looked before her, not consciously seeing, but absorbing into the intensity of her mood, the solemn glory of the afternoon with its long swathes of light between the far-off rows of limes, whose shadows touched each other. (18)

Hablot Browne’s Illustrations for Bleak House extend this textual theme. There are ten “dark plate” illustrations, in particular, that accompany dark moods in the text. As Ericksen explains, “The illustrator created these dark plates by first making fine machine-ruled lines through the etching ground and then later applying the acid and stopping-out processes. In this manner an infinite range of tones could be produced” (37). This evening-out of the tone also allowed the background elements to take on more visual importance (Harvey 152). The lack of human figures in many of them further accentuates the background. In the plate titled “Tom-all-Alone’s,” for instance, the lack of characters draws attention to the architecture and to the poverty of the street.

Perhaps the most intense chiaroscuro among these illustrations is the plate titled “A New Meaning in the Roman” (586). The title is a reference to the pointing Roman figure of Allegory painted on the ceiling, who is now pointing to the blood stain on the floor. The light streaming through the window, which also points directly to the blood stain, suggests a selective attention—an epistemology of the apartment. Its juxtaposition with the pointing finger suggests that the light is a kind of accusatory vector, as well. In this sense, the beam of light takes on the function previously held by the fire: that of divine judgment. As Richard Stein interprets it, “emphasis falls on contrasted signifying modes: allegory has been socialized, traditional iconography supplanted by a more diffuse and complex visuality” (183). Indeed, the repetition of forms here in this illustration suggests a proliferation of allegory.

The text and illustration of this scene diverge. In the text, the narrator reprises the ominous dark imagery in saying that “no light is admitted into the darkened chamber” (585), whereas sunlight appears to be coming through the window in the illustration. Whereas the candles in the illustration are lit, they are extinguished in the text: “He [Allegory] is pointing at a table, with a bottle (nearly full of wine) and a glass upon it, and two candles that were blown out suddenly, soon after being lighted.” This image recalls the earlier extinguished candle, also associated with Tulkinghorn.

“Consecrated Ground,” another dark plate, depicts a graveyard illuminated with dramatic rays of light from the streetlamps. As in “A New Meaning,” this illustration features a pointing figure, and the pointing motion is mirrored by the action of the light. The ray of light here, due to the way it falls around the corners of gravestone, is shaped like a pointing hand, which gives Joe’s pointing hand a circumstantial halo. Stein argues that “where Jo points through the churchyard gate to Nemo’s grave, the boy’s gesture directs our eyes past the edge of the image, toward an invisible world of death that resists imaging and interpretation” (179). Lady Dedlock’s face is turned away, heightening her perceived anonymity here, to both Jo and to the reader.

Some of the most explicit acknowledgment of light and shadow comes from two illustrations named, appropriately, “Light” and “Shadow.” The subject matter of these plates is telling: “Light” is a bright domestic scene; “Shadow” is dark and ominous. The passage “Light” illustrates is taken from the chapter titled “Enlightened,” most likely a reference to the scene where Esther learns of Richard and Ada’s marriage:

…[Ada] rose, put off her bonnet, kneeled down beside him with her golden hair falling like sunlight on his head … “Esther, dear,” she said very quietly, “I am not going home again.”

A light shone in upon me all at once.

“Never any more. I am going to stay with my dear husband. We have been married above two months.” (613)

Here, light imagery is used to symbolize knowledge, as it does in the name of the Enlightenment era. This association is supported by, among other elements, the centrality of the bookcase in the illustration and the papers scattered throughout this apartment. The knowledge of Ada and Richard’s marriage is perhaps that which gives these two characters the halo-like glow which forms the lightest area of the illustration. The darkest area of the illustration, in keeping with this symbolism, is the area behind Esther, away from which she appears to be moving. In fact, the light appears to be coming from the direction of the viewer. The twin circular areas of light would almost seem to indicate a pair of eyes, as if somehow acknowledging the presence of the viewer/reader. Since the strongest light surrounds Ada and Richard, there is also the symbolic connection between light and romantic love, a symbolism we will encounter again with Middlemarch.

The companion illustration, “Shadow,” depicts a lone Lady Dedlock, ascending a dark staircase after having encountered Mr. Bucket. She is eying, in passing, a poster, which in the illustration reads “Murder! £100 Reward!”. The passage reads:

She scarcely makes a stop, and sweeps up-stairs alone. Mr. Bucket, moving towards the staircase-foot, watches her as she goes up the steps the old man came down to his grave; past murderous group of statuary, repeated with their shadowy weapons on the wall; past the printed bill, which she looks at going by; out of view. (635)

This is a remarkably visual passage, putting the reader in Mr. Bucket’s place as he watches his suspect Lady Dedlock. “Murder,” in various forms, is “repeated” several times in the illustration: in the suspected Lady Dedlock, in the poster, and in the statuary. Michael Steig points out that this statuary “appears to be Abraham and Isaac at the point when Abraham is about to sacrifice his son,” and cites Ronald Paulson’s interpretation that this might “allude to the contrast between God’s and man’s justice” (153). With that in mind, the shadow that “repeats” this image might be read as another instance of light acting as a divine judge.

George Eliot’s Middlemarch, in keeping with Henry James’s critique, features light and dark symbolism to a somewhat lesser degree. It is not, however, completely absent, and the way in which she uses these symbols is very different from in Bleak House. The phrase “lights and shadows” appears five times in the text, and each time accumulates meaning. The first usage is to contrast Bulstrode’s stringently sanctimonious chiaroscuro with the mezzotint moral gradations of those he hypocritically judges:

This was not the first time that Mr. Bulstrode had begun by admonishing Mr. Vincy, and had ended by seeing a very unsatisfactory reflection of himself in the coarse unflattering mirror which that manufacturer’s mind presented to the subtler lights and shadows of his fellow-men … (84)

The second instance also appears after a quarrel, this time between Dorothea and Casaubon. Once again, the “lights and shadows” mirror a discrepancy between imagination and reality. In the above passage, it is Bulstrode’s haughty pietism; in the passage below, it is the contrast between Dorothea’s imaginary idealized marriage and its realities:

Dorothea was crying, and if she had been required to state the cause, she could only have done so in some such general words as I have already used: to have been driven to be more particular would have been like trying to give a history of the lights and shadows, for that new real future which was replacing the imaginary drew its material from the endless minutiae by which her view of Mr. Casaubon and her wifely relation, now that she was married to him, was gradually changing with the secret motion of a watch-hand from what it had been in her maiden dream. (124)

The third scene with this phrase comes directly before chapter XXII, when Dorothea and Will have one of their first flirtatious conversations in Rome.

We are all of us born in moral stupidity, taking the world as an udder to feed our supreme selves: Dorothea had early begun to emerge from that stupidity, but yet it had been easier to her to imagine how she would devote herself to Mr. Casaubon, and become wise and strong in his strength and wisdom, than to conceive with that distinctness which is no longer reflection but feeling—an idea wrought back to the directness of sense, like the solidity of objects—that he had an equivalent centre of self, whence the lights and shadows must always fall with a certain difference. (135)

Here the contrast between “lights and shadow” seems to parallel the contrast between Dorothea’s “devotion” and her “feeling”—what we might assume is her budding sexual awakening. This chiaroscuro still reflects the imagination/reality duality, but it has taken on a slightly new form—the imagined realm now parallels the world of social responsibility (to Dorothea’s marriage, for instance), while the real parallels that of passion, of “directness of sense, like the solidity of objects.”

Next, “lights and shadow” are associated with Mr. Farebrother: “Mr. Farebrother came up the orchard walk, dividing the bright August lights and shadows with the tufted grass and the apple-tree boughs” (251). Although this scene is more of a landscape painting than a psychological treatise, there are still the implications of an impending drama. In the following scene, Farebrother delivers the Garths the news that Fred Vincy is leaving town. This is the first of many times he will act as an intermediary—fulfilling his role as clergyman by dividing, in a sense, the “lights and shadows.”

A more subtle scene involving light takes place in the Roman gallery. Here again, light is associated with sensuality. It is necessary to quote this passage at length:

… the two figures passed lightly along by the Meleager, towards the hall where the reclining Ariadne, then called the Cleopatra, lies in the marble voluptuousness of her beauty, the drapery folding around her with a petal-like ease and tenderness. They were just in time to see another figure standing against a pedestal near the reclining marble: a breathing blooming girl, whose form, not shamed by the Ariadne, was clad in Quakerish gray drapery; her long cloak, fastened at the neck, was thrown backward from her arms, and one beautiful ungloved hand pillowed her cheek, pushing somewhat backward the white beaver bonnet which made a sort of halo to her face around the simply braided dark-brown hair. She was not looking at the sculpture, probably not thinking of it: her large eyes were fixed dreamily on a streak of sunlight which fell across the floor. But she became conscious of the two strangers who suddenly paused as if to contemplate the Cleopatra … (121)

This passage exemplifies the tension between the Dorothea’s “Quarkerish” qualities and her “breathing,” “blooming” nature, here intensified by the juxtaposition with the “volutuousness” of the statue. This dichotomy is superimposed upon that of the statue, which also has a double identity. It is now known as Ariadne, whose name literally means “most holy,” and would therefore be associated with Dorothea’s religious personality, yet it was then known as Cleopatra, one of the most sexually charged characters of antiquity. While Ariadne is sleeping in marble, Dorothea is daydreaming, with her hand as her pillow. Light plays a major role in this scene, as that which gives Dorothea her “halo,” and that which hypnotizes her. Light, as the medium through which all vision takes place, is here the mode through which Will’s gaze fixes itself on Dorothea. That artistic gaze is then that which divides the light from the dark, and thus the social from the primal, the responsible from the passionate. Where Dickens uses light and darkness to intensify divine moral judgments of his characters, Eliot uses it to liberate them from these divine judgments.

In closing, it might be useful to respond to Henry James’s criticism with that of his contemporaries. While it may be true that Middlemarch lacks a certain chiaroscuro, perhaps that is its literary strength. To quote Ruskin again, “It is the constant habit of nature to use both her highest lights and deepest shadows in exceedingly small quantity … thus reducing the whole mass of her picture to a delicate middle tint” (180). This “delicate middle tint” might be a good way of describing Middlemarch.

Works Cited

Andrews, Malcolm. “Illustrations.” A Companion to Charles Dickens. Malden, MA: Blackwell, 2008. 97-125. Print.

Dickens, Charles. The Pickwick Papers. New York: Oxford University Press, 2008. Print.

—. Bleak House. New York: W. W. Norton & Company, Inc., 1977. Print.

Eliot, George. Middlemarch. New York: W. W. Norton & Company, 2000. Print.

Ericksen, Donald H. “Bleak House and Victorian Art and Illustration: Charles Dickens’s Visual Narrative Style.” The Journal of Narrative Technique 13.1 (2012): 31-46. Print.

Friedman, Norman. “The Shadow and the Sun: Archetypes in Bleak House.” Form and Meaning in Fiction. Athens, GA: University of Georgia Press, 1975. Print.

James, Henry. “George Eliot’s Middlemarch.” Middlemarch. New York: W. W. Norton & Company, 2000. 578-581. Print.

LeGouis, Catherine. “Optics and Rhetoric: Images of Light in Zola.” Romanic Review 84.4 (1993): 423-436. Print.

Luckiesh, Matthew. Artificial Light: Its Influence upon Civilization. The Century Co., 1920. Web. 13 May 2012.

Marks, Patricia. “Light and Dark Imagery in Barnaby Rudge.” Dickens Studies Newsletter 9.6 (1976): 73-76. Print.

Ogilvie, John. The Imperial Dictionary of the English Language. Blackie & Son, 1884. Web. 16 May 2012.

Pool, Daniel. What Jane Austen Ate and Charles Dickens Knew: From Fox Hunting to Whist: the Facts of Daily Life in Nineteenth-century England. Simon & Schuster, 1994. Web. 16 May 2012.

Ruskin, John. “From a Letter.” Charles Dickens Critical Assessments. Ed. Michael Hollington. Mountfield, UK: Helm Information, 1995. Print.

—. The Works of John Ruskin: Modern Painters, V.1. New York: John Wiley and Sons, 1890. Print.

Sala, George. “The Secrets of the Gas.” Household Words 1854: 338-345. Web.

Stein, Richard L. “Bleak House and Illustration: Learning to Look.”

Approaches to Teaching Dickens’s Bleak House. New York: Modern Language Association, 2008. Print.

—. “Dickens and Illustration.” Cambridge Companion to Charles Dickens. Cambridge, UK: Cambridge University Press, 2001. Print.

Stevenson, Lionel. “Dickens’s Dark Novels, 1851-1857.” The Sewanee Review 51.3 (1943). Web. 23 November 2014.

Stewart, Garrett. “Dickens and Language.” Cambridge Companion to Charles Dickens. Cambridge, UK: Cambridge University Press, 2001. 136-151. Print.

Imperial Voices: Gender and Social Class among Shakespeare’s Characters, a Stylometric Approach

2015-03-12

“in the imitation of these twain–who as Ulysses says, opinion crowns with an imperial voice–many are infect.” –Nestor, Troilus and Cressida, Act 1, Scene 3.

The following describes an experiment in digital literary analysis, wherein dialog from Shakespeare plays was extracted, categorized, and statistically analyzed using a variety of methods. Dialog from Shakespeare’s kings, for instance, such as that of Lear and Claudius, was statistically compared with dialog from a number of other groups, like Shakespeare’s queens, servants, and fools, in an effort to computationally identify characteristic trends in the language of each group. This investigation concludes that relationships between these groups may be divined through this analysis, even suggesting class hierarchies and gender relations, but that these results are tenuous at best, and more fine-tuned analysis is required.

Extraction

Extracting dialog from a plain-text document would be a labor-intensive manual task, but an electronic text marked up in TEI XML is designed for easy computational manipulation of its elements. Thanks to the markup efforts of organizations like the Wordhoard Shakespeare, the Monk Project, and the Folger Shakespeare Library’s Digital Texts project, many classic works have been transformed into this machine-readable form, including 42 of Shakespeare’s plays. Since the dialog of these plays is assigned metadata that attributes it to its speaker, it thus becomes possible to automate the extraction of dialog. Each spoken line is rendered in XML roughly like this:

<sp who="Hamlet">
        <speaker>Hamlet</speaker>
        <l xml:id="sha-ham301055" n="55">
          To be, or not to be: that is the question:
        </l>
</sp>

Each line is thus enclosed in <sp> tags, each with the who attribute assigned to unique, regularlized XML IDs corresponding to each character in Shakespeare. Since Hamlet is the only character named Hamlet in Shakespeare’s plays, his XML ID is simply Hamlet, but other characters are given unique IDs. Since there are multiple ghosts in Shakespeare’s plays, for instance, the ghost of Hamlet’s father is given the XML ID ham-ghost. in the Wordhoard TEI rendition.

Dialog Extraction

The Folger Shakespeare Library TEI files are extremely meticulously edited, with emendations and textual differences marked up, as well as metadata about the characters, such as their sex. These would be ideal documents to work with, except that the depth of the markup makes them unwieldy in more than one respect. For instance, whereas the Wordhoard Shakespeare dialog, as illustrated above, is marked up on the level of the line, the Folger Shakespeare TEI marks up every word, space, and punctuation mark of the text:

<sp xml:id="sp-0006" who="#FRANCISCO-HAM">
    <speaker xml:id="spk-0006">
        <w xml:id="w0000570">FRANCISCO</w>
    </speaker>
    <ab xml:id="ab-0006">
    <lb xml:id="lb-00051"/>
        <join type="line" xml:id="ftln-0006"
        n="1.1.6" ana="#verse" target="#w0000580
        #c0000590 #w0000600 #c0000610 #w0000620
        #c0000630 #w0000640 #c0000650 #w0000660
        #c0000670 #w0000680 #c0000690 #w0000700
        #p0000710"/>
        <w xml:id="w0000580" n="1.1.6">You</w>
        <c xml:id="c0000590" n="1.1.6"> </c>
        <w xml:id="w0000600" n="1.1.6">come</w>
        <c xml:id="c0000610" n="1.1.6"> </c>
        <w xml:id="w0000620" n="1.1.6">most</w>
        <c xml:id="c0000630" n="1.1.6"> </c>
        <w xml:id="w0000640" n="1.1.6">carefully</w>
        <c xml:id="c0000650" n="1.1.6"> </c>
        <w xml:id="w0000660" n="1.1.6">upon</w>
        <c xml:id="c0000670" n="1.1.6"> </c>
        <w xml:id="w0000680" n="1.1.6">your</w>
        <c xml:id="c0000690" n="1.1.6"> </c>
        <w xml:id="w0000700" n="1.1.6">hour</w>
        <pc xml:id="p0000710" n="1.1.6">.</pc>
    </ab>
</sp>

This level of detail allows for great precision in cases where, say, one would need to mark up textual difference on the level of a single character, but it makes these texts much more difficult to work with on the macro level, mostly owing to their size. At several megabytes each, searching them, displaying them, and transforming them in the browser takes a considerable amount of time. The XSL transformation that converts the XML into XHTML for display in the browser, for instance, takes a modern browser on fairly capable hardware around 1-2 minutes to display.

Instead of using the Folger Library TEI files, which required a considerable amount of XSL transformation, this approach used Wordhoard Shakespeare TEI “unadorned” files. Unlike the Folger files, these are not as heavily marked up, and so they are much easier to work with. Furthermore, there are 42 plays in the Wordhoard corpus, whereas there are fewer than half that number currently in the Folger corpus. Some useful metadata, such as the sex of the characters in the cast list, are missing from the Wordhoard files, but the usefulness of their brevity outweighs any inconvenience caused by that omission.

To parse the XML, a python command-line program was written. The advantage of command-line programs is that they are interoperable–the output of one program can be easily channeled to the input of another by using the | operator, for instance. Similarly, the output of a program can be written to a file using >, or appended to a file using >>. This allows the user to write a single command that would do the work of lots of pointing and clicking, and it allows for the user to interface that command with any other existing command-line program in the operating system. Perhaps the biggest advantage of this approach is that most command-line shells, like BASH, which ships as the standard terminal in most varieties of Linux and Mac operating systems, is that it supports wildcard expansion–using *.xml in place of a filename tells the shell that you want to run your command over all the files containing the extension .xml, which is incredibly useful for working with large numbers of files, such as 42 Shakespeare plays.

This python script parses all the XML files it is given, and iterates over them, looking for sp tags with who attributes that match those in characters.txt, or another user-specified file containing a comma-separated list of characters’ XML IDs. To extract all of Ophelia’s lines from Hamlet, for instance, the user first creates a file named characters.txt and gives it the content Ophelia:

echo "Ophelia" > characters.txt

Then the user runs the python program on the ham.xml TEI file of Hamlet, to extract the dialog and save it to a text file called ophelia-dialog.txt:

python parse.py ham.xml > ophelia-dialog.txt

To extract all of Falstaff’s dialog from all 42 of the Wordhoard Shakespeare plays (which is easier than specifying the three in which he appears), and save that dialog to a file called falstaff-dialog.txt, these commands can be run from within a directory containing all of the XML files:

echo "Falstaff1" > characters.txt
python parse.py *.xml > falstaff-dialog.txt

If the program is given a characters.txt file (or another file which the user specifies) containing a comma-separated list of characters, it outputs the combined dialog of all of those characters. It was therefore possible to extract dialog from large groups of characters in this way.

Character Lists

To comparatively analyze the dialog of Shakespeare’s characters, it was first necessary to assign categories for these characters. This proved to be a difficult problem, since this categorization can be very subjective–who is considered a comic figure by one critic may well be considered tragic or tragicomic by another, and the categories of these characters may (and often do) shift throughout the course of a single play. The categories I chose were “kings,” “queens,” “servants,” “gentlemen,” “gentlewomen,” “officers,” and “fools.” Although categories, none of them are precisely “categorical,” which is to say, none of them are completely unified and unambiguous.

To choose these characters, the Shakespeare character search engine at the Electronic Literature Foundation was used, which accepts a search term and outputs a list of characters in whose role descriptions that term occurs. These data had to be trimmed somewhat, since the word “king” appears in the description “servant to the King,” and thus the results of a search for the word “King” did not always return a list of kings. The “kings” and “queens” categories were probably the clearest to delineate, since most of the characters in these categories carry the titles of “king” or “queen.” Even with those, however, there were plenty of problematic cases. Would Oberon, the king of the fairies from A Midsummer Night’s Dream be considered a king, for the purposes of this study? If this study aims to discover class-specific language in Shakespeare’s works, then it could be argued that supernatural kings don’t represent this class structure, but rather that of the supernatural world they inhabit. On the other hand, if one interprets the supernatural world of the fairies as a projection of the earthly world, with all its hierarchies intact, then it is entirely reasonable to include Oberon among the kings. A complete list of all the characters is available in Appendix I.

The next category, “servants,” was populated by searching the character descriptions for the term “serv,” which returned words such as “servant” and phrases such as “in the service of.” All characters named “Servant” were included, as well as any any characters described as “servant to” another character. All characters described as “in the service of,” however, were not included, since they were almost all described as “gentleman in the service of” or “gentlewoman in the service of”, and therefore placed in the respective categories of “gentleman” and “gentlewoman.” Those categories, “gentleman” and “gentlewoman” were populated by searching for those terms.

The “officers” category contains officers, soldiers, “gaolers,” a sheriff, and a sentinel. Although these are slightly different roles, they are treated here as a single group, since that is often the way they are treated in the cast of characters–in Antony and Cleopatra, the group is described as “Officers, Soldiers, Messengers, and other Attendants.” Messengers were not included in this category, however, and neither were “other attendants,” because the goal here was to aim for a group that will be relatively unified in tone, which is this case hopes to be paramilitary.

The category “fools” was the most problematic. There aren’t enough fools in Shakespeare to make for a category large enough for statistical analysis, and so I’ve included both fools and clowns here. Although he admits they are related categories, Stanley Wells asserts that “Modern criticism distinguishes between the naturally comic characters, or clowns, such as Lance, Bottom, Dogberry. etc., and the professional fools, or jesters, such as Touchstone, Feste, and Lear’s Fool” (“Clown”). Whereas this may be a crucial distinction for criticism, I have nonetheless grouped the two together for pragmatic reasons. A more nuanced study might separate the two, but this investigation is focused on macro-categories for the sake of efficiency. Toward this end, I’ve used the list of fools from Wikipedia’s page (“Shakespearean Fools,” somewhat spurious, in parts, though inclusive), and supplemented it where necessary with search results for “fool” and “clown” from ELF.

The biggest challenge with these categories is the variety of lengths among them. “Kings” was the largest category, containing 99,540 words, or over three Hamlets in length; “queens” is the second biggest, with 25,934 words; “gentlemen” is next, with 14,788; followed by “servants” at 9,554; “gentlewomen” at 3,530; and “officers” at 2,222.

Words Per Category

That the smallest category contained only around two thousand words, and the largest contained almost 45 times that amount, meant that, for one, the probability of a given word appearing in the “kings” category did not depend solely on voice or stylistic concerns, but on the character’s opportunity to speak. We can say, (as I will soon, in fact, be saying) that certain words are characteristic of Shakespearean officers, and that there are certain words which they do not say, but much of this is dependent on the amount of text. Any judgement about the treatment of class or gender in Shakespeare, based on a computational analysis such as this, is in danger of conflating misrepresentation with underrepresentation. Servants may very well refer to their “master” and “mistress” frequently, but perhaps it is more that we rarely get a chance to hear them speak to the extent that we hear the kings and queens. Granted, a lot of this discrepancy can be attributed to the fact that so many of Shakespeare’s play are histories. One could hardly expect plays with titles like Henry VIII or Antony and Cleopatra to revolve around anyone but their titular roles.

With the characters tabulated into categories, the tables were then exported into CSV text files containing comma-separated lists of XML IDs. Here is, for example, the contents of queens.csv:

Gertrude,Hippolyta,Titania,Cleopatra,CymbelineQ,
Elinor,Tamora,Hermione,Valoislsab,WoodvileEl,
MargaretQ,AragonCath,BoleynA,ElizabethBav

This list was then used to extract dialog from the plays by running the command:

python parse.py -c queens.csv *.xml > queens-dialog.txt

Cleaning Up the Texts

One of the features of this Python script is that it keeps the text encoded in unicode throughout. This means that any characters that do not appear in ASCII, like em-dashes or curly quotation marks, will not be mangled when they are extracted. Unfortunately, this also means that other tools, such as the University of Newcastle’s Intelligent Archive, will not be able to adequately interpret these unicode characters. When The Intelligent Archive encounters straight quotation marks, for instance, it dismisses them and does not consider them part of the word, but when it encounters curly quotation marks, it converts these into unusual characters like “œ” and leaves them attached to the word, thus generating wordlists containing corrupted words like “œthy.” To correct this, this command was run in the working directory:

for a in $(find . -name '-dialog.txt') ;
    do iconv -f utf-8 -t ascii -c "$a" > "$a.ascii" ;
done

This loop iterates over all the files ending in -dialog.txt (in this case, all the extracted dialog), runs the Linux program iconv on those files, and then writes them to new files, appending the .ascii extension. The -c flag tells iconv to throw out any characters that can’t be converted into ASCII, which solves the issue with the Intelligent Archive. Em-dashes and a lot of other punctuation are removed from the texts during this process, but since punctuation isn’t an object of this study, that doesn’t present a problem.

Statistical Analysis

Several varieties of computational statistical analysis were performed on the resulting texts. First, the texts were tokenized and split into segments using The Intelligent Archive. A few methods were used at this stage to lessen the impact of the word count discrepancy. In one set of tests, the texts were limited to the word count of smallest text, by setting the option “first X words of text.” The problem with this method, however, is that the first 2000 words from a king are likely to come from a single play. The analysis would then be comparing all the dialog of the servants and fools with, say, the character Henry VIII, instead of all the kings. This is a major problem, but also one that mostly affects the kings category. The smallest category isn’t affected by this at all. Another technique used to lessen this effect was to divide the text into 500 word blocks, and then choose a random set of four to eight blocks for analysis. Yet another technique was randomizing the lines in the texts themselves by filtering them through a command in BASH:

for a in $(find . -name '-dialog.txt') ;
    do cat "$a" | sort -R > "$a.random" ;
done

This loops over all the extracted dialog text files, writes their contents to the standard output with cat, sorts the lines by random (-R) criteria with the sort command, and writes them to new files.

All of these techniques are problematic in some way. If, say, the analysis shows that the word “and” is the most common word in the servants category, and that number is much lower or higher in another category, can we say with any certainty whether this phenomenon is an effect of style or voice? It seems equally as likely that, even despite these efforts to choose equally-sized random samples of each block of dialog, the fact that certain characters are given more time to speak will skew the results in ways that can’t be easily factored out. This caveat is seemingly supported by the fact that the results differ greatly when these methods of randomization and segmentation are changed. I will not pretend, then, that any of these results are definitive; rather, that they are useful as exploratory experiments only.

First 2000 Words of Each Text; 800 Most Frequent Words

The above figure shows a dendrogram generated from a cluster observations analysis performed on the proportions of the 800 most frequently used words in each text, using the first 2000 words of each text. These results are very suggestive. Broadly, one could say that the leftmost group of six represents those employed at court, or in a royal, aristocratic setting, and the outlier, “officers,” represents those employed largely outside of court–members of the working class. Within the aristocratic group, there are two subgroups: a noble class and a servant class. That gentlewomen are paired with servants here is less of a statement about gender as it may seem–“gentlewomen” is a very sparse category, represented by only a few characters, all of whom seem to take a somewhat servile role by definition. (All but one of the gentlewomen are described as “attending on” a lady of some sort, such as Margaret, a “gentlewoman attending on Hero,” from Much Ado About Nothing.) As previously noted, however, the fact that this analysis only used the first non-randomized 2000-word segment from each text means that there’s a high likelihood that the larger categories only contain dialog from one or two plays. This can account for some, though not all, of this grouping.

First 4000 Words; 1000 Word Segments; 300 MFW

This figure represents an experiment conducted with the first 4000 words of each text, with each text broken into 1000-word segments. The texts were segmented like this in order to verify that the statical analysis was working–if Shakespeare’s queens sound roughly alike, then they should appear in similar areas of these charts. Since the “gentlewomen” category only contained around half of the required 4000 words, both the “gentlemen” and “gentlewomen” categories were eliminated. Here, principal component analysis (PCA) was used to generate two factors based on the similarity of the 300 most frequently used words in each text. Again, the results are very suggestive. On the whole, there are four main areas, in which the characters appear next to each other rather nicely. In the lower left quadrant, there are all the “kings” texts; in the upper center, there are all the “queens” texts, and in the lower right, there is an area where an area of “fools” texts collides with one of servants. One way to interpret these axes is that they might represent class and gender. The location of the queens texts seems to say that the y-axis is showing gendered stylistic differences. Similarly, the location of the fools suggests that the x-axis represents class standing, with the kings at the far end of the aristocratic spectrum, the servants in the middle, and the fools (and, it should be remembered, clowns of all sorts) on the opposite end. These same experiments, when conducted with 800 of the most frequently used words, show similar trends.

In an attempt to correct some of the skew caused by the differences in text length, a comparison was conducted between only the minor characters, all of whom have a similar amount of text. Although gentlemen are grouped closer to the servants than gentlewomen in this experiment, the remaining results, shown in Figure 4, are roughly the same as before. As one would expect, fools and officers, the two most anomalous groups according to their eccentricity and social standing, are the statistical outliers in this chart. Once again, however, the fools, owing perhaps to their standing within the court (though admittedly, this doesn’t apply to the clowns), are shown to be more similar to the gentlemen than are the officers.

Full Texts; Minor Characters

When the texts of these categories were randomized (filtered through sort -R), a slightly different picture emerged, as shown in the figure above. Again, the first 2000 words of these texts were used, and then subdivided into 1000-word blocks, and the first 800 most frequently words of the resulting set were analyzed. On the whole, these parts appeared close to each other in the PCA score plot. The categories, however, showed different affinities. This time, kings and queens appeared much more similar. The outliers in this projection are again officers, but this time, also gentlewomen. That these two categories are the ones with the least text is perhaps telling–maybe these results have been skewed in some way as a result of that fact.

A tempting way to interpret this cluster diagram is as a sphere of influence. With the kings and queens in the upper-right quadrant (the relative position is computed, whereas the absolute position is somewhat arbitrary), they are surrounded by an entourage of characters with varying degrees of influence on them. The officers have the least influence on the kings and queens, followed by the fools and gentlewomen.

Randomized Text; 800 MFW

A more convincing picture might be found in a cluster analysis of the same data set, shown in the figure above. As before, the kings, queens, and gentlemen all group together, and the gentlewomen and the servants make a pair. This time, though, the fools and clowns are the outliers, followed by the officers at the opposite end. This result is identical between sets of 800 and 300 of the most frequently used words. This seems to confirm most of the interpretation previously suggested of the first figure.

Randomized Text; 800 MFW

These results all vary considerably. This variation has the double purpose of highlighting recurring motifs as potential candidates for evidence, and showing the instability of these analyses. What almost all of these experiments show is that the dialog of Shakespeare’s kings and queens is statistically dissimilar to the dialog of other characters. In most cases, the kings are queens are more like each other than any other characters. Most of the statistical divisions shown here take place along class lines. A more in-depth analysis of the language characteristic of these categories might help to shed some light on the exact nature of this division.

Distinctive Language Analysis

Next, the texts in these categories were compared with one another using David Hoover’s Full Spectrum Spreadsheet. This program identifies words that appear frequently in one text and rarely in another, and sorts them according to this ratio, in order to find words that are distinctive of each category. The resulting wordlists are indications of the language of their categories, insofar as they distinguish themselves from the categories with which they are being compared. The distinctive words of Shakespeare’s kings, for instance, when compared with that of fools, is very different from the words distinctive of kings when compared to queens. There are many place names and names of people in this former group, for instance–“Percy,” “Harry,” “Douglas,” “Wales,” “Westmoreland” and “Northumberland” all appear among the top 25 most distinctive words. These proper names appear much less frequently in comparison with other categories of characters. This would seem to suggest that the fools and clowns of Shakespeare speak more in abstractions than the kings. If one remembers the Fool’s song from King Lear, “Have more than thou showest, / Speak less than thou knowest,” this rings true.

When compared with officers, the words distinctive of kings are very different. The most distinctive word, somewhat unsurprisingly, is “love”–it would be an unusual soldier that would deliver a soliloquy about love, yet this would not be impossible for a Shakespearean king. Related words in this kings’ list include the sentimental terms “heart” at #17, “sweet” at #21, and “gentle” at #32–one would naturally not expect “sweet” and “gentle” to be words that a soldier would speak. There are also many kinship terms in this kings’ list–“son” at #7, “father” at #11, “cousin” at #43, and “wife” at #70. This suggests that the officers investigated here do not discuss their families–they are perhaps presented as if they have no families, as if duty were their only concern.

This sensitive image of Shakespeare’s kings changes when one compares them with queens. There, the words distinctive of kings are words of hierarchy–“lords” at #2, “Earl” at #10, “Duke” at #13; and militaristic words: “sword” at #23, “march” at #39, “war” at #47. There are still kinship terms, but they almost all denote male relatives: “brother” at #12, “uncle” at #20, and “father” at #40. In contrast, the kinship words that appear in the list characteristic of queens are all female: “wife” at #23, “daughter” at #24, and “woman” and “women” at #27 and #28. In fact, the list of most distinctive words in the kings’ dialog does not contain female pronouns. The word “she” appears high on the fools’ list (#4), the servants’ list (#9), and the queens’ list (#18), but does not appear in any of the kings’. The same is true for the word “her.” This phenomenon suggests that kings’ and queens’ domains are roughly gender-segregated into men’s and women’s spheres. Male royalty are predominantly concerned with other men, and female royalty are predominantly concerned with other women.

The words distinctive of kings, when compared with those of servants, are also telling. The three words most characteristic of kings in this context are “thou,” “thy” and “thee,” pronouns which denote familiarly or condescension. Since servants in Shakespeare don’t often have the luxury of condescension, the analogous words in the servants’ category are honorifics suggestive of deference–“master” (#1) “madam” (#4), and “mistress” (#8). The servants category features many such servile words when compared to the kings’ words: “please” (#3) and “sir” (#2) being the most frequent, and also “patience,” “service” “humbly,” and “honest” in the top 25. Interestingly, while the servants category features the plural “gods,” the kings category with which it is juxtaposed features the word “god” and the possessive “god’s.” This same phenomenon happens among the distinctive words of kings when compared with queens–“god” is ranked #5 for kings, and “gods” #24 for queens. Unfortunately, since the word “god” is rarely used to refer to the Christian diety, we cannot use these data as indicative of mono- or polytheistic settings.

Amazingly, none of the contractions that appear in these distinctive word lists are distinctive of kings. The word “‘tis" appears as distinctive of servants (#6), fools (#7), and queens (#51), but not kings. The fools’ wordlist contains “o’,” “we’ll,” “that’s,” and “‘a,” among others; the officers’ contains “what’s” and “let’s”; the queens’ list contains “he’s” and “there’s,” and the servants’ list contains “I’ll,” “that’s,” and “ta’en.” None of these words appear in any of the four kings’ lists. Are contractions like these spoken on a level of informality typically not seen in kings (but not, strangely, not unfamiliar to queens)? The OED does not help to answer this question, and neither does Blake’s dictionary of Shakespearean informal language (72, “Contractions”), so perhaps this is a matter better left to linguists or specialists in Renaissance language.

On the whole, categories’ distinctive words reflect the typical picture of those categories. “Ass” and “jest” appear in the fools’ (jesters’) list; “watch,” “prisoner” and “guard” appear in the officers’ list; and “crown” is a distinctive word for kings in three lists. In some cases, though, words appear as distinctive of unexpected categories. “Power” is not a word one would associate with gentlewomen, yet it is the third most distinctive word for that category, as compared with gentlemen. A concordance of the word “power” in the gentlewomen dialog was conducted to get to the root of this mystery:

grep power gentlewomen-dialog.txt > power.txt
while read p  
  do grep -B 20  "$p" *.xml
done < power.txt

The first of these lines searches gentlewomen-dialog.txt for the word “power,” and outputs the results to a file called power.txt; the subsequent lines read the lines of power.txt individually and search all the plays (all the XML files) for the lines in the file, giving 20 lines of context (-B 20) so that the speaker may be identified. The output from these commands reveals that the speaker of this word “power” is, in all cases, Helena from All’s Well That Ends Well–a passionate character, indeed.

Other unexpected words from these lists include “love” at the top of the king’s list. This isn’t a word one would expect to be characteristic of Shakespeare’s kings, but when compared with officers, it is their most distinctive word. Although “blood” is a characteristic word for kings in three lists (#18, v. fools; #3, v. gentlemen; #11, v. servants), it also appears as the nineteenth most distinctive word of gentlewomen. Again, a concordance using the BASH commands above reveals that “blood” is also, in all three cases, Helena’s word, a fact which remains consistent with her fiery nature.

Also of note in these wordlists is the fact that the word “nay,” while it appears high on the lists of fools, queens, servants, and gentlemen, doesn’t appear at all in the list of distinctive kings’ words. Although a concordance of the dialog shows that the word appears several times among Shakespeare’s kings, the fact that this word appears more frequently among the other groups of characters seems to suggest that the kings are relatively affirmative, whereas the others could rightly be called “naysayers.” It is tempting to take this interpretation further, but the presence or absence of such a versatile word could not with certainty lead to a clear reading.

Conclusion

Here we have seen that class structure in Shakespeare, although by no means unified and unambiguous, can be revealed to some extent by a statistical comparison of categories of dialog. Similarly, gender differences–those between kings and queens, for instance–can also be shown using PCA analysis. Many of the distinctive words found with the Full Spectrum Spreadsheet are unsurprising for their categories, yet many unusual or surprising words also appear, a phenomenon which leads to insights about particular anomalous characters. A more qualitative inquiry might examine in greater detail the anomalies found in these lists, and contextualize them more among the personalities of their particular characters.

Although this investigation is not traditionally conclusive, it has raised some questions which might lead to further study. The limitations of such macro-oriented work as this, which deals with dialog on the level of Shakespeare’s entire dramatic oeuvre, also present opportunities for more minute, focused work. One such study, for instance, might separate fools from clowns, and compare the language used in both. Another might compare the speech of officers with that of soldiers, or that of princes with that of kings. Many of the observations of language mentioned here–such as the use of contractions among non-royal speakers–deserve further inquiry.

Bibliography

Blake, N F. Shakespeare’s Non-Standard English: A Dictionary of His Informal Language. London: Thoemmes Continuum, 2004. Ebrary. Accessed 21 May 2013.

Wells, Stanley. “Clown.” A Dictionary of Shakespeare. Oxford, UK: Oxford University Press, 1998. Oxford Reference. 2003. Accessed 21 May. 2013.

Presentation Slides

Slides from this paper’s presentation at the workshop Computer-Based Analysis of Drama, at the Bavarian Academy of Sciences and Humanities, in March 2015.

A Proposal for a Corpus Sharing Protocol

2015-03-15

Digital humanists working in computational text analysis need a better way to share corpora. Following is a rough sketch of a way to share texts in way that facilitates collaboration, provides for easy error correction, and adheres as much as possible to decentralized, open-source, and open-access models.

Problems

The Problem of Corpus Availability

Franco Moretti gave a talk at NYU two weeks ago, detailing digital analyses he had conducted on large numbers of novels. The handout he circulated beforehand, whose authors also included Mark Algee-Hewitt, Sarah Allison, Marissa Gemma, Ryan Heuser, and Hannah Walser, included a section about the difficulty of obtaining many of these corpora:

In August, requests were sent to Hathi and Gale – with both of which Stanford [has] a solid, long- standing agreement – for their 300 volumes; two months later, we haven’t yet received anything. Of the 100 existing only in print, about half were held by the British Library, in London, which just a few months earlier had kindly offered the Literary Lab a collection of 65,000 digitized volumes from its collections; unfortunately, none of the books we were looking for was in this corpus. The special collections at UCLA and Harvard (which held about 50 of the 100 books) sent us a series of estimates that ranged (depending, quite reasonably, on the conditions of the original, and on the photographic requirements, that could be possibly very labor-intensive) from $1,000 to $20,000 per novel; finally, six novels were part of larger collections held by Proquest, which offered us a very generous 50% discount; even so, those six books would have cost us $147,000, or $25,000 per title.

It is unreasonable that a text that has long been out of copyright would cost researchers $25,000, and that this represents a 50% discount on the usual rate. The very premise that public-domain works are held captive by for-profit institutions is one that seems at odds with the ideals of science and academia—it greatly obstructs the repeatability of experiments in text analysis, and sets the barrier for entry into the field unnecessarily high. I asked Prof. Moretti if he had any ideas about how to minimize this problem, and he answered that there were legal reasons why he couldn’t provide texts from his corpus to other researchers. Stanford had particular agreements with these data providers, and if I wanted to use that data, I would first need to be affiliated with an institution that could strike corpus sharing agreements with the data providers, and then make agreements with the data providers through that institution. Those are major hurdles for an independent scholar who wishes to do large-scale text analysis.

The problem of corpus availability is deep and pervasive. At last year’s Digital Humanities conference, I saw a fantastic presentation by the Stanford Literary Lab on the detection of poetic meter. Since they had analyzed a set of texts from the Literature Online (LION) database, I asked them how they were able to get permissions for that data set. They replied that they’d simply asked ProQuest for it. With this in mind, I contacted John Pegum at ProQuest, in the hopes of obtaining texts I might use for large-scale analyses. His reply was polite and thorough, but concluded that I very likely couldn’t afford to pay for this privilege. When I responded with a few other options—one of which was working directly on a server of their choosing, without making a copy of the text—I never received a reply.

Even some of the best text repositories, like the Oxford Text Archive, are designed in such a way as to make them prohibitively difficult to use for macroanalysis. Many of the texts in the Oxford archive are behind a permissions wall, and seemingly by default. To gain access to a text, a researcher must apply to Oxford and specifically request a particular text. If that scholar is then interested in analyzing hundreds of texts, that would require hundreds of requests.

This problem was one of the major topics of discussion at the workshop Computer-Based Analysis of Drama I attended last week in Munich. Many of the presenters used texts from the TextGrid repository, and it was suggested that this could be a platform for the sharing of corpora among researchers. Yet TextGrid is apparently losing funding soon, and might go down. So how can we find a common platform for sharing texts, so that others might benefit from them?

The Problem of Corpus Immutability

Another issue with the current paradigms for the sharing of corpora is the problem of corpus immutability. If texts are only available from a centralized source, then new versions of that text depend on the maintainers of that centralized source for all changes. Corpora of scanned texts are full of OCR errors, and there are few opportunities for user corrections of those errors. Project Gutenberg handles this problem with volunteer proofreaders, but the fundamentally static nature of their web pages makes it difficult for those outside of their infrastructure to contribute textual corrections. At Computer-Based Analysis of Drama, Martin Muller of Northwestern University demonstrated a fantastic system for crowdsourced correction of these errors, Annolex, that is the one of the best such systems I’ve seen. Yet, at the moment, it only works on a select number of early modern English texts. How can we make this sort of crowdsourced error correction even more democratic, and even more distributed?

Summary

What is needed is a platform for:

easily and instantly uploading corpora for sharing among scholars and researchers
tracking the versions of those corpora
collaboratively correcting texts in those corpora that contain OCR or other errors
tracking problems with the texts, and openly discussing those problems
allowing for the machine-readability of these corpora with a REST API
adherence to an open-source and open-access philosophy
accomplishing all of the above with minimal programming

A Proposed Solution

Such a platform as described above has already long been in use among coders, and would need only minor usage modifications to be used as a plaform for sharing corpora. For those that aren’t already familiar with it, GitHub is a repository of repositories, tightly integrated into the version control system git, where coders can upload their projects and collaborate on others’ projects. Each repository features an issue tracker, wiki, and a statistics suite that can help you to see at a glace the history of a project’s versions. It works very well already for handling computer code, and would work just as well for handling text corpora.

One of the best things about GitHub is its push-button forking ability. To create a copy of any existing repository, just click the “fork” button on that repository’s home page. This creates a copy of the entire project that is now in your user account. You can make any changes you like to your copy (even without leaving the browser), and when you’re finished with your contributions, you can submit a pull request to the original repo to request that your changes be merged into the original project. This allows for distributed error-correction, maintaining all the while the history of the contributions to the text.

The web app generator platform Yeoman uses GitHub for their decentralized generator collection. Rather than manually maintain a list of user-created generators, they dynamically search GitHub for any repository with certain metadata, like the prefix generator- or the keyword yeoman-generator. Then they sort those repositories by the number of times they’ve been starred. This approach ensures that their list of generators is always up-to-date, and sorted fairly democratically. This model could be applied to corpora.

I propose that we upload our corpora to GitHub, with repositories named with the prefix corpus-. For example, if I am uploading a corpus of Shakespeare TEI files, I might name my repository corpus-shakespeare-TEI. That repository could contain a set of TEI XML files, or a TEI corpus file. This would ensure that the repository is as easily machine-readable as it is human-readable.

Eventually, a scraping engine might be built which could, like Yeoman’s generator list, dynamically search GitHub for any repo beginning with corpus-. Then, performing large-scale text analyses on any set of these corpora would be orders of magnitude easier than manually assembling a database of texts.

Possible Concerns

Although this is a decentralized solution when compared with other, more static, repositories, GitHub is still a centralized hub.
- This might be solved by dynamically pulling repositories from a number of similar git sources, like BitBucket.
As Dario Kampkaspar reminded me, git has peculiarities with diffing whitespace that might make it less than ideal for tracking changes in XML.
- This might be solved by turning on the ignore whitespace flag in git.
GitHub has a repository size limit of about 1GB, with a soft limit of about 100M per file.
- This might be solved by breaking up repositories into submodule repositories. One parent corpus repo can hold several submodule repos.

Discussion

What do you think? Is this a protocol you might be interested in adopting? Why or why not? Please leave comments below.

A Bookmarklet For Creating Annotags

2015-04-25

At I Annotate this weekend, I made a bookmarklet that will allow you to generate an Annotag from a book’s bibliographic entry:

Tweet This Book

To use it:

Drag the button above onto your bookmarks bar.
Visit a website that has books. It currently works with Worldcat and Project Gutenberg.
Click the bookmarklet.
Add a location suffix to the hashtag, if you want, like :p26.
Tweet a comment about the book.

The bookmarklet extracts book identifiers from your URL, encodes it in base-62, and takes you to Twitter, where this information will be be waiting for you, encoded as a hashtag. Now it’s even easier to tweet about books!

If you’d like to help improve the code, it can be found here on GitHub.

A Macro-Etymological Analysis of James Joyce’s A Portrait of the Artist as a Young Man

2015-05-10

[This is a preprint of my paper, published in Reading Modernism with Machines, by Palgrave Macmillan in December 2016, reposted from the original draft.]

The English language is a palimpsest, bearing traces of the languages it has contacted. French, Latin, Ancient Greek, and Irish are among the languages that have contributed words to English, and these ancestor languages comprise modes of expression that recall the contexts of their acquisition. When a writer chooses the word “chew” over “masticate,” or “enchantment” over “spell,” what does that decision indicate? How can we measure these stylistic vectors? This study uses a computational analysis of the etymologies of words in James Joyce’s novel A Portrait of the Artist as a Young Man in order to identify etymological registers, resonances, and levels of discourse. In particular, this study will attempt to measure the maturing language of the novel’s protagonist Stephen Dedalus through his use of Latinate words, and to identify ways in which macro-etymological signals reflect structural elements of the novel.

The works of James Joyce are ideal for macro-etymological analysis. Joyce was famously multilingual, and many see his novels as a crescendo of linguistic experiments. In the words of Laurent Milesi, “Joyce’s oeuvre is best seen as constantly trying to inform an evolutive linguistic poetics” (1). Finnegans Wake, the culmination of his career in literary experimentation, is arguably unparalleled in its paranomasia and polysemy—in its composition, Joyce employed word roots from forty languages. But this impulse was present in Joyce’s early works, as well—his words are deliberately chosen to suggest their ancestors and cognates. They are serio-comic puns made to extend along etymological axes to new meanings in other languages.

Joyce himself was keenly interested in etymology. In his early critical essay “The Study of Languages,” he argues that “in the history of words there is much that indicates the history of men, and in comparing the speech of to-day with that of years ago, we have a useful illustration of external influences on the very words of a race” (“The Study of Languages” 15). Joyce’s interest in etymology was that of the application of word history to English usage. He argues for the study of Latin, of which “a careful and well-directed study must be very advantageous,” because it “acquaints us with a language, which has a strong element in English, and thus makes us know the derivations of many words, which we then apply more correctly and which have therefore a truer meaning for us” (16). This itself may be an etymological pun, since the word etymology is derived from the Greek etumon for “true.” It follows that, by studying the etymologies of Joyce’s words, we might discover more of the diversity of what Joyce considered to be “truer.”

In Stephen Hero, an early version of Portrait, Stephen Dedalus is described as having “read Skeat’s Etymological Dictionary by the hour” (Stephen Hero 32). Although purely autobiographical readings of the two novels remain controversial, we may safely assume that Joyce had also read and loved this work. In an etymological reading of the Dubliners story “Ivy Day in the Committee Room,” Michael Brian argues that Joyce had a such a “detailed and profound knowledge” of Skeat’s dictionary, and that it had such an influence on this story that “one could say [it] is written in Skeatish” (220). Stephen Whittaker takes it as obvious that Joyce was intimately familiar with Skeat, to the extent that it is more interesting to him whether he worked from the third or fourth edition of the dictionary (178).

In Portrait, Stephen routinely muses about words, considering their sounds, shapes, and beauty. “Suck,” Stephen considers “a queer word” (8), but “wine” he thinks “a beautiful word” (39). Seeing the word “fœtus” carved into a desk “startle[s] his blood,” (75) but upon hearing Cranly say “mulier cantat,” he remarks on the “soft beauty of the Latin word” (205). It is this logophilia that justifies, in part, the following quantitative methodology, even at the risk of decontextualizing individual words. “One difficulty in esthetic discussion,” Stephen seemingly cautions us, “is to know whether words are being used according to the literary tradition or according to the tradition of the marketplace” (157). This is one of the difficulties of computational literary criticism, as well—the so-called “bag of words” model of digital text analysis cannot sufficiently account for context. Conversely, Joyce’s attention to words and their histories valorizes an investigation such as this.

Marjorie Howes argues that Joyce “consistently embedded the complexities of colonialism and nationalism in particular words,” and cites his use of ivory, as a spiritual metaphor (Mary is a “tower of ivory”), a sexual image (Eileen’s hands were like ivory), and a colonial commodity (255). Stephen daydreams about this word, and imagines it prisming: “The word now shone in his brain, clearer and brighter than any ivory sawn from the mottled tusks of elephants. Ivory, ivoire, avorio, ebur” (P 150). The splitting of the word into its four cognates approximately traces its etymology, from English to (Norman) French to Latin. In fact, each of these four forms for “ivory” are given in Skeat’s dictionary and in the OED in precisely this order, although this is not their direct lineage. This word history, therefore, traces a path that locates language among nations, and finds “the history of men” in “the history of words” (“The Study of Languages” 15). This vector is also Joyce’s own biographical path of exile, from Ireland to France and finally to Rome, where he first began to rework Stephen Hero into Portrait.

The narrative style of Portrait is another of its properties that makes it appropriate for macroanalysis. Whether called Erlebte Rede, or, in Flaubert’s term, le style indirect libre, it is style in which the boundaries between the narrator’s language and the characters are blurred. When Wyndham Lewis disparaged Joyce’s phrase “Uncle Charles repaired to the outhouse,” complaining that “people repair to places in works of fiction of the humblest order,” Hugh Kenner responded by explaining that “‘repaired’ wears invisible quotation marks. It would be Uncle Charles’s own word should he chance to say what he was doing” (17). Kenner thus dubbed this Joycean narrative technique the “The Uncle Charles Principle,” which he defines by explaining that “[Joyce’s] words are in such delicate equilibrium, like the components of a sensitive piece of apparatus, that they detect the gravitational field of the nearest person” (16). For Kenner, this style is primarily observed on the level of the word. “[Joyce] is not,” he writes, “like Beckett, an Eiffel nor a Calder of the sentence. The single word—‘repaired’; ‘salubrius’—is his normal means to his characteristic effects” (20). This might be because, as Joyce was aware, the histories of each word made them richly polysemous. This property of Portrait is one that makes macroanalysis meaningful—the histories of the individual words aren’t simply functional aspects of the language, but crucial stylistic and ontological units saturated with traces of their origins.

Since Uncle Charles himself made only a momentary appearance in Portrait, a more significant effect of the Uncle Charles Principle may be observed in the language of Stephen Dedalus, whether expressed directly or through the narrator. Stephen’s language, and therefore largely the language of the novel as a whole, begins with juvenile songs and ends with mature prose. The following experiment is designed to quantify that development, by analyzing each of the chapters of the novel individually. The initial hypothesis is that the macro-etymological analyzer will show an increase in proportions of words of Latinate origin throughout the course of the novel. This hypothesis is confirmed, but not without surprises.

The Experiment

The Macro-Etymological Analyzer is a web app written using a LAMP stack—Linux, Apache, MySQL, and PHP. It ingests a text, tokenizes it, and looks up each word in the Etymological Wordnet, a relational database created from Wiktionary data by the computer scientist Gerard de Melo. The program finds the first language ancestor of each word, and categorizes it according to language family. Since words of French origin and words of Latin origin often share roots—many English words come from Latin through French or Anglo-Norman—these are grouped together into the category “Latinate,” along with words of Italian or Spanish origin. Words descended from Old or Middle English, German, or Dutch are categorized as “Germanic”; words of ancient and modern Greek origin are denoted “Hellenic”; and words of Irish or Scottish origin are “Celtic.” The program then determines the proportions of words of each category1. A Portrait of the Artist as a Young Man contains 90% words of Germanic origin, 5% words of Latinate origin, and less than 0.1% each of Hellenic, Slavic, Iranian, Afroasiatic, and Celtic. A further 4% of the words in the text were not found in the dictionary, many of them proper names. These data alone are not very interesting, however, since we have nothing yet with which to compare them. We must therefore begin by calibrating the program.

Calibration

To find significance in these etymological signals, the Macro-Etymological Analyzer was trained on genres extracted from the Brown University Standard Corpus of Present-Day American English, a much-studied linguistic corpus of approximately one million words, created in the 1950s. The corpus is broken into genre categories such as “science fiction,” “belles lettres,” “humor,” and “news.” Each of these categories was extracted using the Python NLTK and analyzed. Figure 1 shows the occurrence of Latinate words in categories of the Brown Corpus. The genres are divided fairly cleanly between fiction and non-fiction, with the fiction genres “adventure” and “romance” on the low end of the spectrum, and the non-fiction genres “learned” and “government” on the high end. Strikingly, the genres “Lore” and “Religion,” which are arguably of ambiguous fictionality, fall in the middle. “Science Fiction,” which is probably the most non-fictional of the fiction genres, lies in the same quadrant, and exhibits the highest proportion of Latinate words of a fictional genre. Based on this calibration, we might say that high proportions of Latinate words (hereafter “L scores”) in Portrait would have a good chance of exhibiting styles similar to learned text, official documents, or non-fiction.

Figure 1: Brown Corpus: Latinate Words

Figure 2: Brown Corpus: Hellenic Words

Among proportions of Hellenic words, as shown in Figure 2, the picture is similar, but with a few key differences. Here, “religion” has a higher rank, and “government” a lower. Since Hellenic words represent such a tiny percentage of any given text, however—a total of 66 words for Portrait—we cannot treat measurements of this category as equally statistically significant. The same is even more true for proportions of words of Celtic origin, since only a single word was detected in that category. Germanic etymologies were inversely correlated with Latinate etymologies, and so these values are already roughly represented by L scores. Each of these categories deserves an in-depth discussion.

Languages

Latinate

The calibration experiments performed above suggest that high proportions of Latinate words are correlated with non-fiction and formal or authoritarian language. In part, this can be explained by the history of the introductions of Latinate words to English. Directly following the Norman Conquest of 1066, French became the language of aristocracy, and where French words entered English, it was often in this domain. A classic example is that names of animals—cow, pig, and deer, for instance, are almost all of Old English inheritance, while the names of those meats at the table–beef, pork, and venison, are of French. The English-speaking lower classes would be more likely to be in contact with the animals themselves, while the French-speaking upper classes would be likelier to be concerned with the commodity.

As previously discussed, the hypothesis for the analysis of Portrait was that there would be an increase in the L scores across chapters in the novel. Figure 3 shows that this hypothesis is partially confirmed. There is a significant rise in the proportions of Latinate words over chapters 1, 2, and 3, which would seem to correlate with the maturation of Stephen’s thought and speech. The L score plateaus or drops in chapters 4 and 5, however. How might this be interpreted?

Figure 3: Chapters, L Scores

To answer this question, it is necessary to conduct a more granular analysis. Figure 4 shows the text is divided into sections based on John Paul Riquelme’s structural divisions (“Structural Rhythm” 307). The L scores for these divisions exhibit much less of a simple progression from low to high. Where the climax of the chapter-based analysis seemed to be in Chapter 3, the climax here appears to be Chapter 4, Part 1. With the exception of Chapter 2, the longest and only five-section chapter, the highest L scores for each chapter come in the first section. The final sections of each chapter are among the lowest in L scores.

Figure 4: Sections, L Scores

Seen broadly, there is a pattern here suggestive of a what Riquelme calls a “structural rhythm”—a repeating sawtooth shape. A number of critics have noticed this cyclical structure. Sidney Bolt describes it thus:

At the beginning of each chapter Stephen is presented as the subject of a distressing tension, which develops to a crisis leading to a resolution. At the beginning of the next chapter, however, this resolution is seen to have produced a new tension, and the process is continued in a new form. This wave-like, pulsating movement is characteristic of every scene. (63)

Thomas Connolly calls this form a play between spiritual and corporeal forces. “Each [of these forces] nullifies the other,” he argues, “and a nexus results until the aesthetic perception of the beautiful breaks the knot and kinesis yields to stasis” (22). Diane Fortuna describes these cycles in terms of labyrinth imagery and the Dedalus myth, and adds that “aside from the initial subsection of Portrait, each of the subsequent 18 divisions of the novel presents at least one image of rolling, cyclical, or circling motion” (197). Fortuna’s observation could be read as an approximate description of the rolling, cyclical etymological trends shown in Figure 4.

One seminal description of this phenomenon is David Hayman’s reading of this structural oscillation as one between epiphanies and anti-epiphanies. The epiphanic moment is “a lyrical and wish-fulfilling moment during which the illusory is made to appear as immediate and valid”; it is “both art and event.” These moments then engender an “anti-aesthetic impulse to action” (164–5). While the epiphany is a “vision” or “illusion,” it is followed by an anti-epiphany that “show[s] Stephen to be increasingly involved with the world” (174). Riquelme calls this oscillation “a stylistic double helix,” and adds that “Joyce employs the two epiphanic modes of stark realism—‘the vulgarity of speech or of gesture’—and visionary fantasy … as delimiting extremes in his character” (“Styles of Realism” 119, 104). These properties—lofty visions and earthly pragmatics—map roughly to properties of L and G registers.

A closer reading of these sections might be more useful than a simple mapping of criticism to macro-etymological data, however. For that, we must make the analysis even more granular. The section with the highest L score is 4.1, which Riquelme titles “Spiritual Discipline.” Grant Redford, for one, claims that this is the climactic section of the novel (108). Within this section, the highest L score can be found in the second quarter. Here is a single sentence excerpted from that subsection:

The imagery through which the nature and kinship of the Three Persons of the Trinity were darkly shadowed forth in the books of devotion which he read—the Father contemplating from all eternity as in a mirror His Divine Perfections and thereby begetting eternally the Eternal Son and the Holy Spirit proceeding out of Father and Son from all eternity—were easier of acceptance by his mind by reason of their august incomprehensibility than was the simple fact that God had loved his soul from all eternity, for ages before he had been born into the world, for ages before the world itself had existed. (124-5)

This passage is verbose, florid, and multi-syllabic; its subject matter is religious, authoritarian, and deathly serious. Compare that with the passage with the lowest L score, at the end of section 1.3, “Christmas Dinner,” when the argument about Parnell becomes heated. Mr. Casey’s livid yet comic remark here neatly illustrates the Germanic register used in this section: “She stuck her ugly old face up at me when she said it and I had my mouth full of tobacco juice. I bent down to her and Phth! says I to her like that” (30). With the notable exception of “tobacco,” which is ultimately descended from an indigenous Haitian language, most of these words are monosyllabic and of Germanic origin. The rhythm here is faster, and the tone lighter. There is a certain playfulness evident in the onomatopoeia Phth!, a kind of neologism which we shall see is characteristic of Germanic Joyceanisms.

Germanic

While the proportions of words of Germanic origin in Portrait are, roughly speaking, inversely proportional to those of Latinate origin, they warrant discussion. Words of Germanic origin are frequently monosyllabic, polosive, and evoke raw, unfiltered speech that is often undecorated with euphemism and social formality. When Cantwell says “He’d give you a toe in the rump for yourself,” Stephen thinks, “that was not a nice expression” (7). What if Cantwell had used the French-derived synonym “derrière,” or the Latin-derived “posterior”? That might not still be a nice place to have a toe, but the expression would be more polite. “Rump” is “not a nice expression” because it bears the resonances of the Germanic register.

Early in the novel, young Stephen overhears someone use the word “suck,” and thinks, “suck was a queer word … the sound was ugly” (8). This passage, and indeed, this “queer word” has been much-discussed, most notably in Derek Attridge’s study (59). This may also have been what H.G. Wells had in mind when he accused Joyce of having a “cloacal obsession”:

He would bring back into the general picture of life aspects which modern drainage and modern decorum have taken out of ordinary discourse and conversation. Coarse, unfamiliar words are scattered about the book unpleasantly… (Wells, quoted in Deming 86)

If we remember that Stephen compares the sound of “suck” to that of “dirty water” going down the drain, modern drainage is literally that which creates this “coarse, unfamiliar” word sound. Wells’s critique highlights the reason why passages like this one were so “coarse” for readers contemporary with Joyce—the sounds and registers of their words. In fact, Portrait was rejected by early English publishers on this basis. In a reader’s report for the publishers Duckworth & Company, Edward Garnett calls the novel “too discursive, formless, unrestrained,” because “ugly things, ugly words, are too prominent” (Deming 81). Are words like “suck” ugly because they belong to the Germanic register, and carry those associations?

Stephen’s thoughts about the word “suck” begin a onomatopoetic theme that is chiefly associated with words of Germanic origin. Stephen later explains the word “kiss” in onomatopoetic terms—when he thinks of his mother’s kiss, Stephen thinks, “her lips … made a tiny little noise: kiss” (10-11). Mr. Casey’s Phth! falls into this same category. Jeri Johnson notices the preponderance of these words, and argues that “if Stephen could be said to have a theory of language at this point, it would be the bow-wow or onomatopoeic theory: the word for the thing imitates its actual acoustic equivalent in reality: ‘suck’ has its name because things that ‘suck’ make ‘sucky sounds’” ( xxvii).

In addition to their sonic associations, Germanic-derived words in Joyce have strong visual connotations. “The word was beautiful: wine,” Stephen thinks. “It made you think of dark purple because the grapes were dark purple that grew in Greece” (39). Although wine has a distant ancestor in Latin (vīnum), its immediate parents are Middle and Old English. Stephen’s associations are that of a certain dark purple color and Greece, which recalls the Homeric cliché that appears three times in Joyce’s Ulysses: “epi oinopa ponton”—“the wine-dark sea.”2 This is important to keep in mind, since, on the same page, a Latin lesson begins, in which Father Arnall “asked Jack Lawton to decline the noun mare,” or sea. Jack fails to decline the noun, and “could not go on with the plural,” an implicit choice of the Germanic sea over the Latin mare. The first plural of mare is maria, which is also the Latin name for Mary. Does Lawton’s failure to produce “Mary” in front of Father Arnall prefigure Stephen’s eventual rejection of the Sodality of the Blessed Virgin, of which he was prefect? This would be a far-fetched hypothesis on the subject of any other author, but given Joyce’s famous love of puzzles, it is entirely plausible, and it takes place along an etymological axis.

When Stephen is about to confess his sins in section 3.3, we see another passage with a low L score, notable for its alliteration:

His blood began to murmur in his veins, murmuring like a sinful city summoned from its sleep to hear its doom. Little flakes of fire fell and powdery ashes fell softly, alighting on the houses of men. They stirred, waking from sleep, troubled by the heated air. (130)

The double alliterative structure here—a string of s- words interrupted by a string of f- words—recalls the verse style distinctive of Old English poems such as Beowulf. Furthermore, Most of these words are of Germanic origin, which lends them the immediacy that the passage requires to evoke Stephen’s guilt and anxiety.

Hellenic

Portrait’s words of ancient Greek origin deserve a brief discussion. Greek words are some of the more difficult to quantify, since most of the Greek loanwords in English come to us through Latin, and a few (like “alchemy”) through Arabic. When classical Greek works began to be rediscovered in 1453, after Greek scholars fled Turkish-occupied Constantinople, this brought with them a number of associated loanwords. This could explain why many Greek loanwords seem at home in Aristotle or Plato—drama, comedy, and pathos recall the Poetics; phenomenon, noumenon, and democracy seem appropriate to a Socratic dialogue. As the analysis of the Brown Corpus hints, religious words, too, are heavily Hellenic: angel, evangelist, hagiography, bible, and so on, are all descended from Greek. We might find, therefore, that an aesthetic treatise of the kind Stephen presents in 5.1, or a religious sermon like Father Arnall’s in 3.2, might contain a higher proportion of words of Hellenic origin.

Figure 5: Sections, H Scores

Figure 5 shows that those two sections have, respectively, the first and third highest H scores of any section. Father Arnall’s sermon in section 3.2 features the emotionally-charged Hellenic words agony, (which appears an amazing eight times in this section alone), demon and zealous, along with the more tame words baptism, poetry, and eon. In section 5.1, those words are more befitting their setting in a physics classroom—physics, energy, and kinetic, along with didactic. These categories of religious and learned language are consistent with the analysis of the Brown corpus.3

The section with the second-highest H score is 4.3. Interestingly, this is the section where Stephen’s classmates taunt him in Greek: “Stephanos Dedalos! Bous Stephanoumenos! Bous Stephaneforos!” (141). These words are a polyglot pun on his name and Greek words for a sacrificial cow adorned with a wreath (O’Hehir, Classical 528). However, this is not what the Macro-Etymological Analyzer is detecting—since the program doesn’t recognize words of languages other than English, it treats Bous and Stephanoumenos as errors. The words of Hellenic origin in this section, then, are other English words: ecstasy and antogonism, for instance. That Joyce is using more than the usual number of Hellenic words here fits with the Dedalian myth, for on this same page, we see the epiphanic culmination of this metaphor, in the imagery of flight:

His heart trembled; his breath came faster and a wild spirit passed over his limbs as though he were soaring sunward. His heart trembled in an ecstasy of fear and his soul was in flight. His soul was soaring in an air beyond the world and the body he knew was purified in a breath and delivered of incertitude and made radiant and commingled with the element of the spirit. An ecstasy of flight made radiant his eyes and wild his breath and tremulous and wild and radiant his windswept limbs. (141)

Moments later, there is an Icarian anti-ephiphany that risks bathos, as Stephen’s thought is interrupted by the voices of a schoolmate playing in the water: “Oh, Cripes, I’m drownded!” (ibid.).

It could certainly be argued that the Hellenic words represented here are more useful to close reading than they are to distant reading. Since there are so few Hellenic words, they are statistically insignificant. However, in literary analysis, the significance of a single word could form the basis of a critical argument, while it may remain statistically uninteresting.

Celtic

Although the Macro-Etymological Analyzer identified only one word descended from the Irish language, sugan, the language has a deep effect on the styles of the novel. O’Hehir’s Gaelic Lexicon identifies sixteen words of Irish descent in Portrait (A Gaelic Lexicon for Finnegans Wake, and Glossary for Joyce’s Other Works 335–6). Some of O’Hehir’s words, like cool (from cúl, goal) are homographs with unrelated English words, a fact that might help to explain why they cannot be found by the program4. Others, like “smugging,” are of dubious Irish etymology—O’Hehir supposes that this word may be derived from smug or smuga, meaning “snot, nose drip,” or “slime” (ibid.). In classic Joycean fashion, this word is not glossed. It appears early in the novel, when Athy relates that the Simon Moonan and Tusker Boyle are caught “smugging” in the restroom. This is such a somber revelation that the rest of the boys are silenced by the thought, but Stephen does not understand—“what did that mean about the smugging?” he thinks (35).

Johannes Hedberg guesses that smugging can be traced to the Old English word smūgan, by way of the Middle English verb smuȜen (25), but Alarik Rynell contends that Hedberg’s etymology is erroneous, and that it is more likely decended from Old English smugge, “a small secret place” (367). Rynell uses the English phonesthemes of smugging to argue that “smugging must indeed have seemed an appropriate colloquialism for masturbating.” This is also Attridge’s theory (63). Most others assume that it’s a euphemism for homosexual play (Howes 255, for instance), although Fargnoli claims it is “entirely made up and has no established meaning” (207). The OED gives “to caress, fondle,” citing another of Joyce’s uses of the word in Ulysses, as well as the early 19th century poet Scottish poet Ebenezer Picken. That the only two citations for this sense are from Celtic writers lends some credence to the theory of Celtic etymology. On the other hand, that all of these theories assume some kind of taboo schoolyard sexual act, suggests the Germanic origin of the word, not only on the basis of G score of Brown Corpus romance texts, but also given the large number of other similar four-letter sexual words in the Germanic register.

More important to this discussion than the words themselves is the political dimension of the Irish language, especially as it existed in Ireland on the eve of independence. The revival of the Irish language was intimately associated with the nationalist movement, from which Joyce as a self-imposed exile had distanced himself both physically and intellectually, but with which he nonetheless felt some affinity. Although neither Stephen nor Joyce himself knew much Irish—Stephen stops taking Gaelic League classes after the first lesson—they seem to struggle with the dual political and linguistic dominance of Britain over Ireland. A passage that illustrates this is Stephen’s conversation with the English dean over the word tundish. Stephen thinks:

The language in which we are speaking is his before it is mine. How different are the words home, Christ, ale, master, on his lips and on mine! I cannot speak or write these words without unrest of spirit. His language, so familiar and so foreign, will always be for me an acquired speech. I have not made or accepted his words. My voice holds them at bay. My soul frets in the shadow of his language. (159)

Here, “his language” could be read as both the Dean’s British English dialect and English more generally. Since Irish is the ancestral language of Ireland, English is an “acquired speech” in this historical sense. More immediately, the tonal differences in their speech distinguish their two Englishes. Anthony Burgess has a notable phonetic interpretation of this passage, suggesting that Stephen likely chooses these four words because they are pronounced differently in British and Hibernian dialects—dipthongs instead of long open vowels, final schwas instead of retroflex Rs (28). The words are ontologically different, as well—“home” refers to different cities for the two men; “Christ” is very different for the Catholic and the Protestant; and as a student, Stephen’s “mastery” is that of a subject he is taught, while the Dean’s is that over people, that of a colonist and a schoolmaster.

These colonial undercurrents are useful to a discussion of etymology in Joyce, because they help to reveal choices of etymological modes as domains of nations, with histories and political uses. Joyce’s decision to have Stephen’s uncle Mat Davin use Irish-derived words like camann (from the Irish camán, the stick used in hurling) enforces the earlier description of him as a “young peasant” who “worshipped the sorrowful legend of Ireland” (151-2). Stephen refers to this same object with an Anglo-Saxon word when he scoffs at the most recent Irish uprising, calling it “a rebellion with hurleysticks” (169). Johnson explains that this is “a ‘sneerer’s’ comment on the failed Fenian Rising of 1867, training having taken place not with guns but with camann” (274). Joyce’s use of camann and hurleysticks is not interchangeable, but chosen to evoke histories, politics, and tones that each word carries.

Unknown Words

Camann was one of about four percent of the words of Portrait that the Macro-Etymological Analyzer failed to find. These words proved to be revealing about Joyce’s style, especially concerning etymological associations. Many of the unknown words are proper names, and proper names were purposely removed from the database, as they would skew the results unnecessarily.5 Other unknown words, however, are Joyce’s inventions. Some of these are true neologisms, while others are portmanteau words or unhyphenated compound words.

While much critical attention has been paid to the neologisms in Finnegans Wake, since they are undoubtedly its distinctive property, not much has been discussed regarding Portrait, even though at least one word, pandybat has its first OED citation in the novel. Regarding Joyce’s work as a whole, however, Katie Wales identifies two neologistic strategies: “conversions” and “compounds” (115). “Conversion,” Wales relates, “extends the semantic range of existing words by changing the grammatical function.” “Compounds” refers to portmanteau words, which are littered throughout the novel. These two categories are roughly equivalent to Joseph Prescott’s conception of Joyce’s “renovation” and “innovation” (308). By “renovation,” Prescott claims that Joyce “imposes on words of common currency a fresh lustre, usually the brilliance of their first years.” Among the examples Prescott gives for “rennovation” is a passage from Ulysses where Joyce uses the word “crazy” in its etymological sense of “fractured” (309). In illustration of his category of “innovation,” Prescott calls Joycean neologism “dynamic onomatopoeia,” citing the “crescendo” of cat noises in Ulysses (311). When onomatopoeia is “triumphant” in Joyce, he argues, it constitutes the “anastomosis of style and subject.” Given the etymological associations with onomatopoeia established earlier, this fusion of style and subject could be said to take place along etymological vectors, as well.

To identify a list of Joycean neologisms beyond the error words of the Macro-Etymological Analyzer, the text was run through a command-line spell-checking program, and the results sorted, with this chain of Linux commands:

cat portrait.txt | aspell -a | cut -d ' ' -f 2 | \
grep -v '*' | sort | uniq > misspelled.txt

The result was a list of words the command aspell determined were “misspelled.” After manual curation to remove words in Latin, proper names, and real but obscure words, this became a list of Joycean terms. Most of these words are compound words, and often formed from two Germanic words, like suddenwoven or rainladen. These words exhibit several themes. First, there are color words, like ambered, bloodred, greenwhite, and redeyed, along with hueless and nocolored. Next, there are kinship terms, including fosterbrother, fosterchild, greatgrandfather, halfbrother, and granduncle. Another category features agrarian or pastoral terms like cowdung, cowhairs, goatish, milkcar, boghole, and bogwater. Finally, there is theme related to dirt, filth, and the street: sootcoated, thumbblackened, and greasestrewn. All of these categories are associated with the Germanic register.

It should perhaps not be surprising that so many of Joyce’s neologisms and portmanteau words are of Germanic origin, since word compounding in this style is a feature of many modern Germanic languages, most notably modern German. In fact, many of these words, if separated into their constituent words (great grandfather) and translated into German, prove to be one German word (Urgroßvater).

Conclusions

Joyce achieves many of the narrative effects of A Portrait of the Artist as a Young Man through the use of etymological registers. Just as the language of his narration, according to the Uncle Charles Principle, follows the thoughts of his characters, his oscillations between Germanic and Latinate linguistic modes mimic oscillations between epiphanic and anti-ephiphanic scenes. Macro-etymological analysis, therefore, demonstrates that it might be well-suited to become part of suite of analytic tools that can participate in the detection of structural patterns of a novel. Along with word frequency analysis, principal component analysis, metrical detection, and segmentized type/token ratio calculation, macro-etymological analysis might form a part of a greater textual analytic system that can inform and improve computational literary criticism.

Works Cited

Attridge, Derek. “‘Suck Was a Queer Word’: Language, Sex, and the Remainer in A Portrait of the Artist as a Young Man.” Joyce Effects. Cambridge, UK: Cambridge University Press, 2000. 59–77. Print.

Bolt, Sidney. A Preface to James Joyce. Second Edi. London: Longman, 1981. Print.

Brian, Michael. “‘A Very Fine Piece of Writing’: An Etymological, Dantean, and Gnostic Reading of Joyce’s ‘Ivy Day in the Committee Room’.” ReJoycing: New Readings of Dubliners. Lexington, KY: University Press of Kentucky, 1998. 206–227. Print.

Burgess, Anthony. Joysprick: An Introduction to the Language of James Joyce. New York: Harcourt Brace Jovanovich, 1973. Print.

Connolly, Thomas E. “Kinesis and Stasis: Structural Rhythm in Joyce’s Portrait.” University Review 3.10 (1966): 21–30. Print.

Deming, Robert. James Joyce, the Critical Heritage. New York: Barnes & Noble, 1970. Print.

Fargnoli, A Nicholas. James Joyce A-Z. Oxford, UK: Oxford University Press, 1995. Print.

Fortuna, Diane. “The Art of The Labyrinth.” Critical Essays on James Joyce’s a Portrait of the Artist as a Young Man. New York: G.K. Hall & Co., 1998. Print.

Hayman, David. “A Portrait of the Artist as a Young Man and L’Éducation Sentimentale: the Structural Affinities.” Orbis Litterarum 19 (1964): 161–75. Web.

Hedberg, Johannes. “Smugging. An Investigation of a Joycean Word.” Moderna Sprak 66 (1972): 19–25. Print.

Howes, Marjorie. “Joyce, Colonialism, and Nationalism.” The Cambridge Companion to James Joyce. Ed. Derek Attridge. Second Edi. Cambridge, UK: Cambridge University Press, 2006. 254–2471. Print.

Johnson, Jeri. “Introduction.” A Portrait of the Artist as a Young Man. Oxford Wor. Oxford University Press, 2000. vii–xxxvii. Print.

Joyce, James. A Portrait of the Artist as a Young Man. Ed. Jeri Johnson. Oxford, UK: Oxford University Press, 2000. Print.

—. Stephen Hero. New York: New Directions, 1944. Print.

—. “The Study of Languages.” Occasional, Critical, and Political Writing. Oxford, UK: Oxford University Press, 2008. Web.

Kenner, Hugh. “The Uncle Charles Principle.” Joyce’s Voices. Berkeley: University of California Press, 1978. 15–38. Web.

Milesi, Laurent. James Joyce and the Difference of Language. Cambridge, UK: Cambridge University Press, 2003. Print.

O’Hehir, Brendan. A Classical Lexicon for Finnegans Wake: a Glossary of the Greek and Latin in the Major Works of Joyce incl. Finnegans Wake, the Poems, Dubliners, Stephen Hero, A Portrait of the Artist as a Young Man. Berkeley: Univ. of California Press, 1977. Print.

—. A Gaelic Lexicon for Finnegans Wake, and Glossary for Joyce’s Other Works. Berkeley: University of California Press, 1967. Print.

Prescott, Joseph. “James Joyce: A Study in Words.” Publications of the Modern Language Association of … 54.1 (1939): 304–315. Web.

Redford, Grant H. “The Role of Structure in Joyce’s Portrait.” Joyce’s Portrait: Criticisms and Critiques. Ed. Thomas E Connolly. Appleton, 1962. 102–115. Print.

Riquelme, John Paul. “Stephen Hero, Dubliners, and A Portrait of the Artist as a Young Man: Styles of Realism and Fantasy.” Cambridge Companion to James Joyce, the. N.p., 1990. 103–130. Print.

—. “The Parts and the Structural Rhythm of A Portrait.” A Portrait of the Artist as a Young Man. New York: W. W. Norton & Company, 2007. 307–8. Print.

Rynell, Alarik. “On the Etymology of James Joyce’s Smugging.” Moderna Sprak 66 (1972): 366–369. Print.

Smith, John B. Imagery and the Mind of Stephen Dedalus : a Computer-Assisted Study of Joyce’s A Portrait of the Artist as a Young Man. Lewisburg, PA: Bucknell University Press, 1980. Print.

Wales, Katie. The Language of James Joyce. New York: St. Martin’s Press, 1992. Print.

Whittaker, Stephen. “Joyce and Skeat.” James Joyce Quarterly 24.2 (1987): 177–192. Print.

At the moment, these are proportions of the total tokens, but a future version of this program will calculate proportions of the types.↩︎
This phrase is discussed at length in William Gladstone’s Studies on Homer and the Homeric Age, where Gladstone argues on the basis of color words in Homer that the ancient Greeks lacked the ability to perceive colors like blue.↩︎
The pattern of Hellenic words in Figure 5 also closely resembles patterns of religious images identified in a 1979 computational study of Portrait by John B. Smith. In this study, Smith counts “images” that belong to certain taxonomies like “fire” and “water,” and plots them according to their location in the novel. The category of “religion” aligns very roughly with the Hellenic plot in Figure 5.↩︎
Word sense disambiguation is a featured planned for future versions of the Macro-Etymological Analyzer.↩︎
Portrait would show unusually high proportions of Hellenic words, for instance, in every section where the word Stephen would appear. Analyses of Christian bibles showed similar results every time the word Jesus was mentioned, irrespective of the author’s choice to use words in the Hellenic register.↩︎

Note

This paper was prepared for submission to the forthcoming volume Reading Modernism With Machines. The source files may be found in this GitHub repository, where you can also find a PDF version and a DOCX version. I very much welcome feedback in the comments below!

Introducing Git-Lit

2015-09-08

A vibrant discussion followed my March 15th post, “A Proposal for a Corpus Sharing Protocol.”. Carrie Schroeder, Allen Riddel and others on Twitter pointed out that, especially in non-English DH fields, many corpora are already on GitHub. These include texts from the Chinese Buddhist Electronic Text Association, the Open Greek and Latin Project at Leipzig, and papyri from the Integrating Digital Papyrology Project. The Text Creation Partnership has released some 25,000 of their texts in January of this year, and uploaded them to GitHub. One of the more interesting Git corpus projects I became aware of following this discussion is GITenberg. Led by Seth Woodworth, the project scrapes a text from Project Gutenberg, initializes a git repository for it, adds README and CONTRIBUTING files generated from the text’s metadata, and uploads the resulting repository to GitHub. They have gitified around 43,000 works this way. The project also converts Project Gutenberg vanilla plain text into ASCIIDOC—a good example of this is the GITenberg edition of The Adventures of Huckleberry Finn. This is an amazingly ambitious project that holds the promise of wide-ranging applications for editing, versioning, and disseminating literature.

One such application might lie with the 68,000 digital texts recently created by the British Library. James Baker, a digital curator of the British Library, left a comment on my original post, suggesting that the method I describe might be used to parse and post the Library’s texts. He sent me a few sample texts of the ALTO XML documents that the Stanford Literary Lab had used. I adapted some of the GITenberg code to read these texts, generate README files for them, and turn them into GitHub repositories. I’m provisionally calling this project Git-Lit.

Introducing Git-Lit

Git-Lit aims to parse, version control, and post each work in the British Library’s corpus of digital texts. Parsing the texts will transform the machine-readable metadata into human-readable prefatory material; version controlling the texts will allow for collaborative editing and revision of the texts, effectively crowdsourcing the correction of OCR errors; and posting the texts to GitHub will ensure the texts’ visibility to the greater community.

Why This is Important

Git-Lit addresses these issues:

Electronic Texts are difficult to edit. Imagine you’re reading an e-text on a Kindle, and you notice an OCR error. How can you fix it? You can alert Amazon, who may or may not forward your message to the publisher. Since correcting the text would require the publisher to recompile and resubmit it, they will likely decide it isn’t worthwhile to make the correction. Similarly, if you notice an error in a Project Gutenberg text, it could take years for the correction to make its way back to the original text, even if you’re an active member of their Distributed Proofreaders collective. In both cases, there does not yet exist an efficient, streamlined way to improve the quality of electronic texts. What is needed, therefore, is an open-source, decentralized model for community-centered editing. This model already exists for software development in the form of git. By posting a text to GitHub, we can take advantage of the fork/revise/pull request workflow that programmers have long enjoyed for software collaboration.
Textual corpora are difficult to assemble. With some exceptions (notably the NLTK corpus module), downloading a text corpus involves compiling texts from any number of heterogeneous sources. A would-be text analyst must click through a series of web pages to find the corpus he or she wants, and then either download a .zip file that must be expanded, or email the corpus assembler for a copy of the corpus. With multiple texts, this can be a labor-intensive process that is not easily scriptable or automated. Git provides an easy way to solve these problems. By making texts available through the git protocol on GitHub, anyone that wishes to download a text corpus can simply run git clone followed by the repository URL. Parent repositories can then be assembled for collections of texts using git submodules. That’s to say, a parent corpus repository might be created for nineteenth-century Bildungsromane, for instance, and that repository would contain pointers to individual texts that themselves are their own repositories.
ALTO XML is not very human-readable. ALTO XML, the OCR output format used by the British Library, the Library of Congress, and others, is extremely verbose. It encodes the location of each OCRed word, and often gives the OCR certainty for each word. This is great for archival purposes, but isn’t an ideal starting-point for the kinds of text analysis typically done in the digital humanities. What we need is a script to transform this verbose XML into a human-readable format like ASCIIDOC that maintains as many of the original features of the text as possible.

How it Works

A British Library text contains ALTO XML textual data as well as a Library of Congress METS XML metadata file. Git-Lit does the following:

Reads the metadata file to determine the text’s title, author (if any), and other pertinent information.
Initializes an empty git repository within the text directory, and makes an initial commit containing the text at its raw state.
Generates a README file with the metadata, a CONTRIBUTING file explaining how to contribute towards improving the text, and a LICENSE file which, for now, is just the standard GNU Public License v3.
Commits these new files to the git repository, effectively creating a new version.
Creates and pushes a new GitHub repository for the text.

I ran Git-Lit on the four sample texts in the data directory, and generated the four GitHub repositories that can be found on the Git-Lit organization. You can read, fork, modify, or comment on the IPython Notebook that does this on the project repository at GitHub.

Future Phases of this Project

As this project develops, we’ll create indices for the texts in the form of submodule pointers. Category-based parent repositories might include “17th Century Novels,” “18th Century Correspondence,” or simply “Poetry,” but the categories are not mutually exclusive by necessity. This will allow a literary scholar interested in a particular category to instantly assemble a corpus by git clone=ing the parent repository and checking out its submodules with =git submodule update --init --recursive.

Later, we’ll write a scripts to transform the texts in more useful formats, like ASCIIDOC and TEI XML. This will make archival-quality versions of the texts, and will allow for rich scholarly markup.

How to Contribute

Please join this initiative! To contribute, contact me, or find an issue you can tackle on the project issue tracker. Also, feel free to add your own features, restructure the code, or make any other improvements. Pull requests are very welcome!

Tag, Catalog, Iterate: A Non-Linear Analysis of Katherine Mansfield’s “The Garden Party”

2016-01-25

Works of literary criticism are typically built around their central arguments. The argument is often an opinion about a theme, function, or other textual attribute, and it is presented along with textual evidence that supports it. This usually means that a critical work closely examines textual details that are relevant to its thesis, while moving past or even ignoring less relevant details. Figure 1 shows a simplified illustration of this process. If each segment in the primary text contains details of types A, B, C, or D, a typical critical work will select one or more of these details for its analysis, while other details remain either less examined or fully unexamined.

Figure 1: Narrative Criticism

There is nothing inherently problematic with this style of criticism—the thread that ties together these elements is that which makes the critical work enjoyable and easy to follow. This is also the structure that helps a critic to outline his or her contribution to the discussion: “Critic X discusses textual element A, and critic Y discusses textual element B, but they’re both ignoring the key to understanding the text, which is element C.” This can be a reasonably satisfying line of reasoning, but what if there were a more inclusive, less centralized way of talking about a text? Could we imagine a pluralist, iterative style of literary criticism? Would such a style be useful, or even desirable?

First, it is necessary to admit that this is not a new idea. What I will be calling “iterative criticism” here is present to some degree in any metaliterary work that gradationally catalogs, maps, or annotates a literary text. One such work is Don Gifford’s monumental Ulysses Annotated, a book of annotations for James Joyce’s Ulysses. Staggering in the breadth of its scope, it analyzes the novel on the level of the word and phrase, to such an extent that its own length exceeds that of its primary text. As a non-narrative work that refrains from privileging any particular reading of the text, it can be read non-linearly: a reader interested in, say, the third word of Chapter 3 can simply find the relevant annotation and start reading. This property is probably that which has led to its being given the enigmatic Library of Congress subject heading “Joyce, James — Dictionaries,” rather than “Joyce, James — Criticism,” like most other Joycean critical works. But Gifford’s volume remains critical, despite the way works of annotation are often (quite literally, one might say) marginalized.

Another notable work of iterative criticism is Roland Barthes’s S/Z. Subtitled “an essay,” it is a book-length analysis of Honoré de Balzac’s short story “Sarrassine” with a nearly phrase-level focus. Barthes divides the story into 561 textual units he calls “lexias,” some of which contain a few sentences, and some which contain just a few words. Each of these lexias he discusses in detail, according to five “codes,” or groupings of annotations: hermeneutic, proairetic, semic, symbolic, and referential. A typical annotated lexia looks like this:

Nobody knew what country the Lanty family came from, ★ A new enigma, thematized (the Lantys are a family), proposed (there is an enigma), and formulated (what is their origin?): these three morphemes are here combined in a single phrase (HER. Enigma 3: theme, proposal, and formulation).

Here, Barthes catalogs the emergence of “Enigma 3,” belonging to the hermeneutic code (“HER”). This enigma will make several more appearances in Barthes’s essay, the set of which forms a critical narrative in miniature. While S/Z is certainly meant to be read linearly, from beginning to end, these codes and enigmas form a network of non-linear subnarratives that provide alternate trajectories. In fact, one could almost call S/Z a proto-hypertext, in that its layering and self-referentiality provide a multiplicity of pathyways through the critical discourse. According to a 1994 definition by computer scientists Frank Halasz and Mayer Schwartz, hypertext provides “the ability to create, manipulate, and/or examine a network of information containing nodes interconnected by relational links” (30). This is precisely what S/Z accomplishes. One might even imagine that if Barthes had experience with a hypertext language like HTML, the codes and themes he cites might be linked together technologically as well as textually. That is one of the goals of the following experiment.

The Experiment

The following is an experiment in iterative literary criticism, where the text of Katherine Mansfield’s story “The Garden Party” is broken into 205 Barthesian lexias, annotated, and tagged. These tags are then linked to each other, forming miniature topic-based critical works. Clicking on a tag scrolls the page to the next annotation with that tag, and if there are no more, the page loops back to the first. A reader interested, in say, the semiotics of flowers in “The Garden Party” might read an annotation tagged “flora”, while a reader interested in gender dynamics might choose the tag “sexuality.” Since many lexia have more than one tag, the reader may switch between related tags once annotations with their first chosen tag have been exhausted. This effectively creates a non-linear critical work that more resembles a decision tree than a straight line. In this way, the reader effectively assembles his or her own critical narrative in the act of reading. A revised flowchart for this style of criticism might look something like Figure 2.

Figure 2: Iterative Criticism

One of the advantages of this iterative style, compared with the narrative style, is that it opens the textual field to the greater possibility of surprises. New details emerge that might have otherwise been elided had the detail needed to fit into a prose paragraph, and the paragraph into an overarching argument. The two-column format used below privileges no particular reading, and as such, it allows for minority readings—critical theories that may not have enough textual evidence to be made into a standard-length journal article—to have equal status with majority readings.

Another advantage of this style is that, by cataloging the appearance of textual themes, tensions, and images chronologically—that is, charting their occurrences according to where they happen in the story—we can derive a better picture of how those literary elements unfold, and where they disappear. To Barthes, this is the music of the text, which he charts literally in Figure 3.

Figure 3: Barthes, Score for “Sarrassine” Lexias 1-13 (29)

Rather than use semes and cultural codes, however, this edition of “The Garden Party” is tagged nonhierarchically, using roughly fifty tags. The types of these tags range everywhere from color images like “green” and “black” to the story’s treatment of social class, given by the tag “class.” Some significant objects, such as Laura’s hat are tagged, as well as the story’s envelope(s). A full list is given in the iPython notebook used to generate tag statistics. Figure 4 shows all tags with more than three occurrences, sorted according to how often they are used.

Figure 4: Most Frequently Occurring Tags

The tag that appears the most often is “class.” This is unsurprising, given the story’s overt treatment of social class. The second most frequent tag is “interruptions,” which charts both syntactic truncation (“isn’t life” of L204, for instance) and proairetic truncation, such as the interrupted breakfast of L7. Other notable tags include “flora,” used whenever literal flowers appear, or when floral metaphors are used, such as in L150. These tags can help to track trends, themes, and moods as they unfold in the chronology of the story.

Figure 5: Flora, Sexuality, Death

Figure 5 shows three of these tags: flora, sexuality, and death, plotted according to where they occur in the story. Flora and sexuality tend to collocate, as one might expect, and for the most part, references to flowers and references to death are fairly separate, with the exception of the group that occurs around the time of the party. (The X values in this chart correspond to the lexia numbers divided by 5, grouped here for smoothing.) Upon closer examination, it turns out that these are some interesting collocations of floral and morbid imagery. One is Laura’s hat, black as if in mourning but “trimmed with gold daisies”; another is the floral metaphor of the dying afternoon: “the perfect afternoon slowly ripened, slowly faded, slowly its petals closed” (L150).

Figure 6: Green, Light, Black, Darkness

With colors, too, we find interesting collocations, as shown in Figure 6. On the whole, references to greenness or green things (the grass, bushes, and turban, for instance) collocate with the morning and with preparations for the garden party. References to light occur mostly in this portion of the story, too, and references to black and darkness mostly happen after nightfall. However, there are two notable surprises here: the collocation of green and black at 27, directly before the party, and the strange combination of light, black, and darkness around 33. The first corresponds to the black hat of L137 followed by the green band and green tennis court of L140. One might read the appearance of the black hat as a hint of the mourning scene to come, and the greenness of the garden party as the apex of the green imagery that has been building during the party’s preparations. The second unexpected collocation, that of light and darkness at 33 in this chart, is the chiaroscuro generated by the dusky slant of light that makes the road “gleam white” and throws a “deep shade” on the cottages (L171). Based on this chart alone, it might be able to guess the timeframe of the story (in classic modernist fashion, it takes place in a single day) as well as the time of sunset (around location 35 in the chart).

Figure 7: Sounds, Colors, Touch

Sensory descriptions might also be useful to study quantitatively. Figure 7 shows tags of sounds, colors, and touch. A few examples of these are the “chuckling absurd” sound of the piano at L47, the repetition of “pink” in describing the lilies at L53, and Laura’s nibble of her mother’s ear at L58. Most of the sounds, colors, and touches occur in the first quarter of the story, highlighting the sensory richness of the morning preparations, which contrast greatly with the “dark,” “oily” imagery of the cottages. Notably, however, there are none of these impressions during the party itself, or immediately before or after. Only Laura’s memory remains, when she recalls “it seemed to her that kisses, voices, tinkling spoons, laughter, the smell of crushed grass were somehow inside her. She had no room for anything else. How strange!” (L172).

Many more quantitative literary analyses are made possible by this iterative approach. For a fuller list of tag comparison charts, and to run the Python 3 code against arbitrary collections of tags, see the iPython notebook used to generate these charts.

Iterative criticism is not meant to replace narrative criticism. Nor is it meant to represent the circumscribed totality of what can be said about a literary text. There are some birds-eye readings that simply do not fit into the sentence-level focus given here, and that is especially true of historical and biographical readings. But the insight gained from this inclusive, step-by-step technique might help us to discover things that the teleology of narrative criticism hides. It might help us to, in Barthes’s words, “remain attentive to the plural of a text” (11).

Textual Notes

The text of Katherine Mansfield’s story presented below is derived from the GITenberg edition of The Garden Party and Other Stories. The plain text was marked up using the Extensible Markup Language (XML) format of the Text Encoding Initiative (TEI). This format, the standard markup language for archival literary projects, is a semantic markup language—unlike markup languages like HTML 4.0, which describe how a text should look, i.e. =<i>The Garden Party</i>=, TEI XML describes what the text is, i.e. =<title>The Garden Party</title>=. This allows text segments to be selected based on their literary, rather than textual attributes.

The TEI text is transformed to HTML using xsltproc and an XSL stylesheet, and combined with this introductory text, which is transformed from markdown into HTML using pandoc. The files are combined using sed, and the compilation process automated using a makefile written for GNU make. A short jQuery script handles the interactive tag behavior.

Since the text has been broken into segments, pilcrow marks (¶) have been used to denote the beginnings of paragraphs as they appeared in the original text.

This edition has been made using exclusively free and open-source software. This text and the source code for this project is released under the GNU Public License v3, the full text of which is available in the included license.

Click on the link below to start reading the edition:

Start Here

A Programmatic Generator for Pound’s Cantos

2016-02-03

Or, Computational Poetic Generation as Literary Analysis

Note: this project may also be found here, in its original format as an iPython notebook.

“In so far as it represented man’s harnessing of energy through craft, the machine became the most characteristic emblem of the Vorticist movement, more significant … than the vortex itself.” (Bush 36)

Ezra Pound’s Cantos, whether understood as vorticist, imagist, ideogrammatic, or otherwise, is a poem that is the product of those schools, modes, and procedures. Since the techniques of the Cantos, like those of a cubist painting, are often foregrounded, critical works have been able to postulate about their composition in great detail, in some cases pinpointing the exact source materials for Pound’s collages and translations. With the exception of a few other highly allusory works—“The Wasteland,” for instance—few other modern poems are as richly and thoroughly annotated. One might even venture that there is almost enough metatextual data in the many handbooks to the Cantos—William Cookson’s, Roland’s John’s, George Kerns’s, or Carroll Terrell’s, just to name a few—to reconstruct the poem itself. Given the source materials, composition techniques, themes, meter, and style, it might just be possible to design an algorithm to combine these elements in just the right way to generate a canto. Although the generated canto is unlikely to be a work of great artistic merit, the process of translating the procedure of poetic composition into an algorithm will help us to understand its minute machinations. The following describes an experiment to attempt just that—the translation of literary criticism into a generative algorithm, which hopes to be revealing about the structure of the Cantos, the shape of its critical response, and the limitations of algorithmic composition.

Among the many analogies critics have made between the Cantos and the visual arts, Jacob Korg compares the poem’s techniques to those of collage. “The literary equivalent of the painter’s collage is,” he explains, “quotation—not conventional quotation, but the kind that presents itself as an interpolation, interrupting the text, and even conflicting with the writer’s purposes, as if it were an eruption of raw reality” (96). Since algorithmically generated language is still in its infancy, mostly belonging to the domain of semantically agnostic probabilistic models, the collage technique of the Cantos, and in particular Pound’s use of direct quotation, is what makes the epic one of the most appropriate poems to attempt to programmatically emulate. As John Childs puts it, “hardly a page of the Draft of XXX Cantos goes by without the insertion into the poem of a word or phrase drawn from an author other than Pound” (18). By identifying these source materials, and designing functions to extract from them and mix them in appropriate ways, we might be able to create poems similar to Pound’s.

The Experiment

The following is a experiment along the borders of computational and critical methodologies. The format of this investigation, an IPython notebook containing text and code, represents an attempt to problematize the distinction between computer science and literary criticism by intentionally combining the language of both. Some might argue that this paper is already at least half obscure for its intended audience, since most literary scholars are unfamiliar with programming languages, and most coders are unfamiliar with the language of literary study. To help dissolve this linguistic barrier, I have used the Python programming language, whose idioms are closest to English, and I have explained some of the logic behind the program’s functions using interpolated or inline comments (text following the # symbol) wherever possible. However, a few key concepts of Python are necessary to explain here briefly, in order to explain the fundamental programmatic structures below. This table shows a few types of Python expressions, along with their explanations in English:

Python Expression	Meaning
`myVariable = "a " + "test"`	This concatenates the text “a” and “test” and stores the result in a variable called `myVariable`. Now `myVariable` contains the text “a test.”
`def myFunction(x): ... return y`	This defines the function `myFunction`, accepts the input X, and returns the output Y.
`myList[0]`	The brackets here indicate an index of the list. This expression returns the 0th, i.e. first, element of the list myList.
`[x/2 for x in myList]`	This list comprehension defines a new list generated from an operation performed on every element of another list. This list represents half of every element X in the list `myList`.

With these foundations and the interlinear explanations, it should be possible to read the algorithms below. Additionally, text following a pound sign (#) and text enclosed in triple quotes (""") is typically explanatory commentary.

"""
In this section, we will declare the software libraries that we will
need for the code to follow.
"""

# Most of the text analysis done below is done with the help of the
# Natural Language Toolkit, or NLTK, a Python library for text analysis.
# See http://www.nltk.org for more information.
import nltk

# WordNet is a module in the NLTK that allows for easy navigation of the
# Princeton WordNet, a hierarchical lexical database.
from nltk.corpus import wordnet as wn

# Stopwords are commonly-used words like "a," "the," or "and," and
# this list of stopwords is useful for filtering them out of wordlists.
from nltk.corpus import stopwords

# These two machine translation libraries use the Google Translate API.
import goslate
from textblob import TextBlob

# Regular expressions allow us to search for and select text in a
# sophisticated way.
import re

# This library allows us to smartly count things.
from collections import Counter

# Random number generation allows us to randomly choose things.
import random

# This allows us to work with files located within directories
# in the filesystem.
from os import listdir

# This allows us to identify punctuation.
import string

# Pandas is a sophisticated data science library, that allows for
# data transformation and visualization.
import pandas as pd

# This turns on the option to show data visualizations in this notebook,
# and not in separate window.
%matplotlib inline

# This is a library for data visualization.
import matplotlib.pyplot as plt

# This sets the "ggplot" style for visualizations.
plt.style.use('ggplot')

# This library allows for text wrapping (e.g. line breaks).
import textwrap

In order to generate cantos, we’ll first need to study their form. We’ll need to determine the length of a canto, its numbers of stanzas, and its indentation fingerprint. I’ve prepared a markdown-formatted text file, cantosI-X.md below, which contains the text of Canto I through Canto X from A Draft of XXX Cantos, so that the text might be programmatically analyzed. This first class, CantosFile, reads this text, breaks it up by header, and stores it in memory.

class CantosFile():
    """
    Deals with the file containing transcriptions of the original
    Cantos.
    """
    # Open the text file and read its contents.
    cantos = open('text/cantosI-X.md').read()

    # Split the file by header marker ("#"), and take off the first part,
    # which is the title, and is not useful for our analysis.
    cantosList = cantos.split('#')[1:]

    # Count the number of cantos in this file by counting these text
    # segments.
    numCantos = len(cantosList)

Now that the file has been read into memory, we will study each canto individually with the CantoReader class below, which will read an individual canto and return information about it, such as the number of lines, number of stanzas, proportions of each, and information about the canto’s indentation.

class CantoReader(CantosFile):
    """
    A class to read individual cantos and return statistics about them
    that will be used by the CantoWriter class. This class accepts the
    canto number as an argument, so that CantoReader(1) returns a
    CantoReader object for Canto I.
    """
    def __init__(self, cantoNumber):
        """
        When this class is instanted, it will automatically store a copy
        of the canto number (e.g. 4 for Canto IV), and make a copy of that
        canto's text for analysis.
        """
        # Store a copy of the canto number.
        self.cantoNumber = cantoNumber

        # Adjust this number, since Python starts counting at zero,
        # but cantos are numbered starting with 1.
        cantoNumber = cantoNumber-1

        # Get the full text of the canto.
        self.allText = self.cantosList[cantoNumber]

    @property
    def lines(self):
        """
        Break up the canto into lines, and clean up any unnecessary
        lines.
        """
        # First, split the canto into lines.
        lines = self.allText.split('\n')

        # Next, remove the header.
        noheader = lines[2:]

        # So long as there is blank whitespace at the end of the
        # canto, keep removing it, so that we're not counting extranneous
        # lines.
        while noheader[-1].isspace() or len(noheader[-1]) == 0:
            noheader.pop()
        return noheader # Return the cleaned version.

    @property
    def text(self):
        """
        Get the text of the canto, cleaned by the `lines` function above,
        but join it together, so that it's not broken into lines.
        """
        return "\n".join(self.lines)

    @property
    def stanzas(self):
        """
        Break the canto into stanzas by finding double line breaks
        (`\n\n`) and dividing the text along those lines.
        """
        return nltk.tokenize.regexp_tokenize(self.text, '\n\n', gaps=True)

    @property
    def numLines(self):
        """ Count the number of lines in the canto. """
        return len(self.lines)

    @property
    def numStanzas(self):
        """Count the number of stanzas in a canto."""
        return len(self.stanzas)

    @property
    def stanzaLengths(self):
        """Determine the length of each stanza in the canto, in lines."""
        return [len(stanza.split('\n')) for stanza in self.stanzas]

    @property
    def stanzaPercentages(self):
        """
        Determine the proportion of the total canto's lines
        represented by its stanzas. This returns a list like
        `[0.7402597402597403, 0.24675324675324675]` (for Canto I)
        that means that the first stanza takes up about 74% of the total
        canto, and the second stanza takes up about 25%.
        """
        return [stanzaLength / self.numLines for stanzaLength in self.stanzaLengths]

    @property
    def indentation(self):
        """
        Get a list of indentation levels for each line in the canto.
        This returns a list like `[0, 0, 8]` if the first line is flush
        left, the second line is flush left, and the third line is
        indented by eight spaces.
        """
        results = [] # Initialize an empty list to contain our results.
        for line in self.lines: # Iterate through each line,
            match = re.search('\S', line) # looking for whitespace,
            if match is not None: # and if whitespace is found,
                # append the location of the first non-whitespace character
                # to the list.
                results.append(match.start())
            else:
                # otherwise, the line is flush left, so its indentation
                # level is zero. Append zero.
                results.append(0)
        return results

    @property
    def indentationStats(self):
        """
        Returns a dictionary of indentation levels with their corresponding counts and percentages.
        For Canto I, this is {8: [3, 0.0384...]}, since there are three lines indented 8 spaces,
        which make up around 3-4% of the total poem.
        """
        stats = Counter(self.indentation) # Count the indented lines.
        statsDict = {} # Initialize a new "dictionary," or associative array.
        # Iterate through each indentation level in the indentation statistics,
        for indentLevel in stats:
            if indentLevel > 0: # ignoring lines that are flush-left, and
                # divide by the total lines of the poem.
                statsDict[indentLevel] = [stats[indentLevel], stats[indentLevel] / self.numLines]
        return statsDict

    def getInfo(self):
        """
        Gets all information about this canto, and prints it, so that it is
        easily human-readable.
        """
        print('Canto number: ', self.cantoNumber)
        print('Number of lines: ', self.numLines)
        print('Number of stanzas: ', self.numStanzas)
        print('Length of stanzas: ', self.stanzaLengths)
        print('Stanza Percentages', self.stanzaPercentages)
        print('Indentation statistics: ', self.indentationStats)

Now we’ll need a set of functions for reading multiple cantos at a time, so that we don’t have to read them each individually. The following class automates the bulk analysis of cantos, so that information about all of the cantos is easily available.

class CantosReader(CantosFile):  
    def __init__(self):
        """
        When this class is instantiated, use the `cantoReader` class to
        gather information about each canto. Store this in a list of
        canto "objects," which will be a meta-list of canto statistics.
        """
        self.cantoObjects = [CantoReader(i) for i in range(1, self.numCantos+1)]

    def getAllInfo(self):
        """ Print statistics for all cantos in the file. """
        # Iterate through the list of all cantos.
        for cantoObject in self.cantoObjects:  
            # Print the statistics for the current canto.
            cantoObject.getInfo()
            # Add a little spacing to make it look nice.
            print('\n')

    @property
    def lineLengths(self):
        """ Make a list of all line lengths for all cantos. """
        return [canto.numLines for canto in self.cantoObjects]

    def getNumStanzas(self):
        """ Make a list of all stanza counts for all cantos. """
        return [canto.numStanzas for canto in self.cantoObjects]

    @property
    def stanzaLengths(self):
        """ Get all the lengths of all the stanzas, grouped by canto. """
        stanzaList = []
        for cantoObject in self.cantoObjects:
            stanzaList.append(cantoObject.stanzaLengths)
        return stanzaList

Now let’s use these functions to find information about cantos I-X.

CantosReader().getAllInfo()

Canto number:  1
Number of lines:  77
Number of stanzas:  2
Length of stanzas:  [57, 19]
Stanza Percentages [0.7402597402597403, 0.24675324675324675]
Indentation statistics:  {8: [3, 0.03896103896103896]}


Canto number:  2
Number of lines:  162
Number of stanzas:  6
Length of stanzas:  [39, 62, 28, 17, 6, 5]
Stanza Percentages [0.24074074074074073, 0.38271604938271603, 0.1728395061728395, 0.10493827160493827, 0.037037037037037035, 0.030864197530864196]
Indentation statistics:  {8: [53, 0.3271604938271605], 9: [7, 0.043209876543209874], 10: [2, 0.012345679012345678]}


Canto number:  3
Number of lines:  43
Number of stanzas:  2
Length of stanzas:  [19, 23]
Stanza Percentages [0.4418604651162791, 0.5348837209302325]
Indentation statistics:  {8: [2, 0.046511627906976744]}


Canto number:  4
Number of lines:  136
Number of stanzas:  8
Length of stanzas:  [12, 44, 12, 13, 7, 12, 11, 18]
Stanza Percentages [0.08823529411764706, 0.3235294117647059, 0.08823529411764706, 0.09558823529411764, 0.051470588235294115, 0.08823529411764706, 0.08088235294117647, 0.1323529411764706]
Indentation statistics:  {8: [42, 0.3088235294117647], 10: [1, 0.007352941176470588]}


Canto number:  5
Number of lines:  130
Number of stanzas:  3
Length of stanzas:  [59, 7, 62]
Stanza Percentages [0.45384615384615384, 0.05384615384615385, 0.47692307692307695]
Indentation statistics:  {8: [19, 0.14615384615384616], 19: [1, 0.007692307692307693], 46: [1, 0.007692307692307693]}


Canto number:  6
Number of lines:  81
Number of stanzas:  4
Length of stanzas:  [36, 17, 21, 4]
Stanza Percentages [0.4444444444444444, 0.20987654320987653, 0.25925925925925924, 0.04938271604938271]
Indentation statistics:  {8: [20, 0.24691358024691357], 30: [2, 0.024691358024691357]}


Canto number:  7
Number of lines:  137
Number of stanzas:  9
Length of stanzas:  [15, 1, 2, 40, 22, 8, 12, 7, 22]
Stanza Percentages [0.10948905109489052, 0.0072992700729927005, 0.014598540145985401, 0.291970802919708, 0.16058394160583941, 0.058394160583941604, 0.08759124087591241, 0.051094890510948905, 0.16058394160583941]
Indentation statistics:  {8: [31, 0.22627737226277372]}


Canto number:  8
Number of lines:  183
Number of stanzas:  5
Length of stanzas:  [56, 21, 23, 57, 22]
Stanza Percentages [0.30601092896174864, 0.11475409836065574, 0.12568306010928962, 0.3114754098360656, 0.12021857923497267]
Indentation statistics:  {16: [1, 0.00546448087431694], 17: [1, 0.00546448087431694], 18: [2, 0.01092896174863388], 22: [1, 0.00546448087431694], 23: [5, 0.0273224043715847], 8: [17, 0.09289617486338798], 41: [1, 0.00546448087431694], 11: [1, 0.00546448087431694], 13: [4, 0.02185792349726776], 30: [3, 0.01639344262295082], 24: [2, 0.01092896174863388]}


Canto number:  9
Number of lines:  268
Number of stanzas:  15
Length of stanzas:  [45, 31, 17, 4, 42, 1, 1, 21, 14, 2, 2, 2, 2, 57, 13]
Stanza Percentages [0.16791044776119404, 0.11567164179104478, 0.06343283582089553, 0.014925373134328358, 0.15671641791044777, 0.0037313432835820895, 0.0037313432835820895, 0.07835820895522388, 0.05223880597014925, 0.007462686567164179, 0.007462686567164179, 0.007462686567164179, 0.007462686567164179, 0.2126865671641791, 0.048507462686567165]
Indentation statistics:  {8: [62, 0.23134328358208955], 34: [2, 0.007462686567164179], 20: [1, 0.0037313432835820895], 22: [1, 0.0037313432835820895], 24: [1, 0.0037313432835820895], 43: [1, 0.0037313432835820895], 28: [1, 0.0037313432835820895], 46: [1, 0.0037313432835820895], 15: [2, 0.007462686567164179]}


Canto number:  10
Number of lines:  194
Number of stanzas:  14
Length of stanzas:  [60, 27, 11, 10, 5, 3, 2, 11, 8, 4, 2, 12, 18, 8]
Stanza Percentages [0.30927835051546393, 0.13917525773195877, 0.05670103092783505, 0.05154639175257732, 0.02577319587628866, 0.015463917525773196, 0.010309278350515464, 0.05670103092783505, 0.041237113402061855, 0.020618556701030927, 0.010309278350515464, 0.061855670103092786, 0.09278350515463918, 0.041237113402061855]
Indentation statistics:  {8: [34, 0.17525773195876287], 1: [1, 0.005154639175257732], 19: [1, 0.005154639175257732], 31: [2, 0.010309278350515464], 39: [1, 0.005154639175257732]}

Since that format is not very easy to read, it might be more helpful to visualize some of this information. First, let’s examine stanza lengths, to get an idea of how long our generated stanzas need to be, and how variable stanza lengths will be. The chart below represents Canto I - Canto V.

df = pd.DataFrame(CantosReader().stanzaLengths)[:4]
df.plot(kind='bar', figsize=(16,8), alpha=0.5)

Stanza Lengths

Judging from the number of stanza divisions and the variation in the lengths of the stanzas, Cantos I-IV appear to alternate between simple and complex line grouping systems. With this information, it might be best to design individual canto writer classes for each canto, rather than try to design a canto-agnostic generator.

Now let’s examine the total line lengths of the first ten cantos.

df = pd.DataFrame(CantosReader().lineLengths)
df.plot(kind='bar', figsize=(16,8), alpha=0.5)

Line Lengths

Here again, the variation is immense, further solidifying the idea that individual subclasses need to be written for each canto. However, since it’d be best not to repeat code for a particular canto, it will be useful to write a parent class that will serve as a toolbox. The following class contains “helper” functions which will be used by the canto subclasses CantoI, CantoII, and so on.

class CantoWriter:
    """
    This class contains "helper" functions which will be used by the CantoI,
    CantoII, etc, subclasses. Underscores in the function names indicate
    when a function is designed for use by another function, and not meant
    to be executed directly.
    """
    def _untokenize(self, line):
        # Wordnet likes to use underscores for compound words,
        # so we have to replace those.
        line = [token.replace('_', ' ') for token in line]

        # Join broken (tokenized) words like ["do", "n't"].
        tokens = [" "+i if not i.startswith("'") and i not in string.punctuation else i for i in line]

        # Join everything else, and remove extranneous whitespace.
        return "".join(tokens).strip()

    def _translatePOS(self, pos):
        # POS tagging returns a certain shorthand for parts of speech.
        # Let's translate this into something WordNet can understand.
        if pos[0]=='N':
            return wn.NOUN
        if pos[0]=='V':
            return wn.VERB
        if pos[0]=='J':
            return wn.ADJ
        if pos[0]=='R':
            return wn.ADV
        else:
            return None

    def _getSynonyms(self, posword, includeHyponyms=False):
        # accepts tuples of (word, POS)
        """
        This function gets synonyms from the Princeton WordNet, optionally
        adding hyponyms, if desired. The function accepts input in the form
        (word, part-of-speech).
        """
        # Get the word.
        word = posword[0]

        # Get the part-of-speech and translate it to something WordNet
        # can understand.
        pos = self._translatePOS(posword[1])

        # Make a list of stopwords,  
        stop = stopwords.words('english')
        if word not in stop: # and ignore them.
            synsets = wn.synsets(word) # Look up the words in WordNet.
        else:
            synsets = [] # Otherwise, return a blank list.
        if len(synsets) > 0:
            # Find the word's hyponyms, flattening the list if necessary.
            hyponyms = sum([x.lemma_names() for x in synsets[0].hyponyms()], [])
            # Find synonyms.
            synonyms = synsets[0].lemma_names()
            synonyms = [word] + synonyms
            if includeHyponyms:
                return synonyms + hyponyms
            else:
                return synonyms
        else:
            return [word]

    def formatLines(self, lines, originalCantoNum):
        """
        Using the original canto as a model, this will break up a given
        set of lines into stanzas, and indent some of the lines.
        It gets the stanza and indentation percentages from CantoReader,
        and applies those percentages to the newly-generated text.
        """
        # Read the original canto to get statistics.
        originalCanto = CantoReader(originalCantoNum)

        # We need to know how long the generated poem is so far.
        numGeneratedLines = len(lines)

        # Get all the stanza percentages, but ignore the last one, since
        # we can infer it from the remaining percentage.
        stanzaPercs = originalCanto.stanzaPercentages[:-1]

        # Iterate through every stanza, except the last.
        for stanzaPerc in stanzaPercs:
            # Find out where the stanza break should go, in percentages.
            stanzaBreak = round(stanzaPerc * numGeneratedLines)
            # Insert a blank space at the stanza break.
            lines = lines[:stanzaBreak] + [' '] + lines[stanzaBreak:]

        # Now indent the newly-generated poem according to the old.
        # First, get indentation statistics about the original canto.
        stats = originalCanto.indentationStats

        # Go through each of the indentation levels listed in its statistics.
        for indentLevel in stats:
            # Compute the number of lines to indent for our generated text.
            numLinesToIndent = round(numGeneratedLines * stats[indentLevel][1])

            # Initialize an empty list so that we can keep track of indented
            # lines.
            indented = []
            # While there are still lines left to be indented,  
            while len(indented) < numLinesToIndent:

                # choose a random line to indent,
                lineNumber = random.randint(1,numGeneratedLines)

                # so long as the line hasn't already been indented.
                if lineNumber not in indented:

                    # Indent the line.
                    lines[lineNumber] = " " * indentLevel + lines[lineNumber]

                    # Add it to the list of indented lines.
                    indented.append(lineNumber)

        return lines

    def show(self, lines, originalCantoNumber):
        """
        Displays the generated canto.
        """
        print('Canto ' + str(originalCantoNumber) + '\n')
        for line in lines:

            # Get the line number. Add 1, since Python starts counting
            # at zero.
            i = lines.index(line) + 1

            # Add line numbers for lines divisible by 5,
            # so that the results will be easier to discuss.
            if i % 5 == 0:
                print(str(i) + ' ' + line)
            else:
                print('   ' + line)

Canto I

In Canto I, Pound translates from Book XI of a renaissance Latin translation of The Odyssey. As he describes it, “I picked from the Paris quais a Latin version of the Odyssey by Andreas Divus Justinopolitanus (Parisiis, In officina Christiani Wecheli, MDXXXVIII)” (“Literary Essays,” 259). Roland John posits that he uses this particular translation, “from an obscure Latin version,” in order to “confirm that he is writing an epic. … [and as an] homage to Homer as the maker of Europe’s first epic” (11). There is more to this canto, of course—additional Homeric material, as well as Pound’s bibliographic commentary—but the Divus translation is the bulk of the canto, and so this is a good place to start.

In the CantoI class below, the variable source_divus is populated with the first seventy lines of Divus’s translation, originally transcribed from Kenner’s facsimile in The Pound Era (352), but corrected with a copy at the Bayerische StaatsBibliothek and Pound’s own transcription (Literary Essays 259). Pound’s transcription expands Divus’s abbreviations and modernizes much of the spelling, but there are still irregularities in the Latin, some of which Pound discusses in “Early Translation of Homer” (ibid. 264). In order to mimic the translation of Divus, the Latin text is machine-translated using an API connected to Google Translate, with the function translateFromLatin. Since Google Translate typically assumes that a word is a proper name if it cannot be found in its dictionaries, and preserves the word as such, this helps to explain the presence of the Latin words that still appear in the generated text below. (Although this could be considered a defect of the translator application, the presence of a few Latin words certainly comes across as Poundian.)

The style of Canto I is alliterative, not unlike Old English verse. John explains that it is a “modified alliterative line similar to that used in [Pound’s] translation of the Early English poem ‘The Seafarer,’ chosen because it is an example of the earliest form of English verse.” William Cookson further identifies this style as “the first example of the overlaying of times and traditions in The Cantos” (4). To achieve this effect, an alliteration function was written.

The alliteration algorithm, alliterate(), possibly the first of its kind, begins by attempting to guess the parts of speech of every word in a line, using the NLTK. From there, it retrieves all synonyms and hyponyms for the words using Princeton University’s WordNet, a thesaurus-like “lexical database” of English words that organizes words into hierarchical relations. These synonyms and hyponyms are then filtered by part of speech, and the function chooses those with the highest numbers of shared first letters. This constructs some veritably alliterative verse, and some surprising lines, as well.

class CantoI(CantoWriter):

    # Get a copy of the source text, a Latin translation of the Odyssey XI.
    source_divus = open('divus2.md').read()

    # We will need to tell other functions what canto number we're writing.
    originalCantoNum = 1

    def translateUsingGoslate(self, text):
        """
        This is an interface to Google Translate. Of the two translation
        algorithms here, this is much faster, but the API provider service
        is unreliable, and so we have to have a backup.  
        """
        gs = goslate.Goslate()
        return gs.translate(text, 'eng')

    def translateFromLatin(self, text):
        """
        This is a backup translation service that also uses Google Translate.
        It's slower, but more reliable.
        """
        txt = TextBlob(text)
        return txt.translate(to="en").string

    def alliterate(self, line, includeHyponyms=False):

        # First, break up the line into words (tokens).
        words = nltk.tokenize.word_tokenize(line)

        # Next, try to find the parts of speech for the words.
        pos = nltk.pos_tag(words)

        # Look up all synonyms and hyponyms for the words in WordNet.
        syns = [self._getSynonyms(tagword, includeHyponyms) for tagword in pos]

        # Make a table of all the first letters of these words.
        firstLettersSet = list(set([word[0].lower() for word in sum(syns, [])]))
        synsFirsts = [[word[0] for word in thing] for thing in syns]

        # Initialize an associative array of letters for storing
        # letter frequencies.
        letterDict = {}

        # Iterate through all the letters and check to see if they're
        # in a block. Make an associative array out of the results.
        for letter in firstLettersSet:

            # Start with the count at zero.
            letterDict[letter] = 0

            # Iterate through each block.
            for block in synsFirsts:
                if letter in block:
                    # Increment the letter count.
                    letterDict[letter] += 1

        # Find the letter that occurs in the most synonyms.
        maxLetter = max(letterDict, key=letterDict.get)

        # Initialize a new list for the newly-alliterated words.
        alliterated = []
        for word, synList in zip(words, syns):
            if synList == [] or len(synList) == 1:
                # Keep all words that were not in the thesaurus,
                # or had only one synonym.
                alliterated.append(word)
            else:
                foundone = False # We haven't found anything yet.
                for syn in synList: # Search through all synonyms.
                    if syn[0] == maxLetter: # If it starts with our letter,
                        alliterated.append(syn) # go with this one.
                        foundone = True # Found one!
                        break # We can stop looking now.
                if not foundone:
                    # Go with the original word if we didn't find anything
                    alliterated.append(word)
        # Stitch the sentence back together.
        return self._untokenize(alliterated)

    def write(self, originalCantoNumber=1):
        """Generate Canto I."""

        # Translate the source from Latin.
        translated = self.translateFromLatin(self.source_divus)

        # Split the result into lines.
        translatedLines = translated.split('\n')

        # Alliterate the lines.
        alliterated = [self.alliterate(line) for line in translatedLines]

        # Format the lines like Canto I, according to statistics
        # gathered from CantoReader().
        broken = self.formatLines(alliterated, originalCantoNumber)

        # Display the result.
        return self.show(broken, 1)

# Instantiate a new copy of the CantoI object defined
# in the class above.
c1 = CantoI()

# Execute the `write()` function defined above.
c1.write()

Canto 1

   Yet, when I came down to the ship, and the sea,
   The vessel was first found in the sea gods,
   And that which was wickedness in the ship, we set and the veils of the black:
   But we did not go inside, taking cues within ourselves
5         As presently as pain, tears pouring Hubert:
   We, however, from the back of the black ship, the prow
   successful reached the bottom of the screen spreading good friend
   Benecomata voiced serious goddess Circe.
   We, however, each of the arms of the transport out in the ship,
10 simply we sat and old pilot diregebat:
   They are spread throughout the whole of this twenty-four hours sails the sea trasientis:
   And he killed time a sun was shadowed all the time on earth,
   But this consists in the territories of the great depths of the ocean:
   There are Cimmeriorum people of the province,
15 Mist and fog covered, nor has it ever had
   Looks bright sun beam,
   Nor when it tends to the starry heaven,
   For neither from the heaven of the earth, when it is turned back on:
   But the dark is dangerous stretches direct men:
20 We then deduced the ship is outside of sheep
   We took ourselves again to the issue of ocean
   , We passed on, so that we came to the place, which he spake, Circe:
   This is a sacred Perimedes Eurylochusque
   Did atomic number 53 have a sharp blade pulling a thigh,
25 The measure of how much elbow pit dug on either side:
   But all around her humble offerings to the dead;
   Multse first, but later with sweet wine:
   Thirdly, again with water, and flour, and white robes, atomic number 53 have mixed for:
   I prayed a passel, but weak heads of the dead:
30 He set out for Ithaca, but the barren woman to bread and butter his ox, which is the best course,
   Sacrificing in their place, filled with good old planks:
           simply privately vowed to sacrifice a sheep Tiresias
   All the black sheep, which are superior to the States:
   After the prayers for the dead nations has precautionibusque
35 I prayed, received a passel of sheep:
   And the blood ran out of the cavity black in color, they are congregataeque
   Cadavres dead souls from snake pit,
   Nymphs iuvensesque endure many things of the elders,
   And tender virgins, who had lately woeful way he disposed and,
40 many of the maimed air lances
   The work force were killed in the war, the bloody weaponry, they,
   Many people who came from elsewhere around the pit else
   With a great shout, but I must pale fear took in fight.
   Now, after spur his comrades say
45 Cattle, which have already been brutally bump off, still in the air,
   Flay their paraphernalia, offering prayers to the gods,
   Gallant to Pluto, and praised Proserpine.
   simply I'm drawing a sharp sword from his thigh,
   See, I did not dead inability heads
50 Blood was most move before I could hear Tiresias:
   simply first semen the Elpenoris spouse:
   You have not yet buried under the world there was a wide,

   We have left the organic structure, in the house of Circe,
55 Infletum and a plea for another urgent task:
   This indeed as iodine am the illusionist, lachrymator And, pitying in my mind,
   And the swift addressed shouting words:
   Elpenor, how have you come under the cloud, cloud;
   Feet than I anticipated being in the black boat?
60 So atomic number 53 said,'But this adult male, to me, and mourned for his answer Him a word:
   Laeritiade noble, wise Ulysses,
   Fate of hurt me bad, and a mass of wine:
   When you lie down, however, in the house of Circe, I will not be perceived by
   atomic number 53 once turned down a ladder going through a long,
65 But I fell against the wall, but to me the neck
   Of the strings is broken, the soul of them that travel down to the pits:
   But now, I pray, not in the presence of those who were to come in the time to come;
   Through his wife, and the father, who was brought up a small existing,
   Telemachumque the only talker in the houses you left tail.
70 I know that this house is going to the pits
           Aeaeam imellens benefabricatam island in the boat:

A brief comparison of this generated Canto with Pound’s original might be useful here. Pound’s canto begins with the line “and then went down to the ship” (A Draft of XXX Cantos, 3); the generated canto begins “yet, when I came down to the ship, and the sea”—Pound’s version exhibits much more economy of language, with 30% fewer words in the line. The character represented by “I” in the generated version is removed in Pound’s, which might serve to blur who is meant by the “I”—is it Odysseus? Pound? (This economy of speech is something we will try to correct in the generation of Canto II, using the meonymize function.) While line 3 in Pound is alliterated with “s” words—“we set up mast and sail on that swart ship,” the generator apparently alliterates on “w” words: “and that which was wickedness in the ship, we set.” In line 5 in this generated version, we see, mysteriously, “tears pouring Hubert” in place of Pound’s “heavy with weeping.” (Who is this Hubert? One can only suspect that this is either a translation error or an unusually distant hyponym that the alliterate function is inserting.) In at least one case, the alliterate function is alliterating even more than Pound does—Pound’s “with glitter of sun-rays” is rendered here as “looks bright sun beam” (line 16).

In some cases, the function alliterates until the line makes no sense—line 24, for instance, reads “did atomic number 53 have a sharp blade pulling a thigh,” which corresponds to Pound’s line “drawing sword from my hip.” The alliteration algorithm, trying to alliterate with words beginning with “a,” is looking for synonyms for the word “I.” Not realizing that it is a personal pronoun, it guesses that the word is the chemical symbol for iodine, and suggests the possible synonym “atomic number 53,” which starts with the letter “a.” To fix this issue, we will try to find synonyms for Canto II with a word sense disambiguation algorithm in place.

Canto II

Canto II presents a very different challenge than Canto I. As John Childs puts it, “Since Canto 1 is largely given over to discourse in its retelling of a portion of the Odyssey, the devices of imagism are largely absent, and, in fact, The Cantos seem to alternate between the two poles of Imagist”lyricism" and discursive “historicism”; Cantos 1 and 2 are representative of such alternation" (44). One of the characteristic features of this second imagist mode of Canto II, Childs argues, is metonymy. In fact, he continues, it is the use of metonymy that distinguishes imagism from symbolism. The algorithm that will be used to generate Canto II, then, will apply metonymic principles to a mix of Pound’s source materials.

Broadly, metonymy is, in M.H. Abrams’s definition, a process by which “the literal term for one thing is applied to another with which it has become closely associated because of a current relation in common experience” (120). A common example is “the crown” as a term meaning a king. One type of metonymy is a synechdoche, where “a part of something is used to signify the whole, or (more rarely) the whole is used to signify a part” (ibid.). Using the Princeton WordNet, we might be able to emulate metonymy to a small degree by navigating its hierarchy of word relations. The NLTK WordNet API provides functions for finding meronyms for words, for instance. A meronym is a word that is a component of another word, and WordNet charts both “part” and “substance” meronyms. Here are some example meronyms for the word “tree”:

wn.synset('tree.n.01').part_meronyms()

[Synset('burl.n.02'),
 Synset('crown.n.07'),
 Synset('limb.n.02'),
 Synset('stump.n.01'),
 Synset('trunk.n.01')]

wn.synset('tree.n.01').substance_meronyms()

[Synset('heartwood.n.01'), Synset('sapwood.n.01')]

The metonymy function below starts by determining the part of speech of a word, attempts to disambiguate its sense using the Lesk algorithm for word sense disambiguation (WSD), and with the resulting word sense or “synset,” searches for meronyms. If none are found, it will try searching for other ways to convert the line into a more metonymic line.

Another way that metonymy functions is, as David Lodge explains, as a “condensation of contexture,” effected by a “transformation of a notional sentence … by means of deletion of portions of the syntagm” (quoted in Childs 36). A simple instance of this deletion is the removal of the word “like” that can transform a simile into a (metonymic) metaphor. Pound’s affinities for the economy of words of ancient Chinese poetry, coupled with his terse haiku-like experiments, makes deletion an obvious choice for Poundian generative poetics. In fact, in an early essay on the Chinese poem “The Jewel-Stairs Grievance,” he explains that despite the fact that he has “never found any occidental who could ‘make much’ of that poem,” still “everything is there, not merely by ‘suggestion’ but by a sort of mathematical process of reduction” (Early Writings, “Chinese Poetry”). What we shall be attempting here is literally a mathematical process of reduction.

Rather than delete words at random, however, we will design an informed algorithm for metonymic deletion. Citing William E. Baker, Childs argues that one of the syntactic modes of this metonymic deletion is “nominalization,” that deletion “gives primary emphasis to the noun, and that the great majority of fragments are classified as such because they contain a noun or noun phrase which lacks a finite verb to provide grammatical ‘sense’ to the structure” (quoted in Childs, 58-9). With this in mind, the function metonymize will call the function deleteVerbs in order to delete a small proportion of verbs in each line.

The algorithm below will start with eight of Pound’s sources, taken from Carroll Terrell’s Companion to the Cantos. These include selections from Ovid’s Metamorphoses books III and X, Euripedes’s Bacchae, Homer’s hymn to Dionysos, and selections from the Odyssey and Iliad. The Ovid translations are those of Brookes More, and the Homer translations are those of Samuel Butler; both were downloaded from Perseus. The text of Robert Browning’s Sordello was downloaded from Wikisource, and the Mabinogion was downloaded from Project Gutenberg. These sources are “mixed” together with the function sourceMix, which interpolates the source texts every X number of lines. That mix is then “metonymized” with the metonymize function previously described. From there, line breaks are added, and the text is formatted with stanza breaks and indentation to resemble the original Canto II.

class CantoII(CantoWriter):
    # Gather text files of some of Pound's sources for Canto II.  

                    # Robert Browning's epic poem _Sordello_
    sourceFiles = ['sordello.txt',  

                    # Ovid's Metamorphoses, III and X
                   'ovid3.txt',
                   'ovid10.txt',

                   # The Odyssey XI (an English translation this time)
                   'odyssey-xi.txt',

                   # The Mabinogion
                   'mabinogion.txt',

                   # The Illiad III
                   'iliad-iii.txt',

                   # Homer's Hymn to Bacchus
                   'homeric-hymn7.txt',

                   # Euripedes's _Bacchae_
                   'euripedes.txt']

    """ An associative array with the file names and file contents. """
    sourceTexts = {filename: open('II/'+filename).read().split('\n') for filename in sourceFiles}

    def reloadSourceTexts(self):
        """ Reload the source files when necessary. """
        self.sourceTexts = {filename: open('II/'+filename).read().split('\n') for filename in self.sourceFiles}

    @property
    def sourceMix(self, chunkSize=2):
        # Get a fresh copy of the source texts, in case anything has changed.
        self.reloadSourceTexts()
        # Initialize a new list to hold the mixed lines.
        sourceMixLines = []
        # Set a flag so that we know when to stop.
        go = True
        # Keep going so long as this flag is True.
        while go:
            # Iterate through all the source texts,
            for source in self.sourceTexts.values():
                # so long as there's something left to iterate through.
                if len(source) > 0:
                    # Get the first X lines
                    for line in source[0:chunkSize]:
                        # (as long as the line isn't empty),
                        if line != '':
                            # and add it to our new list.
                            sourceMixLines.append(line)
                    # Now delete those lines from the source.
                    del source[0:chunkSize]
                else:
                    # If the source empty, we can stop now.
                    go = False # Stop
        return sourceMixLines

    def _translatePOS2(self, pos):
        """
        The Lesk algorithm used in word sense disambiguation uses the abbreviations
        'a', 'n', and 'v' for adjectives, nouns, and verbs, respectively, but
        the POS tagger uses different abbreviations. This function translates one
        to the other.
        """
        if pos[0] == 'J':
            return 'a'
        if pos[0] == 'V':
            return 'v'
        if pos[0] == 'N':
            return 'n'
        else:
            return None

    def _getLineSynsets(self, tokens):
        """
        Tries to detect the senses of the words by their context.
        It uses the Lesk algorithm to try to guess at the sense of the word
        being used, by the word's lexical similarity (thesaurus distance) to
        other words in the line.
        """
        # Try to detect the parts of speech first.
        pos = nltk.pos_tag(tokens)
        # Don't try to look up frequently used words like "a," "an," and "the."
        stop = stopwords.words('english')
        # Next, let's do some word sense disambiguation on the line to make sure that we're dealing smartly with each line.
        synsets = [nltk.wsd.lesk(tokens, word[0], self._translatePOS2(word[1])) if word[0] not in stop else None for word in pos]
        return synsets

    def _getFirstMeronym(self, synset):
        """
        This is a helper function to get the first part- or
        substance-meronym for a given sense.
        """
        pm = synset.part_meronyms()
        sm = synset.substance_meronyms()
        if len(pm) > 0:
            return pm[0].lemma_names()[0]
        elif len(sm) > 0:
            return sm[0].lemma_names()[0]
        else:
            return None

    def _getFirstHyponym(self, synset):
        """ Gets the first hyponym for a synset. """
        hn = synset.hyponyms()
        if len(hn) > 0:
            return hn[0].lemma_names()[0]

    def _deleteVerbs(self, words, synsets):
        """ Deletes verbs in a line, about half the time."""
        for word, synset in zip(words, synsets):
            # Provided the synset exists,
            newSynsets = []
            newWords = []
            if synset is not None:
                # and if the part of speech is a verb,
                if synset.pos() == 'v':
                    # (only do this about half the time),
                    if random.random() > 0.5:
                        # remove it from the list of synsets,
                        for item in synsets:
                            if item is not None and item is not synset:
                                newSynsets.append(item)
                        for item in words:
                            if item is not None and item is not word:
                                newWords.append(item)
                        words, synsets = newWords, newSynsets
                        # and remove it from the word list.
        return words, synsets  

    def metonymize(self, line):
        """
        This looks through every word in a line, and tries to find a
        meronym for it. If it can't find a meronym, it substitutes a hyponym.
        """
        words = nltk.tokenize.word_tokenize(line)
        synsets = self._getLineSynsets(words)
        if synsets is not None and words is not None:
            words, synsets = self._deleteVerbs(words, synsets)
        # Let's first look for part meronyms.
        newLine = []
        for word, synset in zip(words, synsets):
            if synset is not None:
                firstMeronym = self._getFirstMeronym(synset)
                firstHyponym = self._getFirstHyponym(synset)
                if firstMeronym is not None:
                    newLine.append(firstMeronym)
                elif firstHyponym is not None:
                    newLine.append(firstHyponym)
                else:
                    newLine.append(word)
            else:
                newLine.append(word)
        return self._untokenize(newLine)

    def write(self):
        originalCanto = CantoReader(2)
        numLines = originalCanto.numLines
        # Get the mixed sources from `sourceMix` above.
        mix = self.sourceMix[:numLines]
        metonymized = " ".join([self.metonymize(line) for line in mix])
        wrapped = textwrap.wrap(metonymized, 50)
        formatted = self.formatLines(wrapped, 2)
        self.show(formatted, 2)

c2 = CantoII()
c2.write()

Canto 2

   When amaze; Who will rehear anticlimax anticlimax?
   Who me `` Then I Minos son scepter in his hand
           canter in affirmation on the King Arthur was at
           Caerlleon upon Usk; and one day atrium Narcissus'
5          
           fate, when known throughout the farmstead city
   center cities have it coming When the its own
   aristocrat, the Trojans brevet as a flight I will
   of Dionysus Semele mull monad, the badlands
10 Thebans—Dionysus, whom once Semele, Kadmos'
   aftereffect bring by an ancient crime.—But I am
   weary with unaccustomed corvee; and see! a poplar
           good luck aim the dead poltergeist and round him
5          
15 son of Urien, and Kynon the son of Clydno, and Kai
   the son of Kyner; and Gwenhwyvar and her blind
           Tiresias, —mighty anticipator. Yet Pentheus, bold
   despiser of the Gods, son of Echion, scoffed at of
   dark meat whooping crane or raindrop and midwinter
20 strive them over the airflow branch water the
           beach of the fruitless bay, come across like a
   stripling in the first flush of manhood: his rich,

   mother's daughter, bore, consign by a lightning-
25         bearing blaze. And having taken a mortal dispersed
   phase instead of a convenient lawn a good pass
   uranium 235 blow hill-top once, despite the bustle
   And slack of audience, Pentapolin Named liquid
   oxygen' the Naked cuff, monad Hades, to understudy
30 his clause[ dikai] upon them. handmaidens at

   crochet by the window. And if it should be said
           that there was a porter at all his compliment,
           and, ball of man catcall the great anticipator,
35 upbraided him his hapless paper loss of of Okeanos
   to bring death and annihilation on the Pygmies,
           airspace dark bristle was luff about him, and on
           his strong hard shoulder academic gown a god's,
           iodine-125 am here at the Fountain of Youth of
40 Dirke and water iodine-125 gravestone here fall
           back the turf and, pillowing her ego against his
           single out Sordello, catch on murkily about With
           ravage of six long sad hundred month. Only ``
            After him I Jacob's rod the ghosts of Arthur’s
45         alcazar, there was none. Glewlwyd Gavaelvawr
   eyeful. but the Achaeans countermarch silently, in
   high heart, and to there swiftly head sea bay
5          
   thunder-stricken mother here near the palace, and
50 fag end remnants breast and mingling kisses with
   her words, she told fairytale me Never, —I should
           warn you first, — Of my own default option that he
   had electrocute upon the mountain peak, and he had
   a great bronze clubroom in his hand, unbreakable
55         for guests and strangers, and to receive them with
   honour bearing And column white eighties Tiresias
   of As when bell eyelet wind miserable doom signs
   to one bounce and blaze of Zeus cascade
           everlasting indignity of Hera against Perhaps you
60 may have heard of a swift damsel, who had this, if
   not the worst Yet not the best crutch subpoena
           picometer anticlimax iodine-125 story ever and
   ever. habit Court of Saint James's channel to the
   Hall or to the presence-chamber, and those to
65         thee, if, light denied, thine eyes, most  upon the
   mountain peak blouse, bad for shepherdess but
   better than bandit for and big board their bay
   exultingly; for they thought him the Kadmos behalf
            oracle mother's daughter his gravitate much faster
70         than in the forth so By commemorate view, The very
   man as he was wont to do, And `` And I card Tityus
   son of Gaia stretched upon the flat and discourse
   some nine acres of archipelago. couple who average
            to take up their lodging. ritual dancing! The
75 morning will come, and soon the illuminance will
           dawn, when cabochon further the chalk dust from
           under their amphibrach as they made all son of
   article him with around with the cluster-bearing
   leaf shape of the allamanda.  have said whether
80         her swift acceleration or her beauty you though
   iodine-125 might be Aegypiidae on either beaks
   calves' liver In the bare bones of the atrium King
           Arthur canter upon a capital of green rushes, over
           which was circulation a advent known— all hail the
85 new god Bacchus! Either millenary must build a
           column to this Anglo-Saxon deity, or shalt speed
   over the flat. him, and the osier alight far away
   from his hands amphibrach simper evening
           iodine-125 have the Phrygians, the sun-parched
90         flat of the Persians, was more worthy of your
   hallelujah. When this damsel once doom past
   continental divide its hateful surge, car rental
           of all men monad one man them he had fly in the
   face of Zeus’ mistress Leto as she was covering of
95 flame-coloured satin, and a air cushion of red
   satin was under his elbow. be torn asunder; thy
   be, throughout the greenwood birdlime, will infect
            the finger hole with When they were close up with
   one another, Alexander forward anglophile champion
100 aperture. coxswain catch out at once to his
   fellows and call: arch wintry land of the Medes,
   and blessed Arabian Desert, and bridal, never will
   have beggary of benedick, who will only be second
   moment preterit iodine-125, first to last His
105         headway as you preview it, not a live out through
   Panopeus on her way to Pytho. Then Arthur babble,
   “If I thought you would not sanguinary streams;
   and thy life-blood bespatter blotch blots side.
   hard shoulder waterskin panther, blade golden calf
110         all of Asia which charge along the skid of the
   salt head sea with its beautifully-towered cities
             full of your birth trauma. For your best good you
   should avoid the tie birth trauma whit More in the
   secret than yourselves who canter incline to `` I
115 also the stood in disparage me wait course my
   sisters fall iodine-125 have millenary fusarium
   wilt two harpoon with bronze bravest of the
   Achaeans to meet up with him in single bind,
   strong that he is? Not even the well-built

Unlike the generated Canto I, the Canto II generated above bears almost no resemblance to Pound’s original canto, despite the use of many of Pound’s sources. It is not without its moments of surprisingly vivid imagery, however: “dark meat whooping crane or raindrop and midwinter / strive them over the airflow branch water” (l. 19-20). “Branch water” is, somewhat mysteriously, a meronym of “water,” but is surprisingly Poundean in resonance if we recall his “wet black bough” from “In a Station of the Metro.” As in Canto I, the appearance of isotopes (“uranium 235” in line 27, and “iodine 125” in line 39) signals a word sense disambiguation error, despite the use of the Lesk algorithm used in the CantoII class. (A future version of this program might try to filter out all chemical elements from the hyponym tree.) At the very least, it ends in medias res, like Pound’s canto. Where his ends “and…,” this canto ends with the sentence fragment “not even the well-built.”

Conclusions

While the translation-and-alliteration technique of Canto I was reasonably successful, the mix-and-metonymize method of Canto II exhibited mixed success. In both cases, a more sophisticated set of functions would be needed to pass a poetic Turing test. For instance, a future version of this program might not only subsitute hypo/meronyms and delete verbs while metonymizing, but might translate “NN1 of a NN2” syntactic constructions into NN2-NN1 forms that would more closely resemble the Anglo-Saxon metaphoric compound words that Pound uses. It could also further nominalize a line by converting some verbs into nouns by traversing the lexical hierarchy, looking for nouns that are the smallest distance away. Given the time constraints of this project, however, this experiment proved to be a useful exercise in translating literary analysis into an algorithm, or, put differently, using the algorithm as a mode of literary analysis.

It is sometimes sufficient to analyze a poem by saying that it employs certain literary devices, but it is quite another to explain to a computer exactly how those devices work, on the level of the word, or even the letter. The process of analyzing a poem in order to reconstruct it is, in some small ways, a much more involved task than simply analyzing the poem for the sake of analysis. If we are to contend, for instance, that Pound employs metonymy in The Cantos, then to explain it to a computer, we have to specify exactly what we mean by metonymy, and exactly how it functions, lexically and syntactically. In that sense, although this experiment does not discover anything wildly new about Pound’s work, and neither does it succeed in creating a great poem, it nonetheless paves the way for more experiments of this kind.

Works Cited

Bush, Ronald. The Genesis of Ezra Pound’s Cantos. Princeton, N.J.: Princeton University Press, 1989. Print.

Childs, John Steven. Modernist Form : Pound’s Style in the Early Cantos. London: Associated University Press, c1986. Print.

Cookson, William. A Guide to the Cantos of Ezra Pound. London: Anvil Press Poetry, 2001. Print.

John, Roland. A Beginner’s Guide to the Cantos of Ezra Pound. Salzburg, Austria: Institut für Anglistik und Amerikanistik, 1995. Print.

Kearns, George. Guide to Ezra Pound’s Selected Cantos. New Brunswick, N.J.: Rutgers University Press, c1980. Print.

Kenner, Hugh. The Pound Era. Berkeley: University of California Press, 1971. Print.

Korg, Jacob. “The Dialogic Nature of Collage in Pound’s Cantos.” Mosaic: A Journal for the Interdisciplinary Study of Literature 22.2 (1989): 95. Print.

Pound, Ezra. A Draft of XXX Cantos. New York: New Directions, 1940. Print.

——. Early Writings: Poems and Prose. Penguin, 2005. Print.

——. Literary Essays of Ezra Pound. New Directions Publishing, 1968. Print.

Terrell, Carroll Franklin. A Companion to the Cantos of Ezra Pound. Berkeley: University of California Press, c1980-. Print.

Macroetym: a Command-Line Tool for Macro-Etymological Textual Analysis

2016-02-01

I’m proud to introduce macroetym, a command-line tool for macro-etymological textual analysis, which is now available for download with the Python package manager, pip. It’s a complete rewrite of The Macro-Etymological Analyzer, the web tool for macro-etymological analysis I wrote a few years ago, first described in this post, and presented at DH2014. It can now analyze any number of texts, and texts in 250 languages. Here are a few examples of the program in action:

A simple comparative macro-etymological analysis of two texts

$ macroetym wells-time-machine.txt woolf-mrs-dalloway.txt

              wells-time-machine.txt  woolf-mrs-dalloway.txt
Austronesian                0.045788                0.021177
Celtic                      0.068681                0.047649
Germanic                   40.885226               40.334075
Hellenic                    0.840965                1.080051
Indo-Iranian                0.015263                0.153537
Latinate                   57.724359               57.722893
Other                       0.137363                0.333545
Semitic                     0.282357                0.243541
Turkic                      0.000000                0.031766
Uralic                      0.000000                0.031766

A more verbose analysis of a single text

$ macroetym woolf-mrs-dalloway.txt --allstats

                                woolf-mrs-dalloway.txt
Anglo-Norman                                      7.23
Angloromani                                       0.03
Arabic                                            0.06
Aragonese                                         0.06
Dutch                                             0.29
Dutch, Middle (ca. 1050-1350)                     0.27
English, Old (ca. 450-1100)                      36.56
French                                            7.81
French, Middle (ca. 1400-1600)                    3.96
French, Old (842-ca. 1400)                       21.58
German                                            0.06
German, Middle Low                                0.12
... etc.

An analysis of all the books of Paradise Lost

$ macroetym paradise-lost-books/* --showfamilies Latinate        

          bk/book01.txt  bk/book02.txt  bk/book03.txt  bk/book04.txt
Latinate      52.622816      56.005313      52.644493      50.522588   

          bk/book05.txt  bk/book06.txt  bk/book07.txt  bk/book08.txt
Latinate      55.929858      56.608863       51.46886      54.492665   

          bk/book09.txt  bk/book10.txt  bk/book11.txt  bk/book12.txt  
Latinate      53.625632      54.745275      50.982633      52.195609

Machine-readable output

macroetym paradise-lost-books/* --csv > pl-books.csv

Analysis of texts in languages other than English

macroetym madame-bovary.txt --lang=fra

Installation

Install macroetym using pip3:

pip3 install macroetym

Alternatively, grab the source code on GitHub and install locally:

git clone https://github.com/JonathanReeve/macro-etym
cd macro-etym
pip3 install .

Contributing

Macroetym is free and open-source software! There are a number of open issues with the program that need addressing. If you know a bit of python, feel free to hack around on the code as you see fit. Pull requests are very welcome. Non-code contributions are also welcome in the form of bug reports, documentation, or experiments in text analysis that use the program.

A Macro-Etymological Analysis of Milton’s Paradise Lost

2016-07-12

One of Milton’s terms for the expansive, empty gulf separating the Earth from Hell is the “abyss.” The word appears eighteen times in Paradise Lost, and in seven out of twelve of the poem’s books. It is variously described as “vast,” “dark,” “infinite,” “hollow,” “wild,” “wide,” “desolate,” and “vexed,” (1.21, 2.405, 2.518, 2.910, 3.84, 4.936, 10.314) but also more epistemologically as “immeasurable,” “untractable,” and “unbottomed” (7.211, 10.476, 2.405). Besides emphasizing the distance of Satan’s followers’ fall, these last three Chaotic epithets highlight the etymology of “abyss,” as derived from the ancient Greek ἄβυσσος meaning, according to the OED, “bottomless, unfathomed, unfathomable, boundless” (“Abyss, N.”). This is but one example of the way Milton acknowledges the origins of words, and how the logic of his poem depends heavily on etymology. Much has been made of this. On this basis, critics such as Samuel Johnson, F.R. Leavis, and T.S. Eliot dismiss Milton as un-English. More recently, Christopher Ricks, Hannah Crawforth, and John Hale celebrate him for his multilingual style. However, whether approving or disapproving of this method, no critic has yet attempted to quantitatively measure the etymologies of Milton’s words. The following describes an experiment in macro-etymological analysis of Milton’s Paradise Lost, whereby the origins of the epic’s words are quantified by book, section, and speaker. This experiment hopes to reveal trends in the voices and resonances of the poem’s etymological registers that invite the reader to imagine Biblical or classical parallels.

Milton is perhaps the ideal subject for a macro-etymological analysis. By some accounts, he was familiar with around ten languages, and wrote fluently in at least four (Hale 8). Among these were French, Latin, and Ancient Greek: the major ancestral languages of English. Milton’s multilinguality is one factor that leads Roy Flannagan to dub him “the most etymological of poets,” explaining that “he may use a word in two English senses, several Latin senses, and a Greek sense, all at the same time,” citing in particular his use of “essence” and “effluence” (Milton, The Riverside Milton 316). John Carey and Alistair Fowler observe that “Milton would of course know the etymology of many of the words in Paradise Lost,” and add that “the interplay between words of Romance and of Anglo-Saxon origin is an important feature of Milton’s style” (Milton, The Poems of John Milton 431). It is exactly this interplay that the present experiment proposes to measure.

In an obscure Latin work of Milton’s, The Art of Logic, he argues that etymological work can help to discover the truer (more divine) sense of a concept. He does so using a lapsarian metaphor quite familiar to readers of Paradise Lost and a more specifically linguistic metaphor: that of the tower of Babel.

Languages, both that first one which Adam spoke in Eden, and those varied ones also possibly derived from the first, which the builders of the tower of Babel suddenly received, are without doubt divinely given; hence it is not strange if the reason of primitive words is unknown. But as to those words that are derived or composite, either their origins are to be sought in other languages ancient and now obsolete, or by their own antiquity and the usually corrupt pronunciation of the lower classes are so changed, and by the habit of writing them falsely are so obliterated as it were that a true notation of words very seldom may be had. (Milton, A Fuller Institution of the Art of Logic, Arranged After the Method of Peter Ramus 219–21)

The “sudden receiving” of language at the tower of Babel also takes place in the concluding book of Paradise Lost:

Quite out their native language, and instead
To sow a jangling noise of words unknown:
Forthwith a hideous gabble rises loud
Among the builders; each to other calls
Not understood, till hoarse, and all in rage,
As mocked they storm; great laughter was in heaven
And looking down, to see the hubbub strange
And hear the din; thus was the building left
Ridiculous, and the work Confusion named.

This passage features a smattering of onomatopoeia—“jangling,” “gabble,” and “hubbub”—words with non-classical etymologies, that is, “primitive words” whose “reason” is “unknown.” The OED notes that although “Babel” and the onomatopoetic “babble” are etymologically unrelated, it is related to French “babel,” “confusion of opinions,” and in Genesis 11 it is “folk-etymologically associated with Hebrew bālal ‘to confuse, confound.’” (“Babel, N.”). The linguistic fall from grace that Milton recreates in this Babel scene, therefore, can be viewed as etymological.

Later in The Art of Logic, Milton illustrates his logical system by arguing on Latin etymological grounds that the statement “he is a man, therefore he is from the earth,” homo ab humo, is preferable to its opposite, “he is from the earth, therefore he is a man,” humo ab homo (Milton, A Fuller Institution of the Art of Logic, Arranged After the Method of Peter Ramus 219). This rhyming etymological connection is one which fuels God’s pun in Paradise Lost: “because in thee / Love hath abounded more then Glory abounds / Therefore thy Humiliation shall exalt / With thee thy Manhood also to this Throne” (3.311-14). This sense of “humiliation” is not the OED sense 2 of “humiliate,” in which it is most commonly used today, “to lower or depress the dignity or self-respect of,” although it retains these undertones (“Humiliate, V.”). It is closer to the first sense, now obscure, “to make low or humble in position,” but closer still to the etymological sense of the word, ultimately descended from humus, “ground, earth” (“Humble, Adj.”). Thus, the Son “humiliates” himself by going down to Earth. Furthermore, by incarnating himself, he is enacting the same kind of transmogrification of Genesis 2.7, where Adam is formed from earth, a literalization of humo ab homo.

Milton’s etymological sense, that which refigures “humiliation” into “Earth-going,” is exactly that to which some critics have objected. Some have called this Milton’s “Latinity,” and argue that it detracts from his poetic power in English. Samuel Johnson ridicules Milton’s “play on words, in which he delights too often,” and counts among his faults his propensity “to use English words with a foreign idiom” (Johnson 84, 86). Johnson concludes that “of him, at last, may be said what Jonson says of Spenser, that he wrote no language, but has formed what Butler calls a Babylonish Dialect, in itself harsh and barbarous, but made by exalted genius and extensive learning” (86).

T.S. Eliot happily quotes this passage from Johnson in his critique of Milton, adding that Milton has been a “bad influence” over centuries of poets, in part due to “the peculiar kind of deterioration … to which he subjected language” (32). This deterioration he attributes to Milton’s affinity for Latin and Greek: “Milton writes English like a dead language. The criticism has been made with regard to his involved syntax” (35).

Similarly, Joseph Addison, writing in The Spectator in the eighteenth century, derides Milton for his poor punning. He quotes from a passage in Book 6 where Belial answers Satan, calling it “nothing else but a string of puns, and those too very indifferent” (37):

stumbl’d many, who receives them right,
Had need from head to foot well understand;
Not understood, this gift they have besides,
They show us when our foes walk not upright. (6.609-29)

The pun that Addison notices is a kind of literalizing folk etymology of “understand,” refiguring comprehension as a kind of posture, an “upright” “standing” “from head to foot.” These kinds of puns happen throughout Paradise Lost, and they are often etymological in nature. It is this kind of linguistic wordplay that invites an analysis of the poem attentive to its etymological registers.

Experimental Design

The following macro-etymological analysis was conducted with a tool I developed for this purpose called the Macro-Etymological Analyzer. The tool divides a given text into its constituent words, lemmatizes the words (transforms them into their dictionary forms), removes commonly-used words such as “a” and “the” (“stopwords,”) and looks up each remaining word in the Etymological Wordnet, a multilingual etymological database created by computer scientist Gerard de Melo. If the program finds that a word’s ancestor is Middle English, its default behavior is to go one generation deeper, since a large proportion of English words have Middle English ancestors. The program then tallies the words of the text according to their origin languages and origin language families. This groups together words descended from Old English, Old Norse, and other Germanic languages under the language family “Germanic,” and groups together words descended from Latin, French, Anglo-Norman, and other Romance languages under “Latinate.” Words descended from Ancient Greek are categorized as Hellenic, and so on.

Although the Macro-Etymological Analyzer was originally written as a web app designed to analyze single files, I rewrote the program in Python with a command-line interface (CLI), in order to allow it to be scriptable, and allow for easy analysis of multiple files. Thus, instead of uploading individual files to a server and analyzing them one-by-one, users may execute the CLI interface, leveraging shell expansion: python macro_etym.py paradise-lost-books/*.txt, where *.txt will be expanded to match all the text files in the directory. Furthermore, the tool was modified so that it can easily output a file formatted with comma separated values (CSV), an easily machine-readable format. This allows for simple post-analysis of the data generated by the initial macro-etymological analysis.

Apart from the tool creation, preparation of the Paradise Lost text was one of the most labor-intensive processes in this analysis. To begin, an edition of Paradise Lost richly marked-up in Text Encoding Initiative Extensible Markup Language (TEI XML) was retrieved from the Oxford Text Archive. The greatest difficulty of dealing with this text was that the orthography was retained in its original, a feature incompatible with the modern spellings used in the Etymological Wordnet. To correct this problem, I used the VARD 2 tool to normalize the spelling in a quasi-supervised way. First, the text was automatically corrected using VARD’s native dictionary, given a threshold of 40% certainty. From there, the text was corrected by hand, manually replacing thousands of words with their modern equivalents. This resulted in a new, semi-automatically generated edition of the text, where normalized variants are marked up as, for instance, <normalised orig"exprest" auto=“true”>expressed</normalised>=.

One of the most useful features of this TEI XML edition is that speech attribution is semantically marked up with milestones, as in <milestone n"narr" unit=“sp”/>=, where narr is the narrator of the poem. In this case, the narrator’s speech follows this tag. This allows for the extraction of speech elements, although not in the usual way, using an XML parsing library. Instead, I wrote a script to retrieve the all the speech between these tags using regular expressions. The script categorizes each text chunk by speaker, and writes the aggregated text to individual text files, using the character’s name as the filename. This allows for a simple macro-etymological analysis using the shell expansion command outlined above.

In order to analyze the individual books of the poem, I wrote a similar script that parses the XML and extracts the text contained within <div type"Book">= tags. To divide these into subsections for more fine-grained analysis, I constructed a simple BASH command that uses the GNU command split to divide each book into equal-sized chunks: for file in *; do split -n 10 $file ${file:r}; done.

Results

Overall, once stopwords are removed, roughly 60% of the words of Paradise Lost are Latinate, from Anglo-Norman, French, Middle French, Old French, and Latin, mostly. Another 35% are Germanic, from Old English, Old Norse, or German. The remaining few percentage points are divided between Hellenic (Ancient Greek, at around 1%), Semitic (Hebrew, 0.25%), and other language families. Since Latinate and Germanic proportions account for the majority of the total words of the poem, they correlate negatively with one another. A high proportion of Latinate words in a section almost always means a low proportion of Germanic words. For this reason, this study will mostly examine the proportions of Latinate words in Paradise Lost.

Figure 1: Paradise Lost Books, Proportions of Latinate Words

Figure 1 shows the overall proportions of Latinate words in the books of Paradise Lost. The Latinity of the epic starts, one might say, in medias res, grows as the rhetoric of Hell reaches a climax in Book 2, but dips back down by Book 3 with the less Latinate dialogue between God and the Son. Book 4 is the absolute nadir, representing the lowest proportion of Latinate words: this corresponds with Satan’s (and our) first glimpse of Adam and Eve, and the natural world of Eden which Milton represents using words of Germanic origin. After Book 4, the Latinity of the poem increases until it reaches an absolute climax in Book 6. Since Raphael relates the heavenly war in Book 6 using the armaments and materials of classical epics, this high proportion suggests a correlation with a classicist mode. After Book 6, the books alternate in Latinity, although decreasing steadily until the poem ends.

When this analysis is conducted on a finer scale, that is, when each book is broken into ten sections, a different picture emerges. Figure 2 shows proportions of Latinate words per subsection. Sharp alternations between high and low Latinate proportions characterize most of this poem—the sawtooth shape suggests that Milton is quickly switching rhetorical modes, and that this switching increases its pace as the poem progresses.

In this finer analysis, the most Latinate passage is not in Book 6, in fact, but directly before it, in the penultimate section of Book 5, ai (sections are ordered aa-aj). This section starts at line 716 and continues through 815. It contains Satan’s speech to his followers, where he attempts to convince them with “calumnious Art of counterfeited truth” (770-1), and to tempt them with “Thrones, Dominations, Princedoms, Virtues, Powers” (772), so long as “these magnific titles yet remain not merely titular” (773-4). With the exception of the Anglo-Saxon-derived “Princedoms,” all of the words in this series are descended from Latin, and arrive either through French or through Anglo-Norman.

Figure 2: Paradise Lost Subsections, Proportions of Latinate Words

This section has Satan assume an air of high rhetoric, where he will ask, “Who can in reason then or right assume / Monarchy over such as live by right [God’s] equals, if in power and splendor less, / In freedom equal?” (795-8). This is a similar stance toward monarchy as presented in Milton’s tract, “The Ready and Easy Way to Establish a Free Commonwealth,” also a heavily Latinate text. Roy Flannagan explains that this speech of Satan’s is “characterized by orotundity and repetition,” and that his choice of the Latinate “magnific” at 773, instead of an Anglo-Saxon “great” or “great-making,” “shows how Satan is flattering his audience” (Milton, The Riverside Milton 500).

In contrast, Abdiel’s response to Satan is much less Latinate and thus much more Germanic. He argues that “to [God’s] only Son by right endued / With regal scepter, every soul in heaven / Shall bend the knee, and in that honor due / Confess him rightful king” (815-18). Abdiel’s rhetorical authority comes not from high Latinate eloquence, but from a primal Germanic power: apart from the Latinate “endued,” “regal,” “scepter,” this retort is peppered with the Old English-derived words “son,” “soul,” “heaven,” “bend,” “knee,” “rightful,” and “king.”

The passage with the lowest proportions of Latinate words is Book 3, section ag, which starts at line 429, and continues until 507. The passage that directly precedes this section begins by marking the stark contrast between the scene of heavenly rejoicing for the Son and Satan’s vulture-like circling of the Earth:

Thus they in heaven, above the starry sphere,
Their happy hours in joy and hymning spent.
Meanwhile upon the firm opacous globe
Of this round world, whose first convex divides
The luminous inferiour orbs, enclosed
From Chaos and the inroad of darkness old,
Satan alighted walks (416-22)

The macro-etymological shift from Latinate to Germanic in this section enacts the heaven/Satan contrast along the fault-lines of language history. Directly after this passage, the narrator describes in heavily Germanic speech this liminal space between Chaos and the earth, as “dark, waste, and wild, under the frown of night / Starless” (425-6). Milton then compares Satan, in an epic simile, to “a vulture on Imaus bred, / Whose snowy ridge the roving Tartar bounds” (431-2). (Flannagan explains that “Imaus” etymologically conveys the idea of a snow-capped mountain (Milton, The Riverside Milton 428)). This vulture departs from “a Region scarce of prey” in order to

gorge the flesh of lambs or yeanling kids
On hills were flocks are fed, flies toward the springs
Of Ganges or Hydaspes, Indian streams;
But in his way lights on the barren plains
Of Sericana, where Chineses drive
With sails and wind their cany wagons light:
So on this windy sea of land, the fiend
Walked up and down alone bent on his prey (434-441)

This passage, while very geographical, is also primal, pastoral, and earthy: qualities all associated with the Germanic register. The pastoral image of “yeanling kids” is, of course, ultimately a dark one, where their flesh is feasted upon by this vulture. This makes the pastoral/prey contrast, as in the heaven/Chaos contrast above, all the more stark. Germanic words here, such as “hills,” “flocks,” “flies,” “springs,” “lights,” “plains,” “drive,” “sails,” “wind,” and “sea,” not only serve to contrast with the vulture and thus with Satan, but set the scene, even in this brief aside, for the introduction of the World in Book 4.

The etymological contrast between Satan and Abdiel noted above occurs between many other characters in Paradise Lost, as well. As each character has his or her own voice, motives, and rhetorical style, these features are often reflected in the etymologies of their words. Figure 3 shows the Latinate proportions for each of the speakers in Paradise Lost. The speakers with the highest percentages of Latinate words are the narrator, Adam, and Raphael, while the speakers with the lowest are the minor characters Zophiel, Nisroc, and Zephon¹. With only 84, 131, and 76 words each, however, these last three fallen angels likely do not show high proportions of Latinate words simply because they are not given the chance to—their speech samples are not large enough. Analyzing only the main characters—that is, characters who speak more than two thousand words² —the characters with the lowest proportions of Latinate words are Eve, the Son, and the Father. This seems to suggest that the narrator, Adam, and Raphael all perform higher levels of discourse than God, the Son, and Eve.

To study the voices of the characters in greater detail, I wrote a script to determine each character’s distinctive words. This script works by counting each word a speaker uses, and divides that count by the number of times it is used by other speakers. To avoid difficulties arising from comparing characters with logarithmically different word counts, I again only compared the major characters. The most distinctive word of the narrator is “replied,” which he or she uses almost four times as often as any other character. There aren’t very many Latinate words in this list, although the narrator’s twelfth most distinctive word is “answered.” Since both of these words are functional for telling a story in the past tense, the narrative task might account for some of the narrator’s high proportions of Latinate words.

Before stopwords are removed, the resulting distinctive words are revealing about the characters’ voices. Of Satan’s, John Hale argues that “that voice is many things but always self-preoccupied” (136). Yet a closer look at Satan’s distinctive words shows “you,” “your” and “ye” ranked in his top four, seemingly refuting Hale’s claim. Adam’s distinctive words, on the other hand, are much more self-preoccupied: “us,” “my,” and “we” suggest introversion.

Adam’s most distinctive word, once stopwords were removed, however, is the word “sustain,” descended from Latin and arriving in English via Anglo-Norman. There are the Latinate words “approve,” “absence,” and “favor,” but also the distinctly Anglo-Saxon words “bone,” “dust,” “need,” and “feel.”. By contrast, Eve’s most distinctive words are the Latinate “forbids,” but also the Germanic “early,” “dreamed,” “glistering,” and “help.” The top distinctive words of God and the Son, at ranks #1 and #2, respectively, are the Latinate words “redeem” and “redeemed.” One might hypothesize that these would be words distinctive of fallen angels eager to reenter heaven, so it is surprising that they should correlate instead with the heavenly monarchy.

Figure 3: Proportions of Latinate Words among Speakers

One of the superficial features of Latinate words is that they are frequently composed of root words and affixes. The Latinate prefixes “con-,” “dis-,” “pre-,” “un-,” and “re-” are especially common in Paradise Lost. Figure 4 shows the distribution of these prefixes across the books of the poem. This distribution was generated using a simple Conditional Frequency Distribution from the Python Natural Language Toolkit (NLTK) module, which tests whether a word starts with the prefix. To account for words like “rest,” which in fact begins with “re-” but is not etymologically composed of the “re-” prefix, I created a blacklist containing words like “regent,” “reason,” “prey,” “under,” and “union.”

Figure 4: Prefixes, by Book, Not Normalized for Word Count

“Re-” is by far the most common of these prefixes, accounting for the distinctive word of the narrator, “replied,” as well as God and the Son’s distinctive lemma, “redeem.” Book 10 shows the highest frequency of this prefix, as to be expected for a postlapsarian book of regret and inability to return. In fact, the word “return” occurs sixteen times in Book 10, as in Adam’s lament “dust I am, and shall to dust return” (770) and the Son’s sentence “in the sweat of thy face shalt thou eat bread, / Till thou return unto the ground” (205-6). Milton doubles “re-” words frequently in this book, as well, with “reflux on me redound” (739), “with stern regard he thus repelled,” (866), and “Eve, recovering heart, replied” (966). The book ends with Eve’s “remorse” (1098) and with the couple’s “reverent” (1100) posture of prayer before God.

If “re-” is the prefix most characteristic of Book 10, then “dis-” is that of the preceding Book 9. Introducing the book, the narrator warns that he will speak no more of “rural repast,” (another “re-” word) (4), but must change the tone (and implictly the predominant prefix) of the poem:

… I now must change
those notes to tragic; foul distrust, and breach
Disloyal on the part of man, revolt,
And disobedience: on the part of heaven
Now alienated, distance and distaste (5-9)

About this passage, Christopher Ricks comments that “one’s interest in the phrase is surely aroused by its alliterative crescendo; we rise with pauses through ‘discourse … distrust … disloyal … disobedience’ to the full clash of the cymbals in distance and distaste” (Ricks 69). Apart from their musicality, this passage’s preponderance of “dis-” words emphasize the disobedience that is the narrator’s declared subject of the poem. Other dis- words that appear in this book include the ominiously-voiced “divine displeasure,” (994), “discord” (1124), and “dissent” (1160).

The prefix “un-” operates in a similar way to “dis-,” and is an even more important prefix for the poem, whose original title was “Adam Unpardis’d.” “Un-” is furthermore the site of many of Milton’s neologisms. As Flannagan observes: “Milton is especially fond of making up negative compound words, such as ‘unbesought,’ ‘undelighted,’ ‘disespouse,’ or ‘inabstinence’” (Milton, The Riverside Milton 316). These often occur in chains of three, such as in Belial’s worry that the angels should be further “unrespited, unpitied, unreprieved” (2.184-5). Fowler notes that this is “asyndeton (omission of grammatical connections) … combined with similarity or sameness of prefix,” and locates precedents among the Greek tragedians, Spenser, and Shakespeare (Milton, The Poems of John Milton 516). Thomas Corns calls this a common “stylistic motif” of Milton’s (84–5). These chains happen two more times, with the narrator’s “unprevented, unimplored, unsought” (3.231), and the Son’s “unshaked, unseduced, unterrified”³(5.899).

Figure 5: Prefixes by Speaker, Normalized by Wordcount

Analyzed by speaker, some surprising trends emerge. Figure 5 shows prefix usage by each of the main characters, adjusted for the total number of words each character speaks. The characters that use “re-” the most are not, as one would imagine, Satan, whose ambitions are to “repair” or “return” to Heaven, but God, Michael, and the Son. Similarly, the negative “un-” prefix does not appear the most among Satan or his followers, but among Eve and the Son. “Con-,” of “confuse,” “contempt,” and “converse,” appears most frequently in Adam’s speech, and there is very little variation among the other prefixes.

Conclusions and Further Research

Milton is, as claimed by his detractors and apologists alike, a heavily Latinate poet. The etymological/rhetorical climax of Paradise Lost, as seen in proportions of Latinate words, does not occur where one might expect, around the time of the fall, but with the heavenly war in Book 6. This makes the poem fairly symmetrical in terms of its Germanic and Latinate registers. The characters who exhibit the most heavily Latinate speech are the narrator, Adam, and Raphael, while those who exhibit the least are Eve, the Son, and God. This lends some credence to readings of Paradise Lost that award Satan the prize for the most rhetorically convincing arguments, instead of God and the Son. Finally, this experiment found that “re-” prefixed words are not distinctive of Satan, as one might imagine, but of God and the Son.

Future research might build on the methods developed here for use with other texts, or with further analyses of these texts. Using the tools created for this experiment, it might be possible to create an etymologically annotated edition of Paradise Lost, for instance. This edition would color-code words according to language origin family, in order to show the reader at a glance where clusters of origin language families lie in the text. Furthermore, the tools presented here might be abstracted and used to create a toolchain that could analyze in one step any TEI-encoded XML text, producing etymological data according to its sections and speakers. Since all the data, code, and text used here are open-source and licensed under the GNU Public License, readers of this article are encouraged to contribute to this project, or modify it for their own purposes.

Works Cited

“Abyss, N.” OED Online 2011. Web. 26 Apr. 2016.

Addison, Joseph. “Spectator Papers.” Milton Criticism; Selections from Four Centuries. Ed. James Thorpe. New York: Octagon Books, 1966. Print.

“Babel, N.” OED Online 2011. Web. 25 Apr. 2016.

Corns, Thomas N. Milton’s Language. Cambridge, Mass., USA: B. Blackwell, 1990. Print.

Crawforth, Hannah Jane. Etymology and the Invention of English in Early Modern Literature. New York: Cambridge University Press, 2013. Print.

Eliot, T. S. “A Note on the Verse of John Milton.” Essays and Studies 21 (1936): 32. Web. 25 Apr. 2016.

Hale, John K. Milton’s Languages: The Impact of Multilingualism on Style. Cambridge University Press, 1997. Print.

“Humble, Adj.” OED Online 1899. Web. 27 Apr. 2016.

“Humiliate, V.” OED Online 1899. Web. 27 Apr. 2016.

Johnson, Samuel. “Milton.” Milton Criticism; Selections from Four Centuries. Ed. James Thorpe. New York: Octagon Books, 1966 [c1950]. Print.

Leavis, F.R. “Milton’s Verse.” Scrutiny (1933): 123–136. Print.

Milton, John. A Fuller Institution of the Art of Logic, Arranged After the Method of Peter Ramus. Vol. 11. New York: Columbia University Press, 1935. Print. The Works of John Milton.

—. The Poems of John Milton. Ed. John Carey and Alaistair Fowler. Harlow: Longmans, 1968. Print.

—. The Riverside Milton. Houghton Mifflin, 1998. Print.

Ricks, Christopher. Milton’s Grand Style. Oxford: Clarendon Press, 1963. Print.

Note

The source code for this project, as well as a nice PDF version of this paper can all be found on the project GitHub repository. All of the analyses, as well as many others that didn’t make the final cut for this paper, are available there as Jupyter notebooks. You can run these notebooks on your own computer using Jupyter, and modify them slightly to conduct similar analyses of Paradise Lost or other texts. The command-line program used to generate the macro-etymological results is now available on GitHub and pypi.

A complete list may be found in the analyze-pl notebook.↩︎
These characters are the narrator, Adam, Eve, God, the Son, Satan, Michael, and Raphael.↩︎
These strings are easily detected with the GNU command grep and a regular expression: grep -ir ‘\bun.*\bun.*\bun’ bk/*↩︎

Chapterize: a Tool for Automatically Splitting Electronic Texts into Chapters

2016-08-23

If you do computational analyses of books, and need to break up the book’s text file into its constituent chapters, I’ve just released a tool that you might find useful. It’s called chapterize, and it breaks a book into chapters. This is how it works:

# First, get a copy of Chapterize: 
git clone https://github.com/JonathanReeve/chapterize.git

# Change into that directory: 
cd chapterize

# Now grab a copy of Pride and Prejudice from Project Gutenberg: 
wget http://www.gutenberg.org/cache/epub/1342/pg1342.txt

# Give it a nicer name: 
mv pg1342.txt pride-and-prejudice.txt 

# Run Chapterize on it:  
python chapterize.py pride-and-prejudice.txt

This will create a directory called pride-and-prejudice-chapters, containing chapters 1-61 of Pride and Prejudice, named with leading zeros. Now you can run analyses on each of these chapters. For instance, to compute the macro-etymology of each chapter using my macroetym tool:

# Change into the chapters directory
cd pride-and-prejudice-chapters

# Grab a copy of the macro-etym tool
git clone https://github.com/JonathanReeve/macro-etym

# Run macroetym on each chapter
python macro-etym/macroetym/main.py *.txt

Because chapterize removes metatext (introductions, tables of contents, Project Gutenberg fine print), it can also be used to clean up book data in preparation for text analysis. If you don’t actually need to break up a book into its chapters, but just want to extract its text, use the --nochapters flag:

python chapterize.py --nochapters pride-and-prejudice.txt

This command will create pride-and-prejudice-extracted.txt, containing just the inner text of the novel. So whereas pride-and-prejudice.txt begins with “the Project Gutenberg EBook of Pride and Prejudice, by Jane Austen,” pride-and-prejudice-extracted.txt begins with “it is a truth universally acknowledged.”

Since chapterize is a command-line tool, it’s easily scriptable. Let’s say you have a directory of 100 novels, and you want to remove all their metatext. That’s easily done with a simple shell loop:

for f in *.txt; do python chapterize --nochapters $f; done

If you use chapterize, let me know how it works for you. If you get any errors with a particular book, send me a copy of the book, and I’ll try to make it work.

A Generator of Socratic Dialogues

2016-10-08

In the influential 1948 paper “A Mathematical Theory of Communication,” the mathematician Claude Shannon conducts a thought experiment to construct an algorithmic approximation of language. The algorithm can be described like this:

Choose a book at random from your bookshelf, and open to a random page. Choose a random word on that page. Record this word.
Open another book at random, and read until you find the word you just recorded. Record the word that follows this word. Repeat this step as many times as you like.

This process, when applied to the speech of a particular character or author, can mathematically approximate that character’s speech. While that may be a difficult task to accomplish manually, it’s a trivial computational task using semantically encoded texts. For texts that are encoded in TEI XML, such as the Platonic dialogues available through the Perseus Digital Library, a character’s speech is encoded like this:

<sp><speaker>Socrates</speaker><p>Dear Phaedrus, whither away, and where do you come from?</p></sp>

This markup makes it trivial to exact all of a character’s speech using an XPATH expression like .//sp[speaker’Socrates’]/p=. Once that speech is extracted, this same probabilistic process can then be abstracted and applied to the metasyntactic or textual level, so that it can generate not only strings of words, but textual structures like dialogues. I recently wrote a program that implements this idea. Here’s a dialogue that the program generated, in which it tries to emulate the Platonic dialogue Phaedrus, and also emulate the speech of each character in the dialogue:

Socrates: Do you came down, attain immortality as we must first make the longest ears—No, Phaedrus, then pray to agree and writing speeches.

Phaedrus: Certainly. Yes, certainly. What he must speak then I think he speaks is being guided about, which you mean?

Socrates: Do you seem less sense about rhetoric.

Phaedrus: True. What?

This is nonsense, of course, but you can still hear the voices of the characters come through. Socrates sounds vaguely like Socrates, and Phaedrus sounds vaguely like Phaedrus.

Here’s how it works. Figure 1 shows a graphical model that represents the probabilistic process of the algorithm. The user-supplied value of M denotes the number of lines to be generated (in the above example this is 4). This determines the number of character utterances that will be generated. Each of those utterances, c, are determined by the previous utterancẹ̣—Socrates will speak after Phaedrus, and vice versa. For each character choice, the length of each utterance, l, is probabilistically determined by the text. Each of those lengths is then fed into a Markov chain generator which is given a probability table of that character’s speech. Using that table, words are generated, where the probability of each is determined by the preceding word.

Figure 1

The Socratic dialogue generator should work with any Platonic dialogue in the Perseus library. Just specify the filename, the two characters you want to have talk to each other, and how many lines of dialogue you want to generate.

Probabilistic Detection of Character Voices in Fiction

2017-01-04

In James Joyce’s novel Ulysses, the school headmaster Mr. Deasy quotes Shakespeare in a lecture in financial responsibility to his employee Stephen Dedalus. “[W]hat does Shakespeare say?” he asks, “Put but money in thy purse” (Joyce 1986, 25). As Stephen remembers it, however, this is not merely a saying of Shakespeare’s, but one spoken by Shakespeare’s infamous character Iago. So while Deasy thinks himself to be quoting the wisdom of the early modern playwright, he is, in fact, quoting the Bard’s most notorious arch-villain. This distinction—one between an author and the author’s fictional creations—is, it need not be said, crucial to the understanding of literature. It is that which the following experiment hopes to probabilistically detect.

The problem of computational character attribution is one of literary knowledge production. In concrete terms, it is the difference between the sentence “Madam, I never eat muscatel grapes” and its TEI XML markup, <said who “Edmond Dantès”>Madam, I never eat muscatel grapes</said>=. In the first case, a reader familiar with The Count of Monte Cristo might recognize it as spoken by Edmond Dantès; in the second case, the reader (human or machine) need not know the work to attribute the sentence to its speaker. When an entire novel is marked up in this way, this allows for answers to a wide range of questions, such as the size of the work’s cast of characters, the distribution of character speech, and stylistic properties of individual characters. These are queries that are useful for both close and distant reading—they can provide insight about particular characters and the novel as a whole. They are useful both to the close study of a single work and to the distant study of hundreds or thousands of novels at a time. Although the task of manually marking up a novel may be laborious for a human annotator, this information might be generated semi-automatically from stylistic signatures of the character. The following experiments will attempt to test this hypothesis.

Experimental Design

The design of this series of experiments is based on Box’s Loop, an iterative process for refining a probabilistic model based on its predictive performance (D. M. Blei 2014, 205). The meta-analysis, then, is one of model selection, analysis, criticism, and improvement, while the analysis itself consists of four steps: chunking, vectorizing, dimensionality reduction, and prediction. Chunking involves the choice of documents, and the modification of those documents to fit certain lengths. (Each document only contains text in one character’s voice, but these documents might be of varying length.) Vectorizing is the transformation of those documents into numeric representations, whether through traditional “bag-of-words” term frequency representations or more semantic techniques. Dimensionality reduction transforms those high-dimensional vectors into lower-dimensional ones that are more easily manipulable by the predictive step. Prediction takes the transformed set of vectors and performs probabilistic inference, effectively assigning character voices to each document.

For each of these steps, there are many available techniques and parameters. To identify the best-performing ones, I used a cross-validation grid search that performs a meta-analysis by testing all permutations of these techniques.¹ The grid search tests each configuration against an adjusted Rand score, which compares the labeled data with its predicted clusters. This metric accounts for chance, so that a score close to zero indicates a parameter configuration that performs no better than chance, while a score close to 1 is a perfect clustering, identical with the clustering of the original labels, although not necessarily labeled identically.

To test the efficacy of voice detection, I started with two TEI XML texts, distant from each other in time and genre: Virginia Woolf’s experimental 1931 novel, The Waves and Samuel Richardson’s classic 1748 epistolary novel Clarissa. The Waves is notable in that consists almost entirely of monologues spoken by six characters. Similarly, Clarissa is composed almost entirely of letters, mostly from four of the novel’s approximately 30 characters. These novels were chosen for their large proportions of character voices, and because they were already available in TEI XML format, which made possible the extraction of substantial amounts of labeled text.

The Waves

A manual, preliminary parameter search showed that character attribution worked best with utterances longer than four thousand characters. Furthermore, very long utterances, such as Bernard’s 66,000-character speech that ends the novel, threw off the analysis. With this in mind, I restricted the total 240 utterances of The Waves to just the 19 that were between 2000 and 20000 characters in length. The lengths of these documents conform roughly to a normal distribution, with a mean length of 6487 characters, and a standard deviation of 1960.

From there, I vectorized these documents using a TF-IDF vectorizer, which counts the frequency of each word, and reweights these according to how frequently they are used in the corpus. I set a maximum document frequency of 30% to ignore corpus-specific stop words, and then limited the vocabulary to the top 500 words (these are parameters suggested by the cross-validation search). The resulting vectors I then reduced to five dimensions using principal component analysis. This 19x5 matrix became the input for probabilistic inference. A two-dimensional projection of this matrix is shown in Figure 1A. Of note here is the apparent separation in this projection between the male and female characters, with the male characters in the upper right and the female characters in the lower left.

Figure 1A: The Waves, Labeled

Figure 1B: The Waves, Predicted

The probabilistic model used for clustering assumes that the dimension-reduced, TF-IDF-weighted word frequencies can be modeled with a mixture of Gaussians. I performed inference using Scikit-Learn’s GaussianMixture class, which uses the expectation-maximization (EM) algorithm to cluster the data. Given six components in which to cluster data, the algorithm clustered the data into the six groups shown in Figure 1B. The labels aren’t identical with the original labels in Figure 1A, but the groupings are similar. The inference correctly groups together four out of five of Bernard’s utterances, three out of four of Louis’s, two out of three of Neville’s, Rhoda’s, and Susan’s, but misidentifies Jinny’s. After twenty trials with this configuration, the mean adjusted Rand score was 0.443, with a standard deviation of 0.076—performing much better than chance, although with room for improvement.²

Clarissa

After manually tagging and extracting character-labeled letters from Richardson’s Clarissa, I generated test documents by selecting only letters longer than 8,000 characters and shorter than 50,000 characters. This produced a corpus of 180 letters of varying lengths, the relative lengths of which are indicated by the sizes of the dots in Figure 2. I then vectorized these documents using the top 500 most frequently used words, and reduced the resulting matrix to 25 principal components using PCA, the first two dimensions of which are shown in Figure 2A. As in the Waves experiment, these were all parameters suggested by the cross-validation grid search. Unlike the Waves experiment, however, the grid search suggested a slightly different inference model: a Bayesian Gaussian mixture model. This model differs from the Gaussian mixture model in that it uses variational inference techniques (see D. Blei 2016). Additionally, it doesn’t always find means for all clusters requested—only no more than those requested.

Curiously, the best parameters found by the grid search involved the configuration of the Bayesian Gaussian mixture model with four components, fewer than the number of characters. However, this turned out to have not been an error, since the amount of text represented by the relatively minor characters James and Morden (as evidenced by the paucity of green and red dots in Figure 2B) is very small, and it would almost be fair to assume that there are really only four characters represented here.

The final clustering is shown in Figure 2B. It incorrectly clusters together the correspondence of villains Lovelace and Belford, but forgivably, since these characters are friends and associates, and at most points in the novel partners in crime. It correctly identifies most of Anna’s correspondence, but divides her best friend Clarissa’s into two groups: one closer to Anna, and another closer to Lovelace. A closer analysis, which is perhaps beyond the scope of the present experiment, might reveal that those of Clarissa’s letters closest to Anna’s are letters in fact written to Anna, while her letters that appear closest to Belford and Lovelace might, in fact, be those written to them. In fact, the clustering here, though inaccurate, might be more useful to literary analysis than an accurate clustering: they might reveal not only the character voices themselves, but degrees or modes of these voices. They might show how voice changes according to addressee.

Figure 2A: Clarissa, Labeled

Figure 2B: Clarissa, Final

The adjusted Rand score for this clustering is a slightly lower 0.357, faring much better than chance, but still worse than The Waves. Although these experiments used different parameters and clustering techniques, this discrepancy might be telling.

Semantic Vectorization

The experiments above all rely on word frequency representations of text, and often only on the frequencies of the top 500 words (most likely function words). But what if other properties of the words, such as their meanings, were taken into account? First, I tried transforming documents into 300-dimensional vector representations using the GloVe algorithm in the SpaCy natural language processing Python library.³

The best configuration for this representation, using documents from the top four characters, reducing the dimensions to 5 with PCA, and performing inference with a Gaussian mixture model with four components, showed a mean adjusted Rand score of 0.192, with a standard deviation of 0.023 after twenty trials. Figure 3B shows the results of that experiment. Here, the probabilistic inference manages to separate, at the 0.0 longitudinal line, protagonists from antagonists, and male from female characters, grouping Anna and Clarissa together, and Lovelace and Belford. It does not seem to be able to distinguish between those individual characters, however.

Figure 3A: Clarissa Vectors, Labeled

Figure 3B: Clarissa Vectors, Predicted

I attempted other vectorizations, as well, without much success. A representation of a document as a vector of parts of speech frequencies produced scores roughly equal to those of chance. Another vectorization that represented documents as the frequencies of the root words in each sentence performed equally poorly. Since semantic word vectorizations performed only about half as well as the word frequency vectorizations described above, and similar representations even worse, we must conclude that character voice is most discernible in the frequencies of function words, rather than in the meanings of the words.

Discussion

The difference in the highest possible adjusted rand scores for each novel—0.443 for The Waves, and 0.357 for Clarissa—might be a useful observation, even though these scores were arrived at with very different processes. Perhaps the respective scores indicate the degree to which these novelists are able to write in the styles of their characters. Conversely, this difference might indicate the degree to which these writers chose the stylistic diversity of their characters. If that is the case, novelists with many classes of broadly-painted characters such as Charles Dickens might show higher scores than novelists like Jane Austen, who deal with social subtleties.

Although the technique outlined in this paper might not be appropriate for fully unsupervised character voice attribution, semi-automatic attribution might be possible with some manual tagging of groups. In any case, attributions of very small utterances (with fewer than 2,000 characters) may not be possible with this word frequency representation.

If these techniques do not prove to be very useful in automating character voice attributions, however, they might be useful to literary studies in other ways. By examining the confusion caused by certain probabilistic clusterings, for instance, we might be able to find groups of characters—male and female characters, for instance, or protagonists and antagonists. By using an unsupervised model such as the Bayesian Gaussian model used with Clarissa, we might also be able to infer, with some small degree of confidence, the numbers of main characters. In some cases, groupings among documents or utterances might reveal hidden affinities among characters, as well, or stylistic changes in a character’s voice correlated with his or her addressee(s).

Note

This paper is also available as a PDF at the project’s GitHub repository.

References

Blei, David. 2016. “Variational Inference: A Review for Statisticians,” November.

Blei, David M. 2014. “Build, Compute, Critique, Repeat: Data Analysis with Latent Variable Models.” Annual Review of Statistics and Its Application 1: 203–32. http://www.annualreviews.org/doi/abs/10.1146/annurev-statistics-022513-115657.

Joyce, James. 1986. Ulysses. Edited by Hans Walter Gabler. 1st edition. New York: Vintage.

Pennington, Jeffrey, Richard Socher, and Christopher D. Manning. 2014. “GloVe: Global Vectors for Word Representation.” In Empirical Methods in Natural Language Processing (EMNLP), 1532–43. http://www.aclweb.org/anthology/D14-1162.

Turian, Joseph, Lev Ratinov, and Yoshua Bengio. 2010. “Word Representations: A Simple and General Method for Semi-Supervised Learning.” In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, 384–94. Association for Computational Linguistics. http://dl.acm.org/citation.cfm?id=1858721.

A full list of techniques and parameters tested may be found in the project repository, in the grid search notebooks for The Waves and for Clarissa.↩︎
The code used for this analysis, written in the Python programming language and using the Scikit-Learn machine learning library, is available at this project’s GitHub repository.↩︎
See Pennington, Socher, and Manning (2014) for more on GloVe, and Turian, Ratinov, and Bengio (2010) for more on word vector embeddings in general.↩︎

The Henry James Sentence: New Quantitative Approaches

2017-06-07

The house had a name and a history; the old gentleman taking his tea would have been delighted to tell you these things: how it had been built under Edward the Sixth, had offered a night’s hospitality to the great Elizabeth (whose august person had extended itself upon a huge, magnificent and terribly angular bed which still formed the principal honour of the sleeping apartments), had been a good deal bruised and defaced in Cromwell’s wars, and then, under the Restoration, repaired and much enlarged; and how, finally, after having been remodelled and disfigured in the eighteenth century, it had passed into the careful keeping of a shrewd American banker, who had bought it originally because (owing to circumstances too complicated to set forth) it was offered at a great bargain: bought it with much grumbling at its ugliness, its antiquity, its incommodity, and who now, at the end of twenty years, had become conscious of a real aesthetic passion for it, so that he knew all its points and would tell you just where to stand to see them in combination and just the hour when the shadows of its various protuberances—which fell so softly upon the warm, weary brickwork—were of the right measure. (James 2003, 60)

This single sentence is the longest of Henry James’s novels. Like the house it describes, it is copious, labyrinthine: an architectural wonder. At first glance, it is ugly, cumbersome, and even confusing, but upon rereading, one finds a “real aesthetic passion” for it and its “various protuberances.” The sentence begins by traversing several centuries in a just a few words, continues by covering a twenty-year personal history, and finishes with a cadenza that elongates time just enough to enable us to savor the certain slant of light illuminating the dear old house. James’s style, alive and beating in this sentence, has been the central subject of several complete volumes and countless articles. His sentences, in particular, have been much discussed. Their compounded clauses, digressions, and qualifications allow for temporal compressions and expansions. Their balance—or imbalance—is what has made them the subject of so much critical controversy. This study presents new methods for quantifying these properties of James’s sentences, methods which might also help to illustrate their appeal.

Dependency parsing is a method of computational linguistics and natural language processing that algorithmically infers syntactic dependencies between words in a sentence. Adjectives that describe a noun, for instance, are graphed as the noun’s dependents. Grammatical subjects and objects, similarly, are dependents of the main verb of a sentence. Subordinate clauses are dependents on the main verb, as well, and have their clausal subjects and objects as their own dependents. This study uses SpaCy, a new library for natural language processing written in the Cython programming language, and one of the fastest and most accurate available dependency parsers, to parse James’s sentences (Honnibal, Johnson, and others 2015). This study also uses a Python module I wrote called Sent2Tree, which parses SpaCy’s dependency graphs into standard tree structures, mathematical objects that can then be manipulated using tools like the ETE3 Toolkit, a library originally created for manipulation of phylogenetic trees (Huerta-Cepas, Serra, and Bork 2016). These tools, as we will discover, will allow us to understand the Jamesian sentence in new ways.

Figure 1: Visualization of the dependency-parsed tree of James’s longest sentence

Figure 1 shows a visualization of a sentence tree created from parsing James’s longest sentence above with SpaCy and Sent2Tree. The SpaCy parser has inferred that “passed” is the fulcral verb in the sentence, a central verb, in a literal, if not figurative sense. The parser correctly identifies most of the clausal structures: the interlude about Edward the Sixth is identified as its own branch, and so are those about Elizabeth and Cromwell. The house’s Restoration and eighteenth-century histories are also their own branches, but top-level instead of dependent on “had.” The early history of the “American banker,” not yet revealed to be Mr. Touchett, is on a further branch, and his present history, with its digression into the house’s details and shadows, is the final branch. This segmentation might not be perfectly intuitive—an intuitive division might split the independent clauses on either side of the semicolon—but it nonetheless reveals many of the movements of the sentence: the periods of its histories, and the digression into more lingering, aesthetic language that constitutes the sentence’s finale. This figure captures both the relative balance of the sentence—its tidy list-like history—as well as its ultimate imbalance—its descent into the realm of the slow, minute, and sensory world of shadows and warm bricks.

Sentences in James’s Novels

While the sentence above is James’s longest sentence, and therefore exceptional, it is not anomalous. Sentences of similar complexity appear throughout James’s career, and not only in the later novels, when he is most known for this style. To test exactly where his longer sentences appear, I manually assembled a corpus of twenty novels: all of his novels except for The Other House and the posthumously published A Sense of the Past, both difficult to obtain in electronic form. I assembled the text mostly from editions found at Project Gutenberg and file:henryjames.co.uk, and they represent a mix of American and British editions. The length, in word counts, of each sentence I then quantified, and aggregated into a histogram of ten equal-sized groups.

These groupings, plotted chronologically according to novel, are shown in Figure 2. Bars with an index of zero represent the shortest decile of sentences in that novel, and bars with an index of nine represent the longest decile. Broadly speaking, the strongest trend is the one that divides the so-called Jamesian early style from the late, which here seems to happen during the five-year gap in James’s novel publication streak between 1881 and 1886, during which he travels extensively to America and continental Europe (Haralson and Johnson 2009, 8–9). Whether his travels, his wife Alice’s cancer diagnosis, or other personal factors are responsible for this is beyond the scope of this study, but the fact remains that James’s style after 1886, at least expressed in sentence length, seems to have changed. The only exception is The Awkward Age, which might be explained by noting its highly dramatic structure (James later adapted it into a play), and thus its high incidence of dialogue.

Figure 3 shows the same series, but only the sums of deciles 5-9, corresponding roughly to very long sentences. In this view, which as the scale shows, only represents two percent of the whole, very long sentences like the one quoted above seem most likely to appear in the poshumously published The Ivory Tower, followed by Princess Casamassima, The Golden Bowl, and The Bostonians. Finally, Figure 4 shows a box plot of the same series, more clearly showing the properties of each distribution of sentence lengths. Here, the early/late style divide is most apparent in the division between the box sizes, representing the interquartile ranges of each distribution. Again, apart from the posthumous novel, the most notable outliers are the 1886 pair of Princess Casamassima and The Bostonians. The sentence below is from The Bostonians:

Figure 2: Distributions of Sentence Lengths in James’s Novels

Figure 3: Proportions of Longer Sentences in James’s Novels

Figure 4: Box Plot of Sentence Length Distributions in James’s Novels

This edifice, a diminished copy of the chapel of King’s College, at the greater Cambridge, is a rich and impressive institution; and as he stood there, in the bright, heated stillness, which seemed suffused with the odour of old print and old bindings, and looked up into the high, light vaults that hung over quiet book-laden galleries, alcoves and tables, and glazed cases where rarer treasures gleamed more vaguely, over busts of benefactors and portraits of worthies, bowed heads of working students and the gentle creak of passing messengers–as he took possession, in a comprehensive glance, of the wealth and wisdom of the place, he felt more than ever the soreness of an opportunity missed; but he abstained from expressing it (it was too deep for that), and in a moment Verena had introduced him to a young lady, a friend of hers, who, as she explained, was working on the catalogue, and whom she had asked for on entering the library, at a desk where another young lady was occupied. (James 2006a)

This 217 word sentence, James’s second longest, shares much with the longest sentence. Like the longest, its principal subject is a building. Like many of James’s characters, and indeed James himself, the building is somehow both American and European: the Harvard library, but also a Cambridge chapel. Furthermore, like the previously quoted sentence, it is a topography of a daydream. Though not a journey through time, it is a journey through space, an admiring pan through the library that nonetheless causes him deep “soreness.” This sentence does more than merely describe the library, however, for it moves straight in to the next action: “in a moment Verena had introduced him to a young lady.” The fluidity of this transition is underlined by the immediacy signaled by “in a moment,” which indicates that a sharp temporal shift has taken place. Time, that had been allowed to flow aimlessly and viscously across the objects of the library, now, “in a moment,” snaps back into place, and we again reach the staccato rhythm of action: “a friend of hers / who / as she explained.”

Figure 5: Visualization of the dependency-parsed tree of James’s second longest sentence

Figure 5 shows a visualization of the parsed sentence quoted above. The parser divides the sentence into seven branches. The first is the subject of the sentence; the second, a comparison with the Cambridge capel; the third is the reverie that takes Basil along the objects in the library. The first length of this branch shows the chain-like anaphoric structure in the string of objects connected with “and” and qualified with “that.” This shoot then blossoms into a new structure on the verb “felt,” which introduces Basil’s subjectivity. The branches below parenthetically qualify that subjectivity (it was too deep for expression), and bring the reverie to a close with the introduction of the young librarian. Overall, this structure is one of a digression: a movement into the sensory world of the aesthetic, followed by a movement back into the world of people and action.

Characteristics of the Long Jamesian Sentence

The two sentences we have seen so far, from different novels, exhibit many of the same characteristics. They both expand and contract the reader’s experience of time over the course of the sentence. Their syntactic structures have young, thin shoots, as well as broad, leafy, tree-like formations. But are these sentences characteristic of James’s long sentences more generally? To test this, I first divided all of the sentences from this corpus of James novels into two groups: those with fewer than thirty-four words, and those with thirty-four or more words. This number was calculated to produce a relatively balanced corpus containing roughly 1.6 million words from both categories. I then removed punctuation from the corpus and lemmatized the words, transforming plurals to singulars, and conjugated verb forms to their bare infinitives. These wordlists I then adjusted for the frequency in their categories, and compared with one another, to produce lists of words that are distinctive of each. I repeated this process, using groups of sentences with and more than 100 words, to generate lists of very long sentences, as well. To ensure comparisons of equal word counts, I randomly sampled from the list of shorter words until reaching the word count of the larger, repeating the process three times, and taking the mean of the results.

The most distinctive lemmas of James’s long sentences include many that seem appropriate to aestheically sensible, objective descriptions. There are architectural words, such as place and room; the lemmas distinctive of very long sentences add window and light. These lists are full of adjectives describing size or magnitude: great, small, high, low, and little. Markers of time like hour, second, evening, and occasion are also here. The sensory words sense and feel appear in these lists, alongside the more legal lemmas particular, effect, and fact. There are also indicators of a curious, interrogative mood in question and interest.

In contrast, the lemmas distinctive of short sentences seem appropriate to action: say, speak, tell, ask, look, and come. There are also the cognitive states of think and know, the anticipatory want and shall, and finally the almighty verb be. While the lemma Miss is distinctive of long sentences, the titles Mr, Mrs and Madame all appear in the list of words distinctive of short sentences. This suggests that James pays more lingering, wandering attention to his unmarried female characters, while his married female characters are more pragmatic, and prone to action. Similarly, the character names Horton and Hyacinth, from The Ivory Tower and Princess Casamassima, respectively, appear characteristic of long sentences, while Rowland, Nick, and Isabel of Roderick Hudson, The Tragic Muse, and The Portrait of a Lady are distinctive of short. This suggests that James allows characters like Hyacinth more meandering narrativistic interiority than characters like Isabel.

I conducted a few more experiments to determine the properties of James’s long sentences. The first dealt with the position of long sentences within their novels. My hypothesis was that long sentences seem likelier to to appear earlier than short sentences. This is only weakly true. The mean start location for a long sentence is at 51% of the narrative time of the novel, while the mean location for a short sentence is at 49%. This trend is somewhat magnified using very long sentences: they appear at 45%.

The second experiment concerned the logarithmic probabilities of the words in each category. This measurement, a built-in feature of the SpaCy library, calculates the probability, in log scale, of words occurring in a three billion-word corpus of modern English. If the probability is low, then the sentences contain improbable words, such as “incommodity” of this paper’s epigraph. My hypothesis was that longer sentences would contain more improbable words. This also proves to be true. The mean log probability of long sentences are about .16 lower than that of short sentences, and this difference is about .3 for very long sentences, about 30% of an order of magnitude.

Sentence Balance

To speak of a sentence’s “balance,” in terms of these tree structures, is to speak of a parsed sentence like a mobile, something that might hang over a baby’s crib, or be found in a Joan Miró painting. If Figure 1 were made of wire and equal weights, and if one were to pull it up by the verb “passed,” how would it hang? To measure this, I began by quantifying the total number of descendants of each first-level branch of the tree. In Figure 1, these branches are had, under, finally, remodelled, and so on. Then I took the standard deviation of all the elements in the resulting vector. This gives a measure of the sentence’s balance or imbalance: the relative differences between the total sizes of all of the sentence’s branches, nicknamed the “digression index.” The highest digression index (branch depth standard deviation) is 98, and it belongs to this sentence from The Tragic Muse:

The enemy was no particular person and no particular body of persons: not his mother; not Mr. Carteret, who, as he heard from the doctor at Beauclere, lingered on, sinking and sinking till his vitality appeared to have the vertical depth of a gold-mine; not his pacified constituents, who had found a healthy diversion in returning another Liberal wholly without Mrs. Dallow’s aid (she had not participated even to the extent of a responsive telegram in the election); not his late colleagues in the House, nor the biting satirists of the newspapers, nor the brilliant women he took down at dinner-parties–there was only one sense in which he ever took them down; not in short his friends, his foes, his private thoughts, the periodical phantom of his shocked father: the enemy was simply the general awkwardness of his situation. (James 2006b)

Figure 6: Visualization of the dependency-parsed tree of the most digressive Jamesian sentence

As shown in Figure 6, the syntactic structure of this sentence is extremely unbalanced, at least according to the SpaCy parser. It balances the subject “the enemy,” along the fulcrum “was” with a 139-word object. SpaCy parses some of these dependencies incorrectly, of course: “constituents” is dependent of “sinking” here, which is actually a wholly separate clause. Nonetheless, SpaCy captures the spirit of this sentence, which is a via negativa seeking to explain the nature of Nick’s abstract “enemy” in terms of what it is not.

At each of these negative comparisons, James lingers, exploring each to the fullest, seemingly without regard for the root intention of the sentence. At each, the reader is left wondering whether any one of these potential enemies might be a legitimate threat, after all. As in the sentence from The Bostonians, this is a list of what might have or what could have been. Just as Basil looks at the objects in the library with the “soreness” of “an opportunity missed,” Nick’s would-be enemies are aggregated here as ghostly threats: not true enemies, we are explictly told, but anxieties nonetheless. As with both of the previously examined sentences, there is also a time compression that happens at the end of this reverie, ushering us quickly back into the rapid pace of action: the indulgent digressions on Mr. Carteret quickly give way to a list of short expressions, signaled by the rhetorical “in short”: “his friends / his foes / his private thoughts.” The anaphoric repetition of “his” intensifies the rhythm of the sentence, and gives it an almost poetic meter. All of this functions as a crescendo which serves to contrast with, and highlight, the short, punchy clause that follows: “the enemy was simply:”—we pause on the colon, awaiting the definition—“the general awkwardness of his situation.”

Breadth-First Quantifications of Sentence Trees

Figure 7: A visualization of a dependency parsing of an example sentence.

In addition to sentence balance, which measures the depth of each first-level branch by counting their total descendants, we might also employ a breadth-first approach to the numerical representation of sentence trees, one that quantifies the number of branches at each level of depth. Figure 7 shows an example sentence, “the quick brown fox jumped over the lazy dogs,” parsed with SpaCy. A breadth-first quantification of this sentence would return the vector [1, 2, 4, 2], since there is one word at the root (“jumped”), two at the first level (“fox” and “over”), four at the second, and two at the last. When we average these vectors for every sentence in a novel, we can represent the average sentence structure of a novel.

Figure 8 shows these average vectors for the corpus of James novels. The X axis represents the level of the sentence tree, and the Y axis represents the number of nodes at that level. It appears that the sentence structures in James cluster chronologically. Overall, the general chronological trend is one that tends toward greater syntactic complexity. The most cohesive group is the purple and dark blue cluster of lines that represent James’s early work, 1871-1881. Next, there is a much wider band (suggesting greater variety in sentence structure) in orange and green that corresponds to James’s middle and late years. Next, there is the pair The Bostonians and Princess Casamassima, both written in 1886, that stand on their own in light blue. Finally, the unfinished novel The Ivory Tower is in its own category altogether.

Figure 8: Averaged breadth-first sentence vectors for James novels.

Conclusions

James’s long sentences, the artistic merit of which have been hotly debated among critics, are masterful monsters. Their structures allow for wandering reveries and digressions to be set free and then reigned in; their energies are unbridled yet ultimately, neatly framed. Their greatly varying rhythms take us effortlessly from the lingering eye of an artist, to the voice of a pragmatic narrator, while simultaneously expanding and contracting the readerly experience of time. By dependency parsing these sentences, converting these objects into tree-structures, and quantifying their mathematical properties, we achieve a more accurate numerical representation of James’s style than has be previously been possible. These techniques—measures of sentence balance or digression, and breadth-first quantifications—might be added to the suite of tools that comprise authorship detection, or forensic text analysis more broadly, since authorial style, at least as evidenced in James’s bibliography, seems here to shine through.

Works Cited

Haralson, Eric L., and Kendall Johnson. 2009. Critical Companion to Henry James: A Literary Reference to His Life and Work. Infobase Publishing.

Honnibal, Matthew, Mark Johnson, and others. 2015. “An Improved Non-Monotonic Transition System for Dependency Parsing.” In EMNLP, 1373–8. https://www.aclweb.org/anthology/D/D15/D15-1162.pdf.

Huerta-Cepas, Jaime, François Serra, and Peer Bork. 2016. “ETE 3: Reconstruction, Analysis, and Visualization of Phylogenomic Data.” Molecular Biology and Evolution 33 (6): 1635–8. doi:10.1093/molbev/msw046.

James, Henry. 2003. The Portrait of a Lady. New York: Penguin.

———. 2006a. The Bostonians, Vol. II (of II). http://www.gutenberg.org/ebooks/19718.

———. 2006b. The Tragic Muse. http://www.gutenberg.org/ebooks/20085.

A Macro-Etymological Analysis of The Canterbury Tales

2017-06-06

Chaucer’s Canterbury Tales exhibits one of the richest vocabularies of Middle English literature, a vocabulary that reveals influences from a number of native and foreign languages: Old English, French, Latin, Greek, and Hebrew, among others. While some of this foreign influence may be attributed simply to the history of the English language—the Roman Empire, the Norman conquest of 1066, and several waves of Scandinavian immigrants are each responsible for the addition of new linguistic layers—some of it is attributable to Chaucer’s own knowledge of languages: as a well-traveled member of a quasi-courtier class, he had a good knowledge of European languages. Regardless of whether the poet’s word choices were consciously foreign at the time of writing in the fourteenth century, however, there are etymological resonances in each word that evoke levels of formality, social class, and genre. A commonly-discussed group of words that illustrates this phenomenon is the triplet kingly, royal, and regal. The first is of Old English origin; the second, of French, and the third, of Latin. Each possesses a set of associations with their etymological registers that make these words imperfect synonyms. The following experiment will attempt to computationally detect these registers, by quantifying the etymologies of all the words of the Canterbury Tales.

Pre-Digital Macro-Etymology

Macro-Etymological textual analysis is nothing new. Even before what is now known as the digital humanities, and before its precursor, humanities computing, philologists have painstakingly quantified Chaucer’s words and their etymological origins. One of the most recent of these studies is Joseph Mersand’s 1939 Chaucer’s Romance Vocabulary. Using, presumably, nothing more technically complicated than sweat and graph paper, he concludes that Chaucer’s working vocabulary is 8,430 words, of which 51.8% are of “Romance sources,” and 49% of Germanic sources (1939, 43). Many, if not most, critics writing about Chaucer’s vocabulary after Mersand have opinions about this method. Christopher Cannon, for instance, argues that while Mersand is not factually incorrect, he would be incorrect to draw conclusions about Chaucer’s personal “borrowings” from this data alone (2003, 238). Simon Horobin contends that Mersand’s analysis is invalid, in its reliance on the Oxford English Dictionary instead of the Middle English Dictionary (2007, 79), and J.D. Burnley dismisses Mersand’s findings as “of little use in assessing Chaucerian style, and indeed simply ignores the crucial factor of the contemporary perception of the status of the words” (1983, 135). These dismissals, although annoyingly grouchy in tone, all make fair points, which I hope to address in my algorithmic design below.

Mersand is hardly an innovator in Chaucerian macro-etymological analysis, however, as he himself readily admits. In fact, a full chapter of his book is devoted to early quantitative analyses of Chaucer’s vocabulary and its etymologies. Among these are a study by George Marsh in 1859, Alexander Ellis in 1869, and John Wisse in 1878. Mersand even expresses surprise that no “no definite numerical investigation was made before 1850,” despite allusions to Chaucer’s etymologies as early as 1400 (1939, 21). Each of these previous analyses, however, have methodological problems that Marsand hopes to correct, as I hope to use computational methods to correct and expand upon his.

Methods

The Text

Since this analysis relies on tools of natural language processing that are best suited to modern English, I used an edition of The Canterbury Tales that regularizes and modernizes Chaucer’s spelling: Project Gutenberg’s e-text of D. Laing Purves’s 1870s edition, made “for popular perusal” (Chaucer 2000). It is neither a complete translation, which would change the etymologies of many words, nor even a complete spelling modernization. As Purves describes it, “where the old spelling or form seemed essential to metre, to rhyme, or meaning, no change has been attempted. But, wherever its preservation was not essential, the spelling of the monkish transcribers—for the most ardent purist must now despair of getting at the spelling of Chaucer himself—has been discarded for that of the reader’s own day.” I manually divided this text into prologues, tales, and epilogues, and then parsed the lines programmatically, removing glosses, footnotes, and other artifacts.

The Algorithm

The tool used for this analysis, macro-etym, is a Python script I had written for an earlier analysis, but one which I expanded and customized greatly for this project. It uses an opinionated algorithm for determining the etymology of a word, and one that deserves a brief description. It begins by tokenizing a text using Penn Treebank conventions, then removes stopwords (common functional words like “a” and “the” that don’t contribute much to the analysis), infers their parts of speech, and lemmatizes the tagged results, regularizing plurals to their singular forms, and verbs to their bare infinitives. Then, it searches for the word in the Etymological Wordnet, a database created from parsing Wiktionary etymological data (Melo 2014).

If the word is found, but its ancestors are determined to belong to the same language as the original (for instance, modern English words composed by conjoining two other English words), or if the ancestor belongs to a “middle” language variant, like Middle English, macro-etym searches for another ancestor. For all other words, the first ancestor is used. This logic is intended to foreground meaningful etymological resonances. This means that the few Old English words that are actually of Latin derivation, are labeled as Old English, but this might be considered a feature, since these words tend to be indistinguishable from Germanic words to the modern ear. In addition to ignoring current languages, macro-etym also ignores affixes by default. Rather than parse a word like “automobile” as, say, 40% Greek (-auto) and 60% Latin (-mobile), it just considers these prefixes, like stopwords, to be background functions of the language, and incidental to the etymological resonance of the word, which in this case would be labeled as Latin.¹

If a given word is not found in the Etymological Wordnet, macro-etym will attempt a custom lemmatization of the word according to morphological patterns of Middle English. If the lemma is still not found, it will search for the word in a secondary etymological wordnet of 19,135 words of Middle English, which I created from parsing all the word forms listed in the Project Gutenberg edition of A.L. Mayhew and Walter Skeat’s 1888 A Concise Dictionary of Middle English. If not found there, it will finally search for the word in an experimental tertiary wordnet of 38,074 words, awkwardly assembled by parsing the irregular etymological strings from a plain-text edition of the Oxford English Dictionary.²

Once a word is found in one of the three wordnets, it is then categorized according to language family, the biggest categories being Germanic, containing words of Old English, German, Dutch, and Scandinavian origin; Latinate, containing words of Latin, French, Italian, and Spanish origin; Hellenic, containing mostly words of Ancient Greek origin; and Semitic, containing words of Hebrew origin.

Results

Prologues and Tales

Figure 1: Language Families, by All Prologues and Tales

Figure 1 shows the proportions of Latinate, Germanic, Hellenic, and Semitic language families, organized by tale. Immediately noticeable here is that the scales are quite different: proportions of Germanic words fluctuate between 50 and 75 percent (these numbers are higher without stopwords removed); proportions of Latinate words fall between 20 and 50 percent, and Hellenic and Semitic words all represent less than three percent of the total. Since some of these texts are very short—Chaucer’s final “retraction,” for instance, is only 369 words—we should treat the final two language families with some degree of suspicion, since in those cases, the fluctuations here reflect only the difference of about eleven words. It is for this reason that I will be focusing primarily on Latinate words here.

Figure 2: Latinate Words, All Prologues and Tales

Figure 2 shows the proportions of Latinate words per tale, subdivided into individual languages. Broadly speaking, the prologues and tales with the highest proportions of Latinate words are the prose tales: the Chaucer character’s Tale of Melibee at 51.6%, and the Parson’s Tale, at 50.33%. Chaucer’s final retraction, also in prose, is at 41.0%, and the next highest proportions are in Franklin’s prologue, at 40.4%, the Nun’s Priest’s Tale at 38.7%, and the Clerk’s, at 38.5%. Yet this macro-etymological analysis is not just detecting a prose signal, as opposed to one of metered poetry, as will later become more apparent. There might be a few reasons why Latinate words appear more often in prose. First, the metrical requirements of Chaucer’s line might not as easily allow many contiguous multisyllabic words, as are typical of many Latinate words; they are allowed in prose. Next, the prose modes of The Canterbury Tales are all dramatic departures from their environs, dramatic in both in intensity and theatricality, so it is fitting that their etymological modes are also contrasting.

Conversely, the tales with the lowest proportions of Latinate words are all prologues, and with the exception of the Summoner’s, all from Fragment I: the Reeve’s at 19.9%, the Cook’s at 21.7%, the Summoner’s at 21.9%, and the Miller’s, at 22.3%. Unlike other prologues in the Canterbury Tales, these feature a high incidence of dialogue, which might be expected to have fewer Latinate words than the more constructed modes of many of the tales.

One important facet of this Latinate proportions analysis is its jagged, sawtooth shape across contiguous tales in a given fragment. The first of these patterns appears between the Knight’s tale and the Miller’s prologue: there is a sharp drop in proportions of Latinate words, indicating a sharp difference in tonality. The inflated tone of the Knight’s tale, which ends with a happy wedding, or rather, the Latin matrimoigne and French mariage (I.3095), is quickly deflated by the Miller, who “for dronken was al pale,” and who swears, using words that would be at home in (a Middle English translation of) Beowulf, “By armes, and by blood and bones, / I kan a noble tale for the nones, / With which I wol now quite the Knyghtes tale” (I.3125-7). The Miller’s tale itself, as most tales do, shows a higher proportion of Latinate words than its prologue, but this elevation is temporary, for the quiting dynamic that the Miller inaugurates is continued with even greater force by the Reeve, whose prologue shows the lowest proportion of Latinate words in the fragment. The etymological trend within these fragments is one of Latinate oratorial floridity answered by raw, punchy Germanic talk.

Another notable quiting exchange, and one which is also observable along this etymological axis, is that between the Friar and the Summoner. There, the quiting dynamic reaches its apotheosis: the Friar’s tale explicitly concerns a corrupt Summoner, who is literally a devil in disguise. And lest we believe, after this, that the satirical depictions of fellow pilgrims couldn’t get any more profane, the Summoner’s Prologue answers this with a scene where “out of the develes ers ther gonne dryve / Twenty thousand freres” (III.1694-5). This contrasts greatly with the Latinate moral lesson with which the Summoner ends his tale, and this contrast is clearly shown in the macro-etymological analysis.

A third pair, and one that exhibits one of the most dramatic shifts in etymological tone, is that between the Clerk’s tale and the Merchant’s prologue. The Clerk, who has been forewarned by the Host to “Speketh so plain at this time, … / That we may understonde what ye seye” (IV.19-20), nonetheless embarks on a verbose tale, full of verbal flourishes. The tale is set in Italy, and represents a relatively faithful translation from the Petrarchan story De obedienta ac fide uxoria mythologie, a fact which may help to explain the high incidence of Latinate words (Chaucer 2008, 880). It ends with what is announced, in French, to be “Lenvoy de Chaucer,” and a flurry of words of French origin: reverence, eloquence, aventaille, and apparaille, to choose a few. This high French mode is brought crashing down into the Anglo-Saxon world of bones and ale with the “murye” words of the Host, who exclaims, “By Goddes bones, Me were levere than a barel ale / My wyf at hoom had herd this legende ones!” (IV.1213-5). The Merchant echoes the Host’s sentiments of marital woe in his short Prologue that follows, one which is light on invocation, but thick with casual Germanic dialogue.

When each tale is broken into eight equal segments, and each segment is etymologically analyzed, the multi-line segment with the lowest proportion of Latinate words is from the Reeve’s Prologue. Here, the Reeve tells the Miller that while he may appear old, he is not weak. It shows the quiting theme hard at work, and is thus highly Anglo-Saxon:

But if I fare as dooth an open-ers—
That ilke fruyt is ever lenger the wers,
Til it be roten in mullok or in stree.
We olde men, I drede, so fare we:
Til we be roten, kan we nat be rype;
We hoppen alway whil that the world wol pype.
For in oure wyl ther stiketh ever a nayl,
To have a hoor heed and a grene tayl,
As hath a leek; … (I.3871-3879)

The Reeve, in comparing himself to the medlar, uses a colorful colloquial term for the fruit, “open-erse.” This is a term of decidedly Germanic origin—it is arse, and not the French-derived derrière or the Latin-derived posterior. The rest of the passage is literally a pot pourri of Germanic words, mostly from Old English. The rhetorical effect of this string of staccato Germanic is one that answers the vulgar French with which the Miller ends his tale—“Thurgh fantasie that of his vanytee”; “par compaignye” (I.3835, -9)—with a set of food analogies that is, like the rotten fruit itself, down to earth.

In contrast, the multi-line segment with the highest proportion of Latinate words is from Chaucer’s retraction:

Wherefore I biseke yow mekely, for the mercy of God, that ye preye for me that Crist have mercy on me and foryeve me my giltes; / and namely of my translacions and enditynges of worldly vanitees, the whiche I revoke in my retracciouns: (X.1083-4)

However much this prose passage may be an ironic advertisement for Chaucer’s other works, a list of which will soon follow, it nonetheless evokes a mood of sacerdotal sincerity and rhetorical flourish, achieved in part by its Latinate vocabulary. There is a legal resonance in enditynges, revoke and retracciouns, for instance, which all enter English from Latin, by way of Old French.

When each tale is analyzed according to stanza, however, a different picture emerges. According to this analysis, the multi-line stanza with the lowest Latinate proportion is from the Man of Law’s tale, a description of Constance’s journey:

Forth gooth hir ship thrughout the narwe mouth
Of Jubaltare and Septe, dryvynge ay
Sometyme west, and sometyme north and south,
And sometyme est, ful many a wery day,
Til Christes mooder—blessed be she ay!—
Hath shapen, thurgh hir endelees goodnesse,
To amek an ende of al hir hevynesse. (II.946-52)

Here, the Germanic words ship, mouth, driving, north, south, east, and west are situated in the anaphoraic sequence “sometime…sometime” that evokes an epic mood, aggrandizing Constance’s journey, and performing its “many a wery day” in what one might call a wearysome way. The monosyllabic nature of many of these words allows for the Man of Law’s hyponotic meter to further enhance this effect. The Germanic words blessed, endelees, goodnesse, and heavynesse have a simple quality appropriate to an innocent and much-maligned saint character as Constance, one that constrasts sharply with the description of the Roman Senator’s analogous journey that follows, where the Senator very Latinously “repaireth with victorie / To Romeward, … saillynge ful roially” (967-8).

Another such royal description appears in the multi-line stanza with the highest proportion of Latinate words, the ekphrastic description of Nebuchadnezzar, from the Monk’s tale:

The myghty trone, the precious tresor,
The glorious ceptre, and roial magestee
That hadde the kyng Nebugodonosor
With tonge unnethe may discryved bee.
The twyes wan Jerusalem the citee;
The vessel of the temple he with hym ladde.
At Babiloigne was his sovereyn see,
In which his glorie and his delit he hadde. (VII.2143-50)

As with Constance and the Roman senator, there are contrasting Germanic and Latinate modes that coincide with innocence and experience, Christianity and classicism. Compare the passage above with the Monk’s Germanic description of prelapsarian Adam as “With Goddes owene fynger wroght ws he, And nat bigeten of mannes sperme unclene” (VII 2008-9). As mentioned earlier, there is a certain gravitas to the French word royal that is not as pronounced as in an equivalent word of Old English descent, such as kingly, just as the word gravitas itself sounds more serious than Constance’s hevynesse. Chaucer doesn’t use these modes unironically, however, for the Monk’s historical histrionics is soon negated almost bathotically by the Knight, who, though guilty of excessive verbosity himself, pleads to have “namoore of this,” the Monk’s cast of characters (VIII.3957).

Macro-Etymology of the Individual Prologue/Tale

Figure 3: Prologues and Tales

So far we have discussed the macro-etymologies of tales and prologues, but what might we discover about the macro-etymologies of the various parts of individual tales? To answer to this question, I divided each prologue and tale into eight equally-sized parts, and ran macro-etym across each of them. Somewhat surprisingly, a fairly consistent trend may be seen across the narrative time of each tale, no matter the tale. Figure 3 shows the percentages of Latinate words per segment, averaged across all tales and prologues. On average, the first eighth of each tale shows roughly six percent more Latinate words than the last eighth, with intermediary tales showing a gradual falling gradient. There may be a number of possible explanations for this phenomenon, but my theory is that Latinate words are most likely to appear in descriptions: descriptions of characters, scenes, and ideas, which are most likely to fall at the beginning of a tale. Prayers and invocations, as well, which are high in Latinate words, happen more often, and for longer stretches of time, at the beginnings of tales. Other narrative elements, like dialogue, are statistically much lower in Latinate words, and are more likely to fall in the middle or end of a tale.

Conclusions

While this study is by no means novel, and has been preceded by centuries of analogous macro-etymological analysis, the narrative explanation of these trends, one that situates them among the pilgrims’ interpersonal dynamics—departs from prior philological methods that have explained these trends in terms of Chaucer’s sources, personal vocabulary, or educational history. To summarize, I have found that:

Sudden shifts in etymological register, along the Latinate axis, at least, are coincident with the sudden shifts in tone that accompany the “quiting” interchanges among pilgrims.
Prose tales show much higher proportions of Latinate words than verse tales.
In general, tales exhibit roughly 50% more Latinate words than prologues.
When divided into segments, the average trend across tales is a drop in the use of Latinate words. This is more pronounced in tales than in prologues.

Works Cited

Burnley, J. D. 1983. A Guide to Chaucer’s Language. London? Macmillan.

Cannon, Christopher. 2003. “Chaucer’s Style.” In The Cambridge Companion to Chaucer, edited by Piero Boitani and Jill Mann, 2nd ed. Cambridge Companions to Literature. Cambridge, U.K. ; New York: Cambridge University Press.

Chaucer, Geoffrey. 2000. The Canterbury Tales, and Other Poems. Edited by David Laing Purves. http://www.gutenberg.org/ebooks/2383.

———. 2008. The Riverside Chaucer. Edited by Benson. Oxford University Press.

Horobin, Simon. 2007. Chaucer’s Language. New York: Palgrave Macmillan.

Melo, Gerard de. 2014. “Etymological Wordnet: Tracing the History of Words.” In Proceedings of the 9th Language Resources and Evaluation Conference (LREC 2014). Paris, France: ELRA.

Mersand, Joseph E. 1939. Chaucer’s Romance Vocabulary. New York: Comet Press.

The code for this algorithm is available in the notebook chaucer-macro-etym on the project GitHub repository.↩︎
The code that parses these dictionaries may be found in the notebooks parse-CMED and parse-oed notebooks, respectively.↩︎

A Project Gutenberg Database for Text Mining

2017-06-23

Project Gutenberg is a large store of public domain electronic texts, one which has been around since the 70s. Nearly everyone that has experimented with computational literary analysis has at some point used their electronic texts. Many digital humanists undoubtedly share my frustrations with it: its interface is clunky, its metadata is incomplete, and it’s not very friendly to computational text extraction. Inspired by David McClure’s textual databases, used at the Stanford Literary Lab, I decided to fix this by creating a structured database for Project Gutenberg’s corpus, and augmenting it as much as possible with publicly-available book data. This database contains the complete cleaned text of each work, all its associated metadata, and additional metadata derived from GITenberg and Wikipedia. Here’s an example entry, Dickens’s novel A Tale of Two Cities:

LCC                                                                  {PR}
author                                                   Dickens, Charles
authoryearofbirth                                                    1812
authoryearofdeath                                                    1870
downloads                                                           11809
formats                 {'text/plain; charset=utf-8': 'http://www.gute...
id                                                                     98
languages                                                            [en]
lcsh                    {London (England) -- History -- 18th century -...
title                                                A Tale of Two Cities
type                                                                 Text
_repo                                             A-Tale-of-Two-Cities_98
_version                                                            0.2.2
alternative_title                                                     NaN
contributor                                                           NaN
covers                  [{'attribution': 'Alexis Lampley, 2015', 'cove...
creator                 {'author': {'agent_name': 'Dickens, Charles', ...
description                                                           NaN
edition_identifiers     {'edition_id': 'http://www.gutenberg.org/ebook...
edition_note                                                          NaN
gutenberg_bookshelf                                    Historical Fiction
gutenberg_issued                                               1994-01-01
gutenberg_type                                                       Text
identifiers                                           {'gutenberg': '98'}
jmdate                                                         2011-01-23
subjects                ['Historical fiction', 'French -- England -- L...
language_note                                                         NaN
production_note                                                       NaN
publication_date                                               2015-08-01
publication_note                                                      NaN
rights                                                           CC BY-NC
rights_url                 http://creativecommons.org/licenses/by-nc/4.0/
series_note                                                           NaN
summary                                                               NaN
tableOfContents                                                       NaN
titlepage_image                                                       NaN
url                                    http://www.gutenberg.org/ebooks/98
wikipedia                                                             NaN
filename                /run/media/jon/SAMSUNG/gitenberg/A-Tale-of-Two...
releaseDate                                                           NaN
wp_publication_date                                                  1859
wp_subjects             ['Novels_by_Charles_Dickens', 'Victorian_novel...
wp_info                 {'http://www.w3.org/1999/02/22-rdf-syntax-ns#t...
wp_literary_genres                                 ['Historical_fiction']

The first few fields contain Project Gutenberg’s own metadata about the book: the Dickens’s dates of birth and death, for instance. Project Gutenberg also provides Library of Congress classification and subject data: PR (British Literature) as well as these subjects:

'British -- France -- Paris -- Fiction',
'Executions and executioners -- Fiction',
'France -- History -- Revolution, 1789-1799 -- Fiction',
'French -- England -- London -- Fiction',
'Historical fiction',
'London (England) -- History -- 18th century -- Fiction',
'Lookalikes -- Fiction',
'Paris (France) -- History -- 1789-1799 -- Fiction',
'War stories'

Already you can see that this might be useful for corpus generation. If you’re interested in, say, the properties of novels that deal with lookalikes, you can easily build a corpus using these subject headings, and use those texts for your analyses.

To Project Gutenberg’s metadata, I’ve added metadata from GITenberg. GITenberg, like my project Git-Lit, is an initiative to make editable GitHub repositories for each Project Gutenberg book. They’ve also enhanced the metadata for these works. Their edition of A Tale of Two Cities, for instance, contains a nice ASCIIDOC version of the book which anyone can edit in the browser, and a Creative Commons-licensed cover designed as part of the Recovering the Classics project. All of this data, as well as a link to the GITenberg repository itself, is included in the database, along with GITenberg’s version of the text.

The next few fields come from Wikipedia. I queried DBPedia for the books with titles and authors similar to those in Project Gutenberg, and it returned about 1,800 matches. The wp_info field here contains the complete structured set of data about the book, which I then parsed into the more useful individual fields prefixed with wp_. This includes, for instance, wp_subjects, the categories to which the book’s Wikipedia page belongs. For A Tale of Two Cities, this is:

'Novels_by_Charles_Dickens', 
'Victorian_novels', 
'19th-century_novels', 
'British_novels', 
'1775_in_fiction', 
'1859_novels', 
'Chapman_&_Hall_books', 
'Novels_adapted_into_radio_programs', 
'Novels_adapted_into_television_programs', 
'Novels_adapted_into_operas', 
'A_Tale_of_Two_Cities', 
'Novels_adapted_into_plays', 
'Novels_adapted_into_comics', 
'Novels_adapted_into_films', 
'Novels_first_published_in_serial_form', 'Works_originally_published_in_All_the_Year_Round', 
'Novels_set_in_Paris', 
'Novels_set_in_London', 
'Novels_set_in_the_French_Revolution'

The potential for corpus creation here is immense. For instance, to get a corpus of all novels that Wikipedia lists as set in London, I can just run the Pandas query: df[df.wp_subjects.str.contains('Novels_set_in_London')], which returns a table of 46 novels. To get the average year of birth of the authors in this corpus, I can append .authordateofbirth.mean() (it’s 1844). By using this metadata to query other APIs, I can then do fun things like compare the average Goodreads ratings for novels set in London and in Paris. (Paris wins, with an average rating of 3.8, compared with an average rating of 3.5 for London novels.)

Corpus Statistics

Subjects

Here are the top ten Project Gutenberg-provided Library of Congress subject headings, along with the numbers of associated texts:

('Fiction', 1921),
('Short stories', 1604),
('Science fiction', 1286),
('Adventure stories', 789),
('Historical fiction', 654),
('Conduct of life -- Juvenile fiction', 639),
('Poetry', 634),
('Love stories', 620),
('English wit and humor -- Periodicals', 555),
('Detective and mystery stories', 546)

For those books that could be found on Wikipedia, here are the top ten Wikipedia categories for those books, along with their counts:

('Novels_first_published_in_serial_form', 247),
('British_novels_adapted_into_films', 109),
('19th-century_American_novels', 99),
('Victorian_novels', 96),
('British_novels', 93),
('Novels_adapted_into_plays', 92),
('English_novels', 88),
('20th-century_American_novels', 85),
('American_science_fiction_novels', 63),
('American_novels_adapted_into_films', 59),
('19th-century_novels', 58),

Other interesting categories include Debut Novels (51), 1915 Novels (39), Gothic Novels (37) and Novels Adapted into Comics (36). These categories could easily be used to create sub-corpora. If you’re interested in, say, differences between American and British science fiction novels, whether there’s anything unusually characteristic about 1915 novels, or the properties of novels that have been adapted into comics, it’s easy to construct those experiments using this corpus. Finally, here are the most common Project Gutenberg “bookshelves”, with associated text counts:

[('Bestsellers, American, 1895-1923', 225),
 ("Children's Literature", 178),
 ('The Mirror of Literature, Amusement, and Instruction', 174),
 ("Children's Book Series", 168),
 ('Historical Fiction', 164),
 ('US Civil War', 115),
 ('Best Books Ever Listings', 110),
 ("Children's Fiction", 99),
 ('Movie Books', 86),
 ('FR Littérature', 86)]

It was a dark and stormy night…

I wasn’t very surprised to find 306 works by Williams Shakespeare in the Project Gutenberg database, but I found it very surprising that the next most-represented author is Edward Bulwer Lytton, with 219 texts. Here are a few more:

Various                                            3253
Anonymous                                           719
Shakespeare, William                                306
Lytton, Edward Bulwer Lytton, Baron                 219
Ebers, Georg                                        172
Twain, Mark                                         164
Balzac, Honoré de                                   139
Verne, Jules                                        137
Kingston, William Henry Giles                       133

Languages

And here are the numbers of texts in Project Gutenberg, by language:

English:             43410
French:               2766
Finnish:              1622
German:               1516
Dutch:                 749
Italian:               690
Portuguese:            544
Spanish:               538
Chinese:               425
Modern Greek (1453-):  219

Future Developments

In the coming months, I’m going to try to generate sub-corpora from this database, starting with large corpora like American Literature and British Literature, and moving into more specialized corpora, like single-author corpora. I’ll make these all available through the corpus downloader corpus that I’m developing with DHBox.

Code

To see the (very messy) code that I used to generate this database, check out this project on GitHub. I’ll release the database itself, as well, as soon as I can figure out a way to get it inexpensively online. If you have any ideas for how best to accomplish that, please leave a note in the comments below!

Computationally Identifying Similar Books in Project Gutenberg

2017-12-05

As one of the first digital libraries, Project Gutenberg has lived through a few generations of computers, digitization techniques, and textual infrastructures. It’s not surprising, then, that the corpus is fairly messy. Early transcriptions of some electronic texts, hand-keyed using only uppercase letters, were succeeded by better transcriptions, but without replacing the early versions. As such, working with the corpus as a whole means working with a soup of duplicates. To make matters worse, some early versions of text were broken into many parts, presumably as a means to mitigate historical bandwidth limitations. Complete versions were then later created, but without removing the original parts. I needed a way to deduplicate Project Gutenberg books.

To do this, I used a suggestion from Ben Schmidt and vectorized each text, using the new Python-based natural language processing suite SpaCy. SpaCy creates document vectors by averaging word vectors from its model containing about 1.1M 300-dimensional vectors. These document vectors can then be compared using cosine similarity to determine the semantic similarities of the documents. It turns out that this is a fairly good way to identify duplicates, but has some interesting side-effects.

Here, for instance, are high-ranking similarities (99.99% vector similarity or above) for the first 100 works in Project Gutenberg. The numbers are the Project Gutenberg book IDs (see, for instance, this index of the first 768 works).

1.  The King James Version of (10) -similar to- The Bible, King James Ver (30)
2.  Alice's Adventures in Won (11) -similar to- Through the Looking-Glass (12)
3.  Through the Looking-Glass (12) -similar to- Alice's Adventures in Won (11)
4.  The 1990 CIA World Factbo (14) -similar to- The 1992 CIA World Factbo (48)
5.  Paradise Lost             (20) -similar to- Paradise Lost             (26)
6.  O Pioneers!               (24) -similar to- The Song of the Lark      (44)
7.  O Pioneers!               (24) -similar to- Alexander's Bridge        (91)
8.  Paradise Lost             (26) -similar to- Paradise Lost             (20)
9.  The 1990 United States Ce (29) -similar to- The 1990 United States Ce (37)
10. The Bible, King James Ver (30) -similar to- The King James Version of (10)
11. The 1990 United States Ce (37) -similar to- The 1990 United States Ce (29)
12. The Strange Case of Dr. J (42) -similar to- The Strange Case of Dr. J (43)
13. The Strange Case of Dr. J (43) -similar to- The Strange Case of Dr. J (42)
14. The Song of the Lark      (44) -similar to- O Pioneers!               (24)
15. The Song of the Lark      (44) -similar to- Alexander's Bridge        (91)
16. Anne of Green Gables      (45) -similar to- Anne of Avonlea           (47)
17. Anne of Green Gables      (45) -similar to- Anne of the Island        (50)
18. Anne of Avonlea           (47) -similar to- Anne of Green Gables      (45)
19. Anne of Avonlea           (47) -similar to- Anne of the Island        (50)
20. The 1992 CIA World Factbo (48) -similar to- The 1990 CIA World Factbo (14)
21. The 1992 CIA World Factbo (48) -similar to- The 1993 CIA World Factbo (84)
22. Anne of the Island        (50) -similar to- Anne of Green Gables      (45)
23. Anne of the Island        (50) -similar to- Anne of Avonlea           (47)
24. A Princess of Mars        (60) -similar to- The Gods of Mars          (62)
25. A Princess of Mars        (60) -similar to- Warlord of Mars           (65)
26. The Gods of Mars          (62) -similar to- A Princess of Mars        (60)
27. The Gods of Mars          (62) -similar to- Warlord of Mars           (65)
28. Warlord of Mars           (65) -similar to- A Princess of Mars        (60)
29. Warlord of Mars           (65) -similar to- The Gods of Mars          (62)
30. Adventures of Huckleberry (73) -similar to- Tom Sawyer Abroad         (88)
31. Tarzan of the Apes        (75) -similar to- The Return of Tarzan      (78)
32. The Return of Tarzan      (78) -similar to- Tarzan of the Apes        (75)
33. The Beasts of Tarzan      (82) -similar to- Tarzan and the Jewels of  (89)
34. The 1993 CIA World Factbo (84) -similar to- The 1992 CIA World Factbo (48)
35. Tom Sawyer Abroad         (88) -similar to- Adventures of Huckleberry (73)
36. Tarzan and the Jewels of  (89) -similar to- The Beasts of Tarzan      (82)
37. Alexander's Bridge        (91) -similar to- O Pioneers!               (24)
38. Alexander's Bridge        (91) -similar to- The Song of the Lark      (44)

The first pair here is of duplicates: both are King James Versions of the Bible. The same is true of lines 5 and 8, and lines 12-13: they’re just duplicates. All the other works are members of a novelistic series. Lines 2 and 3 are Alice in Wonderland and its sequel. Lines 6 and 7 are Willa Cather novels of the Great Plains trilogy. Lines 16-19, and 22-23 identify the Anne of Green Gables novels. Lines 24-29 are a cluster of Edgar Rice Burroughs of the Mars series, and there is also another cluster of Burroughs novels, the Tarzan series, at 31-33. Line 35 shows Mark Twain novels of the Tom Sawyer and Huck Finn world. The algorithm even identifies the two 90s CIA World Factbooks as part of a series.

When I lower the cutoff similarity score, I can get even more interesting pairs. Less-recognizable series, like Paradise Lost and Paradise Regained, have similarity scores of around 97%. At that level, completely unrelated novels with the same settings, or written in around the same time period (Victorian novels, for instance), begin to cluster together.

The chart below shows a PCA-reduced 2D vector space approximating the similarity between books 1-20. There are interesting clusters here: the American Constitution and Bill of Rights cluster together, along with the Declaration of Independence, the Federalist Papers, and Abraham Lincoln’s first inaugural address. Lincoln’s second address, however, clusters rather with his Gettysburg Address, and John F. Kennedy’s inaugural address.

PCA of PG Books 1-20

Non-fiction seems to be clustered at the bottom of the chart, whereas fiction is at the top. Quasi-fictional works, like the Book of Mormon and the Bible, are in between. Similarly, Moby Dick, a work of fiction that nonetheless contains long encyclopedic passages of non-fiction, lies in the same area. The most fantastical works, which are also the three children’s books, Peter Pan and the two Carroll novels, cluster together in the upper left.

As always, the code used to create all of this is on GitHub. I welcome your comments and suggestions below!

Fingerprinting the Chapter

2018-03-07

What is a chapter, really? Inspired by Nicholas Dames’s work on the history of the chapter (some of which appeared in the New Yorker a few years ago as “The Chapter: A History”), I’ve started investigating this question. We all have an intuition about chapters—how long they should be, their temporal scope, what they generally contain. Most of us, if given a paragraph chosen at random from a novel, could probably tell whether it began a chapter, ended a chapter, or fell somewhere in between. Yet could we say why? Could we explain our intuition to a computer? Sure, if we read a line like “and they rode off into the setting sun” we might guess that it belongs to the end of a chapter. But is the presence of a setting sun a distinguishing criterion for the end of a chapter? Do chapters usually represent a single day in narrative time? I decided to compute the linguistic fingerprint of the chapter, to get closer to understanding what it is.

First, I gathered as many novels as I could. Corpus-DB, the textual corpus database I’ve been developing, makes it easy to get large numbers of plain text files that match certain categories. At the moment it contains all 50,000 or so texts from Project Gutenberg, and metadata about these books gathered from Wikipedia and other sources. Using this database, I was easily able to assemble corpora of several thousand chaptered novels. I then broke these novels into chapters using two methods. The first, using my tool chapterize, reads the plain text files of the novels, looks for markers like “Chapter I” using regular expressions, and divides the novels accordingly. The second leverages hidden metadata in Project Gutenberg HTML files. If you take a peek at the source code of a Project Gutenberg HTML file, you’ll see named anchors that look like <a name"c2" id=“c2”>=. These IDs correspond to the chapter numbers—in this case, Chapter 2. By counting the number of words of text that occur between these markers, we can measure the chapter lengths of thousands of novels at a time. And by extracting the text that begins and ends these chapters, we can find language patterns that define a chapter’s boundaries. Here are some trends I found.

Chapter Statistics

Figure 1 shows the average number of chapters, the average text length, and the average chapter length for prose fiction works written by authors who were born in the decades shown. There is an amazing range of dates here, indicating the presence of some decidedly non-novelistic classical literature. (Project Gutenberg doesn’t maintain metadata about its texts’ original publication dates, but they do have author birth/death years, strangely, so I’m forced to use those as a proxy. This is a problem I’m trying to correct with Corpus-DB, but my solution isn’t quite ready yet.) This seems to show that by around the 19th Century, chapters are about the same length. They get a little shorter around the 20th Century, and are extreme and erratic before 1700, but I attribute this to the relative paucity of early (pre-1700) and late (post-1920) texts in the corpus, owing to factors like textual availability, canonicity, and copyright restrictions. It appears that novels have been getting shorter, however, and so there are also fewer chapters.

Figure 1: Chapter statistics by author date of birth

Figure 2 shows the average number of chapters per novel, sorted by the novel’s Library of Congress subject headings. Subjects with the most number of chapters are on the left, and subjects with the least are on the right. There are a few things of note here:

The subject with the highest average is 19th Century English fiction, followed by Bildungsromans. Are Bildungsromans so long and chapter-filled because they have to represent the life of a person, rather than just one or two episodes in that person’s life?
“Young men – fiction” is ranked slightly higher than “young women – fiction.”
Civil war novels are ranked higher here than WWI novels. Is this a function of era (20th vs. 19th C), geography, or the nature of the wars themselves?

Of course, there are many more things that can be said about this chart. Please feel free to chime in, in the comments below!

Figure 2: Average number of chapters by Library of Congress subject heading

Language Patterns of the Chapter

For the next experiment, I extracted the first paragraphs of each chapter from a few thousand novels, as well as middle paragraphs and final paragraphs. From there, I calculated TF-IDF scores for each word. TF-IDF just represents word frequencies, adjusted for the frequencies of those words in the corpus. In this case, it represents term distinctiveness to a paragraph group: how frequently a term occurs in first paragraph of a chapter, for instance, compared with how frequently it occurs in other paragraph categories. Here are the words distinctive of the first paragraphs of chapters:

morning, early, breakfast, afternoon, summer, autumn, winter, sunday, weather, october, arrival, june, september, saturday, awoke, situated, november, july, season, december

It seems that the first paragraphs of chapters set the scene: the season, the month, and the time of day (usually the morning). Just for fun, I computed the relative mentions of certain months from this small sample, and found that my intuition was correct: spring is over-represented in the beginnings of chapters, and winter is under-represented, with May as the most mentioned month, and January the least, as shown in Figure 3.

Figure 3: Months mentioned in the first paragraphs of chapters

Here are the words distinctive of middle paragraphs:

replied, isn, ve, retorted, aren, hasn, mustn, shouldn, dey, inquired, doesn, haven, ye, wid, fer, wi, em, yer, protested, nothin

Some of these irregularities are attributable to the tokenizer, which is breaking up contractions in the wrong places, but others are dialect words. This seems to show that dialect—funny-spelled words—is not something that usually starts or ends a chapter. In Dickens, for instance, dialect is usually heard from characters intended to provide some kind of comic relief. These comic characters rarely have a narratorial role, it seems, and rarely bookend chapters.

Finally, here are the words distinctive of last paragraphs:

pg, kissed, farewell, bye, muttered, parted, disappeared, sank, page, asleep, strode, chapter, kiss, withdrew, homeward, sobbing, thanked, wept, murmured, prayed

There is a remarkably sad tone here in “sobbing,” “wept,” and “prayed.” Lots of goodbyes are happening here, as well. “Pg” and “page,” seem to be OCR artifacts, but is “chapter” a computing error, or the narrative voice, signaling the end of the chapter?

What do you think defines a chapter? Let me know in the comments.

Code

The code used to generate all this is in my GitHub repo called chapter-experiments.

Workshop Notebook: Advanced Topics in Word Embeddings

2019-02-20

This notebook originally accompanied a workshop I gave at NYCDH Week, in February of 2019, called “Advanced Topics in Word Embeddings.” (In truth, it’s only somewhat advanced. With a little background in NLP, this could even serve as an introduction to the subject.) You can run the code in a Binder, here.

Word embeddings are among the most discussed subjects in natural language processing, at the moment. If you’re not already familiar with them, there are a lot of great introductions out there. In particular, check out these:

An Example of Document Vectors: Project Gutenberg

This figure shows off some of the things you can do with document vectors. Using just the averaged word vectors of each document, and projecting them onto PCA space, you can see a nice divide between fiction and nonfiction books. In fact, I like to think of the line connecting the upper-left and the lower-right as a vector of “fictionality,” withthe upper-left corner as “highly fictional,” and the lower right as “highly non-fictional.” Curiously, religious texts are right in between.

First 30 Books of Project Gutenberg

There’s more on this experiment in this 2015 post.

Getting Started

First, import the libraries below. (Make sure you have the packages beforehand, of course.)

import pandas as pd
import spacy
from glob import glob
# import word2vec
# import gensim
# from gensim.test.utils import common_texts
# from gensim.models import Word2Vec
from sklearn.manifold import TSNE
from sklearn.decomposition import PCA
from matplotlib import pyplot as plt
import json
from mpl_toolkits.mplot3d import Axes3D, proj3d #???
from numpy import dot
from numpy.linalg import norm
%matplotlib notebook
plt.rcParams["figure.figsize"] = (12,8)

Now load the Spacy data that you downloaded (hopefully) prior to the workshop. If you don’t have it, or get an error below, you might want to check out the documentation that Spacy maintains here for how to download language models. Download the en_core_web_lg model.

nlp = spacy.load('en_core_web_lg')

Word Vector Similarity

First, let’s make SpaCy “document” objects from a few expressions. These are fully parsed objects that contain lots of inferred information about the words present in the document, and their relations. For our purposes, we’ll be looking at the .vector property, and comparing documents using the .similarity() method. The .vector is just an average of the word vectors in the document, where each word vector comes from pre-trained model—the Stanford GloVe vectors. Just for fun, I’ve taken the examples below from Monty Python and the Holy Grail, the inspiration for the name of the Python programming language. (If you haven’t seen it, this is the scene I’m referencing..)

africanSwallow = nlp('African swallow')
europeanSwallow = nlp('European swallow')
coconut = nlp('coconut')

africanSwallow.similarity(europeanSwallow)

0.8596378859289445

africanSwallow.similarity(coconut)

0.2901231866716321

The .similarity() method is nothing special. We can implement our own, using dot products and norms:

def similarity(vecA, vecB):
    return dot(vecA, vecB) / (norm(vecA, ord=2) * norm(vecB, ord=2))

similarity(africanSwallow.vector, europeanSwallow.vector)

0.8596379

Analogies (Linear Algebra)

In fact, using our custom similarity function above is probably the easiest way to do word2vec-style vector arithmetic (linear algebra). What will we get if we subtract “European swallow” from “African swallow”?

swallowArithmetic = (africanSwallow.vector - europeanSwallow.vector)

To find out, we can make a function that will find all words with vectors that are most similar to our vector. If there’s a better way of doing this, let me know! I’m just going through all the possible words (all the words in nlp.vocab) and comparing them. This should take a long time.

def mostSimilar(vec):
    highestSimilarities = [0]
    highestWords = [""]
    for w in nlp.vocab:
        sim = similarity(vec, w.vector)
        if sim > highestSimilarities[-1]:
            highestSimilarities.append(sim)
            highestWords.append(w.text.lower())
    return list(zip(highestWords, highestSimilarities))[-10:]

mostSimilar(swallowArithmetic)

[('croup', 0.06349668),
 ('deceased', 0.11223719),
 ('jambalaya', 0.14376064),
 ('cobra', 0.17929554),
 ('addax', 0.18801448),
 ('tanzania', 0.25093195),
 ('rhinos', 0.3014531),
 ('lioness', 0.34080425),
 ('giraffe', 0.37119308),
 ('african', 0.5032688)]

Our most similar word here is “african”! So “European swallow” - “African swallow” = “African”! Just out of curiosity, what will it say is the semantic neighborhood of “coconut”?

mostSimilar(coconut.vector)

[('jambalaya', 0.24809697),
 ('tawny', 0.2579049),
 ('concentrate', 0.35225457),
 ('lasagna', 0.36302277),
 ('puddings', 0.4095627),
 ('peel', 0.47492552),
 ('eucalyptus', 0.4899935),
 ('carob', 0.57747585),
 ('peanut', 0.6609557),
 ('coconut', 1.0000001)]

Looks like a recipe space. Let’s try the classic word2vec-style analogy, king - man + woman = queen:

king, queen, woman, man = [nlp(w).vector for w in ['king', 'queen', 'woman', 'man']]

answer = king - man + woman

mostSimilar(answer)

[('gorey', 0.03473952),
 ('deceased', 0.2673984),
 ('peasant', 0.32680285),
 ('guardian', 0.3285926),
 ('comforter', 0.346274),
 ('virgins', 0.3561441),
 ('kissing', 0.3649173),
 ('woman', 0.5150813),
 ('kingdom', 0.55209804),
 ('king', 0.802426)]

It doesn’t work quite as well as expected. What about for countries and their capitals? Paris - France + Germany = Berlin?

paris, france, germany = [nlp(w).vector for w in ['Paris', 'France', 'Germany']]
answer = paris - france + germany
mostSimilar(answer)

[('orlando', 0.48517892),
 ('dresden', 0.51174784),
 ('warsaw', 0.5628617),
 ('stuttgart', 0.5869507),
 ('vienna', 0.6086052),
 ('prague', 0.6289497),
 ('munich', 0.6677783),
 ('paris', 0.6961337),
 ('berlin', 0.75474036),
 ('germany', 0.8027713)]

It works! If you ignore the word itself (“Germany”), then the next most similar one is “Berlin”!

Pride and Prejudice

Now let’s look at the first bunch of nouns from Pride and Prejudice. It starts:

It is a truth universally acknowledged, that a single man in possession of a good fortune, must be in want of a wife.
However little known the feelings or views of such a man may be on his first entering a neighbourhood, this truth is so well fixed in the minds of the surrounding families, that he is considered the rightful property of some one or other of their daughters.

First, load and process it. We’ll grab just the first fifth of it, so we won’t run out of memory. (And if you still run out of memory, maybe increase that number.)

pride = open('pride.txt').read()

pride = pride[:int(len(pride)/5)]

prideDoc = nlp(pride)

Now grab the first, say, 40 nouns.

prideNouns = [w for w in prideDoc if w.pos_.startswith('N')][:40]
prideNounLabels = [w.lemma_ for w in prideNouns]

prideNounLabels[:10]

['truth',
 'man',
 'possession',
 'fortune',
 'want',
 'wife',
 'feeling',
 'view',
 'man',
 'neighbourhood',
 'truth',
 ...

Get the vectors of those nouns.

prideNounVecs = [w.vector for w in prideNouns]

Verify that they are, in fact, our 300-dimensional vectors.

prideNounVecs[0].shape

(300,)

Use PCA to reduce them to three dimensions, just so we can plot them.

reduced = PCA(n_components=3).fit_transform(prideNounVecs)

reduced[0].shape

(3,)

prideDF = pd.DataFrame(reduced)

Plot them interactively, in 3D, just for fun.

%matplotlib notebook
plt.rcParams["figure.figsize"] = (10,8)

def plotResults3D(df, labels): 
    fig = plt.figure()
    ax = fig.add_subplot(111, projection='3d')
    ax.scatter(df[0], df[1], df[2], marker='o')
    for i, label in enumerate(labels):
        ax.text(df.loc[i][0], df.loc[i][1], df.loc[i][2], label)

plotResults3D(prideDF, prideNounLabels)

Pride and Prejudice Nouns

Now we can rewrite the above function so that instead of cycling through all the words ever, it just looks through all the Pride and Prejudice nouns:

# Redo this function with only nouns from Pride and Prejudice
def mostSimilar(vec):
    highestSimilarities = [0]
    highestWords = [""]
    for w in prideNouns:
        sim = similarity(vec, w.vector)
        if sim > highestSimilarities[-1]:
            highestSimilarities.append(sim)
            highestWords.append(w.text.lower())
    return list(zip(highestWords, highestSimilarities))[-10:]

Now we can investigate, more rigorously than just eyeballing the visualization above, the vector neighborhoods of some of these words:

mostSimilar(nlp('fortune').vector)

[('', 0), ('truth', 0.3837785), ('man', 0.40059176), ('fortune', 1.0000001)]

Senses

If we treat words as documents, and put them in the same vector space as other documents, we can infer how much like that word the document is, vector-wise. Let’s use four words representing the senses:

senseDocs = [nlp(w) for w in ['sound', 'sight', 'touch', 'smell']]
def whichSense(word):
    doc = nlp(word)
    return {sense: doc.similarity(sense) for sense in senseDocs}

whichSense('symphony')

{sound: 0.37716483832358116,
 sight: 0.20594014841156277,
 touch: 0.19551651130481998,
 smell: 0.19852637065751555}

%matplotlib inline
plt.rcParams["figure.figsize"] = (14,8)

testWords = 'symphony itchy flower crash'.split()
pd.DataFrame([whichSense(w) for w in testWords], index=testWords).plot(kind='bar')

Pride and Prejudice Nouns

It looks like it correctly guesses that symphony correlates with sound, and also does so with crash, but its guesses for itchy (smell) and for flower (touch) are less intuitive.

The Inaugural Address Corpus

In this repo, I’ve prepared a custom version of the Inaugural Address Corpus included with the NLTK. It just represents the inaugural addresses of most of the US presidents from the 20th and 21st centuries. Let’s compare them using document vectors! First let’s generate parallel lists of documents, labels, and other metadata:

inauguralFilenames = sorted(glob('inaugural/*'))
inauguralLabels = [fn[10:-4] for fn in inauguralFilenames]
inauguralDates = [int(label[:4]) for label in inauguralLabels]
parties = 'rrrbbrrrbbbbbrrbbrrbrrrbbrrbr' # I did this manually. There are probably errors.
inauguralRaw = [open(f, errors="ignore").read() for f in inauguralFilenames]

# Sanity check: peek
for i in range(4): 
    print(inauguralLabels[i][:30], inauguralDates[i], inauguralRaw[i][:30])

1901-McKinley 1901 My fellow-citizens, when we as
1905-Roosevelt 1905 My fellow citizens, no people 
1909-Taft 1909 My fellow citizens: Anyone who
1913-Wilson 1913 There has been a change of gov

Process them and compute the vectors:

inauguralDocs = [nlp(text) for text in inauguralRaw]

inauguralVecs = [doc.vector for doc in inauguralDocs]

Now compute a similarity matrix for them. Check the similarity of everything against everything else. There’s probably a more efficient way of doing this, using sparse matrices. If you can improve on this, please send me a pull request!

similarities = []
for vec in inauguralDocs: 
    thisSimilarities = [vec.similarity(other) for other in inauguralDocs]
    similarities.append(thisSimilarities)

df = pd.DataFrame(similarities, columns=inauguralLabels, index=inauguralLabels)

Now we can use .idmax() to compute the most semantically similar addresses.

df[df < 1].idxmax()

1901-McKinley        1925-Coolidge
1905-Roosevelt         1913-Wilson
1909-Taft            1901-McKinley
1913-Wilson         1905-Roosevelt
1917-Wilson         1905-Roosevelt
1921-Harding       1953-Eisenhower
1925-Coolidge       1933-Roosevelt
1929-Hoover          1901-McKinley
1933-Roosevelt       1925-Coolidge
1937-Roosevelt      1933-Roosevelt
1941-Roosevelt      1937-Roosevelt
1945-Roosevelt        1965-Johnson
1949-Truman           1921-Harding
1953-Eisenhower    1957-Eisenhower
1957-Eisenhower    1953-Eisenhower
1961-Kennedy            2009-Obama
1965-Johnson            1969-Nixon
1969-Nixon            1965-Johnson
1973-Nixon             1981-Reagan
1977-Carter             2009-Obama
1981-Reagan            1985-Reagan
1985-Reagan            1981-Reagan
1989-Bush             1965-Johnson
1993-Clinton            2017-Trump
1997-Clinton           1985-Reagan
2001-Bush              1981-Reagan
2005-Bush          1953-Eisenhower
2009-Obama             1981-Reagan
2017-Trump            1993-Clinton
dtype: object

If we reduce the dimensions here using PCA, we can visualize the similarity in 2D:

embedded = PCA(n_components=2).fit_transform(inauguralVecs)

xs, ys = embedded[:,0], embedded[:,1]
for i in range(len(xs)): 
    plt.scatter(xs[i], ys[i], c=parties[i], s=inauguralDates[i]-1900)
    plt.annotate(inauguralLabels[i], (xs[i], ys[i]))

Presidential Inaugural Address Vectors

Detective Novels

I’ve prepared a corpus of detective novels, using another notebook in this repository. It contains metadata and full texts of about 10 detective novels. Let’s compute their similarities to certain weapons! It seems the murder took place in the drawing room, with a candlestick, and the murderer was Colonel Mustard!

detectiveJSON = open('detectives.json')
detectivesData = json.load(detectiveJSON)
detectivesData = detectivesData[1:] # Chop off #1, which is actually a duplicate

detectiveTexts = [book['text'] for book in detectivesData]

We might want to truncate these texts, so that we’re comparing the same amount of text throughout.

detectiveLengths = [len(text) for text in detectiveTexts] 
detectiveLengths

[351240, 415961, 440629, 611531, 399572, 242949, 648486, 350142, 288955]

detectiveTextsTruncated = [t[:min(detectiveLengths)] for t in detectiveTexts]

detectiveDocs = [nlp(book) for book in detectiveTextsTruncated] # This should take a while

extraWords = "gun knife snake diamond".split()
extraDocs = [nlp(word) for word in extraWords]
extraVecs = [doc.vector for doc in extraDocs]

detectiveVecs = [doc.vector for doc in detectiveDocs]
detectiveLabels = [doc['author'].split(',')[0] +  '-' + doc['title'][:20] for doc in detectivesData]

detectiveLabels

['Collins-The Haunted Hotel: A',
 'Rohmer-The Insidious Dr. Fu',
 'Chesterton-The Innocence of Fat',
 'Doyle-The Return of Sherlo',
 'Chesterton-The Wisdom of Father',
 'Doyle-A Study in Scarlet',
 "Gaboriau-The Count's Millions",
 "Rinehart-Where There's a Will",
 "Michelson-In the Bishop's Carr"]

pcaOut = PCA(n_components=10).fit_transform(detectiveVecs + extraVecs)
tsneOut = TSNE(n_components=2).fit_transform(pcaOut)

xs, ys = tsneOut[:,0], tsneOut[:,1]
for i in range(len(xs)): 
    plt.scatter(xs[i], ys[i])
    plt.annotate((detectiveLabels + extraWords)[i], (xs[i], ys[i]))

Detective Novel Vectors

If you read the summaries of some of these novels on Wikipedia, this isn’t terrible. To check, let’s just see how often these words occur in the novels.

# Sanity check
counts = {label: {w: 0 for w in extraWords} for label in detectiveLabels}
for i, doc in enumerate(detectiveDocs):
    for w in doc: 
        if w.lemma_ in extraWords: 
            counts[detectiveLabels[i]][w.lemma_] += 1

pd.DataFrame(counts).T.plot(kind='bar')

Weapons By Novel

Isolating Literary Style with Raymond Queneau

2019-09-07

Recently, I’ve become interested in the problem of literary style. One of its great difficulties involves the separation of a writer’s stylistic signal from that of his or her era, nation, idiom, dialect, or chosen literary genre. Ideally, one might construct a corpus that controls for most of these variables, so that the style itself can be studied more easily. But since such corpora are hard to come by, I had the crazy idea to analyze a book that tells the same story over and over, but in different styles each time. One such book is Exercises in Style, and Raymond Queneau, celebrated literary experimenter, and founder of the famed OULIPO group, wrote it in 1947. Barbara Wright translated it into English in 1958, and because the English NLP toolchain is the most well developed, I chose to analyze the English translation.

Queneau’s concept was to narrate one short episode in ninety-nine styles. Here is the first, “Notation”:

In the S bus, in the rush hour. A chap of about 26, felt hat with a cord instead of a ribbon, neck too long, as if someone’s been having a tug-of-war with it. People getting off. The chap in question gets annoyed with one of the men standing next to him. He accuses him of jostling him every time anyone goes past. A snivelling tone which is meant to be aggressive. When he sees a vacant seat he throws himself on to it. Two hours later, I meet him in the Cour de Rome, in front of the gare Saint-Lazare. He’s with a friend who’s saying: “You ought to get an extra button put on your overcoat.” He shows him where (at the lapels) and why.

Some versions of this story are minimalist, like “Haiku”:

Summer S long neck
plait hat toes abuse retreat
station button friend

This is a somewhat parodic version of a haiku, reminescent of those works of “deformative criticism” which reduce a poem to only its nouns. Although here, it is difficult for both a human and a computer to parse the syntax. Are the three words in the last line simply a sequence of nouns, or are “station” and “button” adjectives that modify “friend”?

Here’s another example, “Prognostication”:

When midday strikes you will be on the rear platform of a bus which will be crammed full of passengers among whom you will notice a ridiculous juvenile; skeleton-like neck and no ribbon on his felt hat. He don’t be feeling at his ease, poor little chap. He will think that a gentleman is pushing him on purpose every time that people getting on or off pass by. He will tell him so but the gentleman won’t deign to answer. And the ridiculous juvenile will be panic-stricken and run away from him in the direction of a vacant seat. You will see him a little later, in the Cour de Rome in front of the gate Saint-Lazare. A friend will be with him and you will hear these words: “Your overcoat doesn’t do up properly; you must have another button put on it.”

This form revolves around the pronoun “you,” which is repeated in nearly every sentence. This and “haiku” are extreme examples, but they do show how style manifests in proportions of parts of speech.

The figure below shows a few selected parts of speech, and their proportions in all ninety-nine styles, in the order in which they appear in the text. (Scroll right to see the whole figure.)

There are a few things of note here. One is that, as Queneau’s styles become more specific, and more experimental, the parts of speech become much more erratic. Another is that some parts of speech are much more stable than others. The variance of nouns is nearly four times that of verbs, and eight times that of adverbs, meaning that the proportion of nouns fluctuates wildly, while the proportions of adverbs remains relatively stable.

Here are the styles that rank the highest and lowest, according to proportions of certain parts of speech. These charts are interactive, so if you hover over a bar, it’ll show you a snippet of that style.

Adjectives

The “philosophic” mode is one whose informational density and precision of language encodes its content into adjectives. Here is an example:

Great cities alone can provide phenomenological spirituality with the essentialities of temporal and improbabilistic coincidences. The philosopher who occasionally ascends into the futile and utilitarian inexistentiality of an S bus can perceive therein wiht the lucidity of his pineal eye the transitory and faded appearance of a profane consciousness afflicted by the long neck of vanity and the hatly plait of ignorance.

Nouns

Besides “haiku,” most of these outliers are styles that are phonetic or linguistic experiments: “paragoge,” “epenthesis,” and “apheresis,” for instance, all describe phonetic changes to words, which Quenaeu and Wright render with phonetic spellings. One might reasonably posit that the POS tagger misattributes some of these, but it does a surprisingly good job, since it is the probabilistic parser in the SpaCy library, which has been trained on an model of English collected from Internet text—text that is not always rendered with standard spellings. But even conceding that this would skew the rankings, when dealing with small differences in proportions, we might posit that Queneau might have highlighted the differences in these linguistic experiments with the aid of more noun-heavy phrases.

Verbs

“Reported speech” is a style which apparently emphasizes verbs:

Dr. Queneau said that it had happened at midday. Some passengers had got into the bus. They had been squashed tightly together. On his head a young man had been wearing a hat which had been encircled by a plait and not by a ribbon. He had had a long neck. […]

I suspect that the high proportion of verbs here can be partially attributed to the addition of a frame-narrative: in addition to the original story’s verbs, there are also the verbs in “Dr. Queneau said,” “Dr. Queneau continued,” and so on.

Pronouns

Of course “You Know” gets the highest proportions of pronouns here, since every other phrase is “you know”:

Well, you know, the bus arrived, so, you know, I got on. Then I saw, you know, a citizen who, you know, caught my eye, sort of.

“Modern Style” also has many more “you”s, but for the reason that its rhetoric appeals to the reader, in an almost epistolary way:

In a bus one day it so happened that I was a witness of the following as you might say tragi-comedy which revealing as it does the way our French cousins go on these days I thought I ought to put you in the picture.

This was a toy analysis in many respects, which I didn’t expect would say anything very serious about literary style. For one, the corpus is so small that a statistical approach isn’t very useful. Still, a few interesting phenomena appear here, and I’m curious to see what this looks like in a bigger corpus.

Here’s the code used to generate all this.

I’d love to hear your comments in the annotations.

Adventures in Reproducing 19th Century Digital Humanities Projects

2019-10-23

In 1887, Thomas Mendenhall, a self-taught physicist, published an article in Science called “The Characteristic Curve of Composition,” in which he studies mathematical properties of sentences written by fiction writers of the day. As an early work of quantitative literary analysis which preexisted the field of digital humanities by many decades, I like to think of it as nonetheless a “digital” study in the literal sense, even if Mendenhall was counting on his fingers instead of using a computer. Of course, what would have been a laborious process for Mendenhall and, presumably, his assistants—counting the number of words in each sentence in a novel—is now just a matter of a few lines of code. So during a hackathon at the Literary Modeling and Visualization Lab, I decided to try to implement his algorithm.

Using primarily novels of Charles Dickens, Mendenhall counts the number of words per sentence, in chunks of a thousand words each, and plots their distribution in a histogram that looks like this:

The “characteristic curve” of Oliver Twist

According to Mendenhall, this is Dickens’s “characteristic curve,” that is, his stylistic signature, as expressed in the distribution of sentence lengths. The X axis here represents the number of words in a sentence, and the Y represents the number of sentences in that category. He shows that for other novels, this curve is roughly the same as in these samples from Oliver Twist. The curves of other writers, however, are somewhat different. Nonfiction, like John Stuart Mill’s Essay on Liberty, is the most different, and other fiction is only slightly different.

During the hackathon, we decided to compare two genres of literary text: gothic novels and epistolary novels. Since Corpus-DB allows for easy corpus generation from genre (via Library of Congress subject headings), I was able to quickly throw together a corpus of about eight gothic novels and ten epistolary novels. From there, I used SpaCy to compute the distribution of sentence lengths, and group them by genre. This is the result, the “characteristic curve” of these two corpora:

Average sentence lengths, per genre

I should’ve used a bar chart here, since this is a histogram with discrete values, but in an effort to be faithful to Mendenhall’s method, I kept it as a line chart. I find it interesting that these two lines are so different from each other. The epistolary novels in this corpus have many more shorter sentences. I attribute this to formal features of the letter, like “Dear Sir,” and “Sincerely.” But then gothic novels have more longer sentences, that is, among sentences longer than twenty words.

Since these lines are so different, it seemed like this might be a good feature to use for a categorizer, which could guess the genre based on the sentence lengths of the novel. After scripting a quick function that compares the differences between all the categories in the histograms, I tried this, but only achieved an accuracy of 66%, so only slightly better than chance. Since some novels are both gothic and epistolary (Dracula comes to mind), I can see how this wouldn’t be an easy guess to make. Still, it’s something.

What other historical literary studies algorithms should I try to implement? Let me know in the comments below.

The code used for this analysis is here, on Colab.

A Proposal for a New Calendar

2020-09-26

On Calendar Reform

Let’s face it: the Gregorian calendar is a mess. There are a serious number of problems with it, but let’s list just a few:

Months have different numbers of days. This not only means that school children have to rote-learn mnemonics for which months have thirty days (“thirty days have September, April, June…”), but that we can’t use a month as a unit of time, in any practical way. The number of periodicals, services, and so on which renew every 30 days is a testament to the instability of the month. Despite the predictable regularity of the moon’s phases, we can’t rely on the calendar month.
Each year is different. We have to print new calendars for each year, since they don’t always start on the same day of the week. That also means that we can’t have holidays on the same dates every year. If your holiday has a prescribed date, like Halloween, then you have to resign yourself to the possibility that it falls on a Monday. Sure you can celebrate it on the Saturday before, but do you see what kind of a mess we’re getting into? The alternative isn’t great, either: President’s Day is the third Monday of February, which means it’s anywhere from the 15th to the 21st, and it’s something you have to figure out each year. Wouldn’t it be great if we could have a single calendar, and use it every year?
The names for the months are nonsensical. Even if you don’t object to months named after the Greco-Roman gods Janus, Mars, Aphrodite, Maia, and so on, and even if you don’t mind that Julius and Augustus Caesar added their own names to this pantheon, you’ll admit that it just gets insane in the last part of the year. September made sense when it was the seventh month (sept- is seven in Latin), October the eighth month, and so on. But to have months named for numbers, and have those numbers be off by two, is just ridiculous.
There are too many concurrent systems. The Gregorian calendar is a lunar and solar calendar, with weeks thrown on top. That’s just too many systems. The year takes its cues from the seasons, and from the rotation of the earth around the sun. But the month is modeled after lunar calendars, which don’t align with the solar. A lunar year is about eleven days shorter than a solar year. And suddenly, player three enters the game: weeks, which aren’t aligned with anything astronomical, and just keep track of seven named days over and over. Weeks aren’t aligned with months or years, either, so months start on different days of the week, and months have different numbers of weeks.

The amazing part is: we’ve known all this about the calendar, for centuries. Every so often, someone appears and proposes an improvement. We don’t seem to listen to them. But we should.

Here are a few of my favorite calendar reform ideas. All of these are improvements over our Gregorian calendar.

The French Republican Calendar

One of the more valiant efforts at calendar reform came during the French revolution. The révolutionnaires, in an effort to consciously rebuild the country, rewrote the calendar from scratch, and they mitigated, if not solved, many of the above problems. Here are some of the French Republican Calendar’s best features:

Each month has thirty days, and begins at the beginning of the week.
The week (decadi) is ten days long. There are three weeks in a month.
To solve the leap-day problem, there are five extra days—all holidays!—with names like La Fête du Génie, “Celebration of Talent”, and La Fête de l’Opinion, “Celebration of Convictions.” These all fall towards the end of the summer.
The months have poetic names, named after meteorological or agricultural cycles. Germinal, is a month of germinating. Thermidor is the month of heat. Floréál is the month of flowers.
Best of all, each day gets a name. So instead of “July 31st,” it’s abricot, or apricot. “Why don’t we grab a coffee on Apricot Day?” sounds so much nicer than “Why don’t we grab a coffee on July 31st?”. Furthermore, since each quintidi rest day is an animal, and each décadi is some kind of object or agricultural tool, you know what day of the week it is already. You don’t have to think, “July 31st, is that a Saturday?”

Germinal

Names of Days

The Positivist Calendar

Proposed by August Compte, in 1849, this calendar has 13 months of 28 days each, and a festival day each month, which was its own day of the week (that is, outside the week system). That allowed each month to start on the same day (Monday). The names are a little too worshipey for my taste, although they’re still a big improvement over Gregorian month names. They are: Moses, Homer, Aristotle, Archimedes, Caesar, Saint Paul, Charlemagne, Dante, Gutenberg, Shakespeare, Descartes, Frederick, Bichat.

This is a very male-heavy list of figures, but it’s still nice to see months named after poets, literary figures, and mathematicians. However, it’s a little ironic that Caesar is still in this calendar, but with a different name.

The Hanke-Henry Permanent Calendar

Designed in the early 2000s decade by two astronomers at Johns Hopkins, the Hanke-Henry Permanent Calendar is called “permanent” because its dates always have the same week days, making it so you wouldn’t need a new calendar every year. Every six or so years there is a leap week called “extra,” as in the French Republican calendar. But the days still vary, month to month: March, June, September, and December have thirty-one days, and the rest have thirty. This is still an improvement over the Gregorian calendar, however.

The ISO Week-Date Calendar

This calendar, which is part of the ISO 8601 standard for dates and times, dispenses with months altogether, and just uses numbered weeks. The more you think about it, the more this is appealing. Consider the way we normally have to convert months into weeks, anyway: “please allow 6-8 weeks for shipping” is a common expression that could be converted to months, but isn’t. Consider the way we currently have to talk about a certain week, like “the first full week in February.” That’s a lot of words for saying what we could say with something like “week 5.” Thee week-date calendar fixes that.

The problem is the way we’ve chosen to superimpose the two incongruous systems of weeks and months. So why don’t we just pick one and stick to it? We can’t really get rid of weeks, since they determine when we work and when we rest. And we wouldn’t really lose anything by scrapping months, and leaning into weeks.

Today’s date (September 26th, 2020) in the week-date calendar looks like this: 2020-W39-6. That’s 2020 (the year), week number 39 (the fourth week in September), and the weekday number (6, Saturday). Colloquially, I imagine one might say, “Friday, week 39,” as in, “I can’t meet this week, or next, but how about Friday, week 39?”.

The only disadvantage, as I see it, is that this just doesn’t sound as elegant as saying “September 26th.” It’s nice to have names for things. They’re a little nicer than numbers.

A Proposal for a New Calendar System

Why not combine the best features of the calendar reforms above? My favorites of the above calendars are the ISO week-date, for its simplicity of only weeks, and the French revolutionary calendar, in that it has memorable poetic names for dates. One criticism of the named months, though, was that they’re very specific to Parisian seasons. It doesn’t snow very often in Marseille. And harvests take place at different times, depending on the latitude. If you included French colonies and territories in tropical climates, a month name like nivose (snowy) would seem almost insulting.

Week Names

Here are some example week names for my home town, New York City. They’re based on weather, holidays, annual traditions, and so on.

party
sidewalk trees
gray skies
endings
doldrums
snowy
paper hearts
presidents
windy
coyote
sleepy (Daylight Savings)
shamrock
bear
melting
showers
taxes
cherry blossoms
deer
flowers
park-picnics
mermaid
Coney
Rockaway
sandy feet
Cyclone
rooftop
barbecue
tank top
sweaty
sunscreen
Hell’s kitchen
Shakespeare
weddings
Battery Park
white clothes
labor
dark clothes
honey locust
pumpkin spice
actual pumpkins
flannel
snow, once
squirrels
scary
election
yellow gingko
thanks
foggy
dark
cold
tree-lighting
pine trees
jingles

Help me out in the comments. What would be your local week names?

Notes on My Online Teaching Methodology

2020-09-08

I’ve had a lot of questions lately about my online teaching methods lately, so I thought I’d share my rationale behind them. The course I’m currently teaching, Introduction to Computational Literary Analysis, in the Department of English and Comparative Literature at Columbia University, has a unique technological stack:

A course website. It’s built using Haskell and the static site generator Rib, and styled with Tufte.css, the CSS framework based on design concepts from Edward Tufte
A course readings platform made with Hypothes.is, open-source web annotation software
A course communications system made with Zulip, the open-source, threaded, text-based chat platform

Why use Hypothesis?

As literary scholars, we talk about texts. We are often engaged in what one might call marginalia: writing, about another text, with reference to a very specific, page- or line-level passage of that text. Very rarely does this kind of writing actually take place in the margins of books, any more, but it recalls a time when conversations would happen in the margins. In the days when books were expensive, luxury items that were passed around among a coterie of friends, this would happen more often. But as books became cheaper, and were loaned around less frequently, conversational annotation became more of a rarity. Furthermore, the recent shift to digital textual media made this even harder: without electronic margins, and electronic pens to write in them, we have had difficulty recreating the experience of social annotation. Our writing has become increasingly divorced from the writing it discusses, and it has become less social, and more solitary.

Mary Shelley’s Annotated Frankenstein, via the Morgan Library

This is where Web 2.0 technology comes in. Whereas Web 1.0 was static—you could read pages but not interact with them—Web 2.0 was dynamic: website visitors could interact with the pages, changing their content. Virtual “bulletin boards,” as the old metaphor went, became possible, and with them, virtual margins.

Hypothes.is is an open-source web annotation platform, designed to do just this. It allows for highlighting, commenting, and discussing a text, in the margins of the text itself. It is already in wide use for annotating news articles, scientific journal articles, and more, but its ideal use case, in my opinion, is for literary texts.

Hypothes.is-annotated edition of The Moonstone, one of our course texts.

Why Zulip?

Again, as literary scholars, teaching in literature classrooms, we talk about text. It only makes sense that we do that using text. Zulip is an open-source text-based chat service, akin to services like Slack, but open-source, and with a few added features, like threading. Threading allows for a non-linear, asynchronous way of sorting discussions by topic. It allows you to join a past discussion that has since moved on to a new topic, and to change the topic without derailing a previous conversation. This does the work, technologically, of a intructor/facilitator, whose labor consists so much of discussion moderating. “Let’s go back to that earlier point,” and “let’s not forget X” are common refrains in a classroom that shoehorns the multiple timelines of students’ thoughts into one linear narrative.

Why not just use Zoom?

Too often, we accept technological services as the necessary background of our lives, without considering their social and economic consequences. We want them to just work, and get out of the way, so that we can do our jobs. However, for literary critics, that is highly paradoxical–and hypocritical, even—since we are so conscious of the social implications of other phenomena. This is to say, if we write an article about latent colonial ideologies in Jane Austen’s Persuasion, but we do it in Microsoft Word, on a laptop running MacOS, which we bought on Amazon, there are economic implications for what we are doing, materially, that are analagous to what we criticize in our article. We’ve voted with our dollars in favor of giant corporations, to the detriment of community-based alternatives.

Economics aside, there are serious ethical issues behind using proprietary software instead of open-source software. To borrow an analogy from switching.software, using proprietary software is like eating a slice of cake at a diner: we can’t be sure what the ingredients really are, and we just have to trust that it’s going to be what we expect. Using open-source software, however, is more like eating a piece of homemade cake: we know the recipe, we trust the ingredients, and we can reproduce it ourselves, if we want. We can even modify the recipe to our liking, and make it better.

Now, we can’t make conscious choices for everything in our lives. That would be exhausting. But some choices are easy to make, and software choices are among those. After the pandemic hit, we all swarmed to Zoom without stopping to consider whether it’s a good idea. Our universities and libraries gave the for-profit Zoom corporation millions of dollars, and its stock price skyrocketed. Meanwhile, we have largely ignored the endless stream of articles that have reported on Zoom-related security problems, the cultural phenomenon of “Zoombombing,” and more.

In my course, I largely replace videoconferencing with text-based conferencing on Zulip. But for the times when we want to meet using video, I use Jitsi, an open-source alternative that’s simpler, nicer, and more ethical.

Why host videos on PeerTube?

University-based video hosting, like ours at Columbia, locks course videos behind a firewall, where they can only be accessed by other students within the university. This seems like it would be bad for the future life of the lectures: if students leave the university, for instance, and want to revisit the course material, they might no longer have access. The big video hosts, like YouTube, are even worse for course materials, since they’re ad-based. I won’t have my students watch ads just to get to my course material. So I opted to post my videos to us.tv, a PeerTube instance. PeerTube is an ad-free, open-source, decentralized video hosting federation, where instances that follow one another share videos between them, using a Bittorrent-like protocol. My lecture videos are thus accessible across a wide variety of servers. This is important when we consider the potential that students might be calling in from countries with censored or restricted Internet: if one server has been blocked, the videos are still accessible on others that follow it.

My Type-Safe Blog in Haskell

2020-09-13

Like most, my first step into the blogging world was through WordPress. But after a few years of wrangling plugins, handing endless updates, dealing with custom CSS issues, and paying for shared web hosting, the idea of using a static site generator seemed more attractive. Whereas WordPress sites are dynamically generated on a web server, static sites are just made up of good old HTML pages, and are therefore much simpler, more secure, and easier to maintain. Plus, you can serve those pages using a free service like GitHub pages. I switched to Jekyll.

But my first step into the world of static site generators quickly grew in complexity. Jekyll allows you to avoid writing HTML by writing in Markdown instead, which is a real improvement. And to avoid the tedium of writing CSS, you can write in Sass instead. Then there’s Coffeescript, which compiles to JavaScript. You can template together all your pages using the templating language Liquid. But for each of these conveniences, there is some technical overhead. First, all of these micro-languages are compiled using a stack of Ruby gems. To manage those gems, you need Bundler, which puts them all together, and probably also a manager which juggles all your versions of Ruby. And in the end, you have to remember lots of syntax: what does a variable look like in Sass, again? What about Coffeescript? Liquid?

For my second time around, I wanted to solve a problem with my CV, which was that I wanted to avoid repeating myself in situations where I had a publication related to a project that was listed in both the Projects section and the Publications section. So I decided to put all that data in a data format, YAML, and then display it using templates. To do all that, I chose the simplest tool I could find. Metalsmith bills itself as “an extremely simple, pluggable static site generator.” And it is. Everything in Metalsmith is a plugin, and so getting the perfect blog set up was just a matter of assembling the right plugins. I even wrote one myself, to handle parsing Markdown within Yaml data. But in the end, I found the JavaScript and node.js world to be really fickle. My dozen Metalsmith plugins had hundreds of dependencies, in total, and updating one would often break another. I kept getting Dependabot security alerts on GitHub for dependencies I didn’t even know I had. I was able to rewrite my CV as YAML data, though, and format it using the satisfyingly minimalist templating language Pug. But I was still really in the same situation as before.

So this most recent time I thought hard about semantic data. YAML had allowed me to write my CV like this:

projects:
  - title: Corpus-DB
    url: https://github.com/JonathanReeve/corpus-db
    github: JonathanReeve/corpus-db
    start: 2017-03
    description: A database and API for plain text archives, for digital humanities research.
    updates:
      - date: 2017-10
        type: award
        description: Winner, [2017 NYCDH Graduate Student Project Award](https://nycdh.org/groups/nycdh-announcements-71439400/forum/topic/2017-nycdh-graduate-student-project-award-recipients/)

This was a big step forward for my deduplication problem, since I could now write a template that could extract all publication types and display those in a different section, without having to maintain those in two different places in the data. There is an implicit schema here: every project has a title, a url, and so on. But there was no way to keep this from breaking in an unexpected way, since I just had to remember that my template expects there to be certain fields in the YAML data.

This is where Haskell comes in. Haskell allows me to define algebraic data types, like this:

data Project = Project {
  title :: Text,
  role :: ProjectRole,
  homepage :: URI,
  github :: Maybe Text,
  pypi :: Maybe Text,
  dateRange :: DateRange,
  desc :: Markdown,
  updates :: [ Update ]
  } deriving Show

data ProjectRole = Creator | CoCreator | Developer | Collaborator | ResearchAssistant deriving Show

What this says is: every project has a title, which is text, but the role I have in the project must come from a set list. Similarly, the URIs for the project must be URI types, which I can then validate and extract in predictable ways. This way, I can write an entry like this:


Project { title = "Corpus-DB",
          role = Creator,
          homepage = "https://github.com/JonathanReeve/corpus-db",
          github = Just "JonathanReeve/corpus-db",
          dateRange = DateRange (date 2017 03) Present,
          desc  = "A database and API for plain text archives, for digital humanities research.",
          updates = [
            Update (date 2020 08) (Publication Abstract
              "Corpus-DB: a Scriptable Textual Corpus Database for Cultural Analytics"
              "https://dh2020.adho.org/wp-content/uploads/2020/07/604_CorpusDBaScriptableTextualCorpusDatabaseforCulturalAnalytics.html"
              (Venue "Digital Humanities 2020"
                     "https://dh2020.adho.org/"
                     "Ottawa, CA")),

At first glance, this may seem like an abuse of a programming language. Programming languages are for logic, the argument goes, and configuration belongs to configuration files, in formats like JSON, TOML, or YAML. We must separate logic and data, right? Actually, it turns out there are tons of advantages to writing data in Haskell.

For one, it is compile-time type safe, meaning that my text editor actually checks that this compiles, as I’m writing it. If it were just plain text in YAML, there’s nothing stopping a date from being something else entirely, like foo. But in Haskell, if I write a date that isn’t two integers, (a year followed by a month), it’ll raise an error immediately.

Another advantage is composability. If I find myself repeating a given entry, I can just abstract it away into its own function. For instance, I have a Venue entry that looks like this:

cuEng = Venue "Department of English and Comparative Literature" "https://english.columbia.edu/" (uni "cu")

And whenever I want to use that, I can just call venue = cuEng, and save myself some typing. That also means that, should the name or URL of any of these entries change, I can change them in one place, without having to change them in seven.

In addition to making everything type-safe, though, I greatly reduce the technologies involved. I’m still writing posts in Markdown, but have collapsed all the other templating languages to just Haskell. The HTML, CSS, and data, instead of being written in Liquid, Sass, YAML, and so forth, are all just in Haskell, which allows me to use the full power of the programming language to do whatever I need it to. For instance, if I’m doing something repeatedly, like adding a bunch of scripts:

<script src="script1.js"/>
<script src="script2.js"/>
<script src="script3.js"/>
<script src="script4.js"/>

I can write that in Haskell like this:

script n = script_ [src_ "script" ++ n ++ ".js"]
map script [1..4]

And it’s all validated at compile time, so when I press save in my text editor, it checks to make sure everything works. Because Haskell is a purely functional language, there are no runtime exceptions.

All of this I achieve using the wonderful static site generator Rib, which built on the Haskell build tool Shake. So if there’s anything else I need to do, which Rib doesn’t provide out of the box, I don’t need a plugin for it; I can just write a Shake action.

There’s still some complexity left over, though, in that I still have to manage Haskell packages. But with Nix that’s trivial. Nix is great at making reproducible builds possible.

Browse the source code for this site here.

My Notetaking System

2020-12-23

Here’s what my notetaking system currently looks like:

Network visualization of my notes, via Org-roam-server

And here’s how I got there. In my 10+ years as a graduate student, at three different institutions, I’ve tried so many notetaking systems. I started on paper, with the Cornell Note methodology.. Then, once I got a laptop, I moved to web-based utilities, like Evernote. After one of these, Springpad, bit the dust as a company, I realized I needed my own solution for notes that didn’t rely on an external web service. I quickly fell in love with Tomboy Notes, which is like a superpowered Sticky Note system. The best feature of Tomboy, along with its later clone, Gnote, was that if you were writing one note, and happened to type the name of another note, it would automatically link to that other note. This meant you could have small, stickynote-sized notes and have them all relate to each other.

This was amazingly empowering, because the notes I was taking were really all interconnected. Especially as a student of early 20th Century British literature: so many of the canonical figures of the field knew each other, published each other, and influenced each other. But Tomboy had its limitations, and really started to choke with a larger amount of notes, edits, and syncs with its server. Here again, I was in a similar situation as before: I had developed a huge note database in software that wasn’t totally reliable.

So I switched to plain-text notes. There’s not much that can go wrong when you’re just editing a plain text file with a text editor. At that same time, I was also learning Vim, the modal text editor, and suddenly writing in any other program felt clunky and inefficient in comparison. So I moved all my notes into plain text. Then I discovered vim-notes, the wonderful Tomboy-like system. But vim-notes used its own brand of markup, which wasn’t quite Markdown, and wasn’t quite any other format, either, which made export and compatibility with external programs a little unweildy. Around that time, I heard about Org-mode.

I say this without exaggeration: org-mode is the best organizational software. It’s a personal information manager, notetaking system, calendar, time tracker, and much more. There’s really nothing like it. If you’ve never heard of it, I’d recommend watching one of the many introductory videos out there. Org-mode was the reason I switched from Vim to Emacs (although I got to keep my familiar keybindings, via Spacemacs and eventually Doom Emacs).

Soon, I discovered that there are a lot of extensions to Org, and some of them help with academic notetaking. One of them, org-roam, emulates Zettelkasten-style notetaking programs like Roam Research. In short, it extends org-mode, by adding backlinks, tags, and search functionality to one’s collection of org notes. It’s a killer app, when considered as part of the greater Emacs ecosystem. The emacs package Org-ref handles bibliographic management. Org-noter handles annotating PDFs and ePUBs, and Org-roam-bibtex brings it all together. Then, Org-roam-server generates a graph, visualizing the links between the notes.

With this system, I’m able to get this workflow:

Find a paper I want to annotate, somewhere online. Copy its DOI or Arxiv id.
Run org-ref’s doi-insert-bibtex to automatically grab its bibtex and insert it into my bibliography. Or run arxiv-get-pdf-add-bibtex-entry to add its bibtex and get its PDF all in one step.
Run org-ref-open-bibtex-notes to automatically generate a new note file from the bibtex entry.
Open the associated PDF, now downloaded and renamed in my papers directory, with org-noter.
Cite the paper in formal writing, using helm-bibtex. This then automatically adds it to the bibliography of the paper I’m writing.

See also part II in this series: Notetaking in Semantic Triples.

Notetaking In Semantic Triples

2021-05-14

This is a continuation of my previous post, describing my notetaking system.

I keep a Zettelkasten using org-roam. Notes are connected using links, which are actually just regular org-mode links. Those links look like this behind the scenes: [[URL][link title]]. Thus, there is an implicit relation between two notes where, if a link to note B appears in note A, you could express that relationship as a subject-verb-object triple, like this:

<Note A> <links to> <Note B>

Which is fine, for most purposes. But what if the relation between the two notes is more specific? For instance, if I have a note for T. S. Eliot, the poet, and a note for his poem, “The Waste Land,” the relation is really more like this:

<T.S. Eliot> <wrote> <The Waste Land>

So how can one achieve this? The linked notetaking strategy of org-roam and family gets us most of the way there. My note ts-eliot.org looks like this:

#+title: T. S. Eliot
Wrote [[The Waste Land]]

Now all we have to do is to make the verb into a note.

#+title: wrote
#+roam_tags: verb

For when a writer writes a creative work.
Example: T.S. Eliot wrote The Waste Land

So now we can write:

#+title: T. S. Eliot

[[Wrote]] [[The Waste Land]]

And we can extend that format for multiple objects:

#+title: T. S. Eliot

[[Wrote]]:
 - [[The Waste Land]]
 - [[Four Quartets]]

This in itself isn’t very useful yet, but now we have a structure that we can parse. In a separate parsing script, I can now write: if a line begins with a link to a verb, and is followed by a link to another (non-verb) note, that constitues a triple, which is then parsed as <T.S. Eliot> <wrote> <The Waste Land>. If a line begins with a verb link and a colon, and it then followed by a list of links, that is then parsed into: <T.S. Eliot> <wrote> <The Waste Land>. <T.S. Eliot> <wrote> <Four Quartets>., i.e., two triples.

At this point, I can now add more metadata to the note, which will link out to the greater semantic web—to Wikidata, for instance. “The Waste Land” note can now contain its Wikidata entity identifier, like this:

#+title: The Waste Land
#+wikidata: https://www.wikidata.org/wiki/Q581458

And Wikidata maintains a huge amount of data about the poem: its first date of publication, its first line, and even its Project Gutenberg ID, from which you can derive the full text of the poem. For many texts, Wikidata also maintains Goodreads identifiers, as well, which allows one to then derive a number of opinions about the text, as well.

You can imagine that this streamlines many aspects of research. I can now use SPARQL queries to ask complex questions like:

How long, in words, was the average line of poetry written in 1922?
How many total books were written by H.G. Wells’s lovers?
What is the distribution of literary genres for the books written by T.S. Eliot’s friends?

I can also feed all my note triples into one of the many Linked Open Data visualization and manipulation platforms.

However, it need be said that this is a little kludgey: it’s hack on top of a hack. So really, for this to become viable at all, something like this should really be integrated into org-roam. Or become its own package. (BRB, learning Elisp.)

Rethinking the MLA Style Research Paper

2021-05-16

In my last post, I argued that PDFs, Word documents, and similar electronic file formats are stuck trying to mimic paper, and that we should consider replacing these with screen-first file formats which liberate them from the constraints of pagination. So where does that put the famous MLA-style research paper, the standard format for English departments nationwide? Here, I argue for a new style for student papers in the humanities.

As a student, instructor, and developer, I’m unusually intimate with the MLA-style paper. I’ve written dozens of these papers, for university seminars, across fifteen years, and at three universities. As an instructor, I’ve taught the MLA style for almost ten years, at five universities, and have graded thousands of pages of these things. And as a web developer at the MLA, I’ve mediated between the MLA’s electronic publishing conventions and the technological constraints of the World Wide Web. I’ve wrestled at length with formatting these papers using Google Docs, Libreoffice, and LaTeX, and I even authored the md2mla script, which compiles Markdown to an MLA-style paper. But don’t get me wrong—I haven’t hated all of it. There are some things to like about the style. It’s simple, cleanly designed, and very legible. Here’s what the top of the first page usually looks like:

Yet there are problems with it, too. One of the first things one learns about the MLA-style research paper is that it should have one-inch margins. This is already a red flag, because it presupposes a paginated medium, for an 8.5 by 11 inch (American letter) page, which uses the Imperial measurement system. Since the only countries that use the ancient imperial system are the US and Liberia, this choice is unfriendly to international uses. (Try printing out an American-authored paper on any printer outside the US.) This choice is also unfriendly to international students, who now have to learn a new system. And of course, the MLA-style paper also suffers from the same limitations as the file format which frames it.

So the time has come to rethink the MLA-style paper. What would be the necessary features of such a style? It should be:

Easy.
Free.
Mistake-proof.
Clean-looking.
Semantic.
Widely interoperable.

I’ll discuss ways to achieve all of these goals. First, it should be remarkably easy. Students shouldn’t have to learn how to be programmers to use it, or typesetters. It should get out of the way, so that students can focus on writing.

Next, it should be free to write and read. That is, students shouldn’t have to buy expensive software, like Microsoft Word or Adobe Acrobat, in order to create it. Everything should run on open-source software.

Then, it should be designed to minimize the possibility of student error. It’s easy to mistake the four fields at the top of the current MLA paper. Is it the professor’s name, and then the course title? Or vice-versa? Adding labels to each of these fields greatly reduces mistakes like this, and shipping a stylesheet eliminates design mistakes.

The resulting paper should be at least as clean-looking as the current format. This means that it should have a stable stylesheet, with whitespace, typography, line spacing, and all other details already decided. That’s to say, students should not have to learn the intricacies of LaTeX page placement, of hand-coding CSS, or worse—of pushing around tiny slide-rule-like graphical ruler widgets in a word processing program, just to emulate hanging indentation. Everything should just work.

It should, of course, contain all the metadata you usually find in a regular MLA-style paper: the student’s name, the course name, the course instructor, and the date of composition. But that metadata should be formatted in a semantic way, both to minimize student error, and to enhance the document’s usability.

The file format should be one that can easily be converted to a commonly-used legacy format, like PDF or Word, in case the student has ambitions to publish the paper at a journal that only accepts legacy formats. But its native format should be HTML, targeting web publishing.

Finally, it should be neither the student’s nor the instructor’s responsibility to handle formatting the bibliography. These are menial tasks better suited to a computer. Thus, the bibliography should be automatically generated, from bibliographic metadata.

Thus, here’s my first attempt at a new format:

A plain-text document,
with an .md file extension,
written with Commonmark markup, which has:
- a YAML metadata header with these fields:
  - Essay title
  - Student’s name
  - Course name, if the paper was written for a course
  - Instructor’s name, if applicable
  - A bibliography in CSL YAML, included in the metadata header
- citations formatted using Pandoc’s citation style
- any equations included in LaTeX, enclosed in $$
- any images included as hyperlinks

The student wouldn’t need to “know” YAML, Commonmark, or Markdown—a web interface may be used to generate all of this. Then, upon submission, the file may be validated, ensuring that all the required fields are included. That can even happen before submission.

Here’s a minimal example:

---
title: My Amazing Research Paper
author: Jonathan Reeve
course: Introduction to Essay Writing
instructor: Professor Foobius
---

Here is my first paragraph.

The file may then be compiled to HTML (preferred), Docx, or PDF, using Pandoc and a template. Citations may be automatically verified, using the bibliographic data, and are automatically formatted as Chicago notes-and-bibliography style. Instructors may even be able to data-mine their students’ papers for the most frequently cited papers. The possibilities that plain-text submissions grant to instructors is limitless.

The compiled HTML file uses Tufte CSS by default, and therefore uses sidenotes instead of footnotes. A Works Cited page is thus unnecessary.

I’ve created a template repository for this style of research paper, along with a script to convert it to HTML.

The Case Against PDFs

2021-05-15

We should stop pretending that we’re all still using paper. At least, speaking for myself and my academic work, email is much more common than paper mail, ebooks are rapidly growing in popularity, and journals are mostly accessed electronically. The global pandemic has driven us even further in digitization. More and more, we sign contracts, pay our electrical bills, and even get our drivers’ licenses renewed, electronically, over the Internet. So then why do we insist on using electronic file formats that mimic the constraints of physical paper, which we no longer use?

Think about this: when you start writing a new document, using a word processor, you will usually see something like this:

Google Docs: Paper UI

This UI emulates a paper page. Not just any page, but the 8.5 x 11 inch, letter-size paper common in the US. This made sense, at one point, when the printer was an essential part of the computer, and when computer documents were only designs for their paper versions. But now, in 2021, it is starting to go the way of the floppy disk save icon: it will become a vestigial organ of computer interaction. But not before costing us years of bad document interface design.

Case in point: I have to fill out some forms (“paperwork”) with my department, in order to get on the payroll. The department office doesn’t have a row of filing cabinets. They’re not printing out these forms. Yet the form is composed in Microsoft Word, using a string of underscores to indicate the fields I’m meant to fill in: Name: _______. So I open the Word file on my computer, and type my name among the underscores: Name: ____Jonathan Reeve____. Presumably, some unfortunate intern then has to receive my form—a Word document attached to an email—and copy-paste, or retype, each field into some spreadsheet. This is an ugly and unnecessary workflow for 2021. Why do we insist on trying to mimic early 20th Century paper-based offices?

It’s not that we don’t have better technologies, ones better suited to our tasks. HTML, for instance, has had form capabilities since the ’90s, and the <form> tag set remains easy to write, even for novices. For those with an aversion to code, there are plenty of free commercial options, like Google Forms, Typeform, and others. All of these solutions take user input and insert it directly into a database or spreadsheet. No word processing necessary. And you don’t have to decide on a page size, either, for your nonexistent pages.

Here’s another example: I’m writing my dissertation. The Graduate School’s submission process is totally electronic, so there will be no printed copies of my dissertation, unless someone chooses to print it. But it must still be submitted in either PDF or Word format. There must be page numbers at the bottom of every virtual “page,” and a table of contents. One-inch margins around each “page.” This probably doesn’t sound like a problem, to many of you reading this—aren’t these margins just the same as margins on a webpage?—but I’ll explain. There are a number of fundamental problems with PDFs.

First, pagination is not only unnecessary in electronic documents, but detracts from their usability. Indices and tables of contents, for instance, are inefficient, and are much better handled with hyperlinks. Pagination also makes figures and images appear far from where they’re described: a reader shouldn’t have to flip through several pages to find a figure referenced on another page. Finally, the reader, rather than the writer, should have control over the margins, text width, line spacing, and so on. The fact that our libraries are filled with large-print editions of books is a testament to the need for variable font sizes. We can customize our preferred font sizes dynamically in most web browsers, with Ctrl-+ or equivalent, but only a raw zoom is possible with PDFs—the font sizes cannot be changed.

Second, PDF file bloat is ridiculous. As Dennis Tenen has shown, in Plain Text, try making a PDF with only the words “hello world,” and then try doing the same in plain text, and compare the file sizes of each. The PDF takes up two thousand times more space than its plain text equivalent. And before you dismiss this as a problem mitigated by modern computers, capable of storing many gigabytes of data, consider that there is a real-world cost to computational inefficiency, when scaled to the level of a nation or the world. When millions of our computers are working to compute the displays of PDF files of many megabytes each, this adds up to significant electricity usage, and thus carbon emissions.

PDFs also lack many of the features of modern electronic documents. Interactive charts, diagrams, and so on, have been possible since the early days of JavaScript, but remain impossible with PDFs. One of my favorite features of the modern web, social annotation with the likes of hypothes.is, is also impossible with PDFs. For academic publishing, especially, consider that citations and bibliographic references are kinds of hyperlinks, and hyperlinks work best in web browsers, where they are a native technology. Sure, you can put links in a PDF, and it will open a browser, but this is not always a very seamless experience. Sadly, the fact remains that the vast majority of PDF documents published today have textual, rather than hypertextual, bibliographic references. Why would anyone want to make it more difficult to look up the papers one cites? Actually, nevermind, don’t answer that.

Now, we all know someone—that one colleague that insists on paper books, paper journals, who even prints out emails. And they’re not wrong—research shows that we remember more of what we read when we read it on paper. But this alone is not enough to reverse the digitization trend. You might love the feeling of pen on paper, and prefer to take notes by hand, but that’s not going to convince all your coworkers to give up their email, in favor of writing letters. So we need to embrace computers as communication devices in their own right, and divorce ourselves of the outdated idea that they’re only fancy typewriters, meant for creating paper documents.

Well, how do we do that? The next time you make a document, make it in Markdown, Org, HTML, or plain text. Make it using Word, if you must, but save it HTML. That’s really the universal format: anyone that has a computer, these days, has a web browser.

Next, I’ll describe rethinking the MLA-style research essay.

Best Practices for Dissertations in the 21st Century

2021-10-25

There’s a lot of talk these days about “next-generation dissertations” and “digital dissertations.” See, for example, Electronic Theses and Dissertations, The Sage Handbook of Digital Dissertations and Theses, and Shaping the Digital Dissertation: Knowledge Production in the Arts and Humanities (Fox; Andrews; Kuhn and Finger). Syracuse University and Katina Rogers recently launched an NEH-funded project, called Next Generation Dissertations, where they define the genre as: “any doctoral project whose form goes beyond the traditional written monograph. It can be a website or other digital product, it can be a graphic novel, a documentary, or a rap album.”

One often-cited example is Amanda Visconti’s dissertation, which involved the creation of an interactive website called Infinite Ulysses—an annotated and annotatable edition of James Joyce’s Ulysses. (Which I love, by the way—I’m trying to do something similar with my Open Editions edition of Ulysses.) I’d often heard this called a “digital dissertation,” since much of the content of the dissertation is behind the scenes, in PHP and Drupal.

But I would like to problematize the term “digital dissertation.” Practically speaking, aren’t all dissertations digital, these days? Virtually no grad student, in the US at least, is depositing a handwritten dissertation, which is never emailed to anyone, or uploaded to the web. Even if it were handwritten, it would still need to be archived digitally.

I would also like to problematize the idea that a dissertation that is a website is in some way nontraditional. If a traditional, monographic dissertation is uploaded to a digital repository, and is available for scholars to read over the web, is that not, then, a website? At the very least, it’s a digital product, no?

If doctoral candidates are writing dissertations that will never be printed—which is very common these days, since libraries and departments are making fewer and fewer hard copies of dissertations—then what’s the use of the paper-shaped digital document? I’ve written about this in a previous post, already, where I argue that almost any format is better than PDF and Word, when it comes to born-digital documents that stay digital. For one, if you’re making endnotes, or even footnotes, in a document, in 2022, you’re purposefully making it harder on your readers, since they have to navigate back and forth in a very long document to try to find your notes.

So I wanted to propose a set of best practices, for 21st century dissertations. This doesn’t apply to documentary films or rap albums—as much—but rather is a way to make sane defaults for traditional, monographic dissertations. These recommendations have been informed by a number of inputs: Next-Generation Dissertations’ Project Examples; the values framework of the Humane Metrics Initiative; the the MLA’s Guidelines for Evaluating Work in Digital Humanities and Digital Media, their Guidelines for Authors of Digital Resources; and the research guide for dissertations and theses at the CUNY Graduate Center.

I list these recommendations in order of importance. At the bottom of the page, I also provide a template which doctoral students can use to bootstrap their own dissertation.

1. It should be always-already public

The dissertation should be always-already public. Not made public when it’s finished. Not made public a year or more after it’s been deposited (“embargoed”). Public from the very beginning. Why? Scholarship isn’t performed in a vacuum. We need comments, ideas, help, cross-pollination. This is seemingly a given in the sciences, where dissertations are often assembled from a number of already published papers, but is less common in the humanities. Part of reason for that, I’ve gathered from talking to colleagues, that is there is a fear that, if your dissertation is public, it will somehow be “stolen”: that a struggling scholar will take your work, slap their name on it, and make a career out of it. It’s a ridiculous fear, the second you examine it. For one, accounts of this happening are rare and anecdotal, and don’t represent anything statistically likely. For another, making your dissertation public is actually a way to protect it against theft. If there is ever any question as to who wrote what, it’s essential to have a verifiable public record of it, already online.

See an excellent response to fears about a public dissertation in Kathleen Fitzpatrick’s “Dissertating in Public” ((Fitzpatrick)).

The best way to make your work public is to push it to a public version-control hub like GitHub or GitLab. That leads me to the second recommendation.

2. It should be always-already version-controlled

The best way to make a public record of your dissertation is to put it under version control from the very beginning. Version control systems like Git keep track of your dissertation in all its revisions and states, across the history of its composition. They can include not only your prose, but your data, code, and everything else.

This is not only a great way of backing up your work, but it’s a great method for collaboration and project management. Once you sync your work to a Git-based cloud service like GitHub or GitLab, you’ll have access to bug trackers, wikis, kanban boards, and automated build systems (CI/CD).

My recommendation for this is to check in your work using a version control system like Git, and push it regularly to a cloud service like GitHub or GitLab.

3. It should be licensed under a copyleft license

Anyone should be able to use your work, download a copy of it to their computer, and share it with others, provided that they credit you, and promise to pass on those same freedoms to others. This is a foundational practice for open knowledge creation, and it’s especially necessary if your work includes something reusable, like computer programs. Again, new knowledge isn’t created in a vacuum—it must be available to others. I prefer the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License, or CC BY-NC-SA 4.0, for prose, and the GNU Public License, version 3, for code.

4. It should be formatted in HTML by default

HTML is the most future-proof format, apart from plain text. Since it’s the backbone of the whole Internet, it’s the most widely-supported format, and the one which is the most likely to outlast them all.

If your university requires PDF, however, you can easily “print” a webpage to PDF just by using the print function in the browser. If it doesn’t look great, a print stylesheet in CSS goes a long way. But if you do print it, it will lose a lot of functionality, so print stylesheets should expose warnings which point readers to the canonical version of the document, on the web.

Your dissertation doesn’t need to be written in HTML, however. Plenty of plain-text formats exist which compile to HTML: markdown, org (which is what my dissertation is written in), and Asciidoc, just to name a few. There are also more esoteric ones like Pollen which are exciting departures.

5. It should be easy on your reader

At minimum, it should meet accessibility standards. Using alt tags on images is a good first step, and aids those using screen-readers to read your work.

Next, it should use sidenotes rather than footnotes or endnotes: notes in the margins, rather than notes at the bottom of the page. Your readers shouldn’t have to flip to a different page, or even a different part of the page, to read your footnotes. My template uses the wonderful Tufte.css to accomplish this.

In-text citations should be hyperlinks for, where possible. Readers shouldn’t have to manually jump back and forth between an in-text citation and a bibliography.

In the bibliography, references should also contain, or be, hyperlinks, where possible. If you found a source online, your readers should be able to visit that same source. Using DOIs in your bibliography is a good idea, since these are stable URLs.

I should also note that URLs are not hyperlinks. Don’t muddy your references with long URLs, which were never meant to be used as link text.

6. It should be easy to annotate

Without the ability to annotate a document, it becomes monolithic in an undesirable way—you remove yourself from any possible conversation that could help your arguments.

Thus, I recommend adding an annotation layer to your dissertation, to allow for the free exchange of ideas. I recommend Hypothes.is. You can add a hypothes.is layer to your HTML just by dropping in a single line of code to the header.

7. It should be machine-readable

Humans aren’t the only ones who need to be able to read your dissertation. To make it available to search engines, databases, and other collections, you should have should have good, standards-compliant, machine-readable metadata.

I recommend using semantic web standards, like those used in schema.org. Schema.org provides a Thesis class which can describe a dissertation. Make sure that this metadata appears in <meta> tags in your HTML output.

Your dissertation should also include a machine-readable bibliographic file, so that software and services that track citations can read this file, without having to parse your references section.

8. It should be reproducible

If your work involves data or code, this should be included in the repository. This ensures that your work is reproducible.

The software environment you use should be declared somewhere, as well. This means, for example, what version of jupyter you are using. For your work to be reproducible, all this information should be declared somewhere. For Python, pipenv and poetry have lockfiles which track the versions of the software you’re using. Just include those files in your repository. Even if you’re not coding, but just using software like pandoc, make sure it’s declared in your environment somewhere.

I use Nix for this, and declare all the software I’m using, and the versions, in a shell.nix. Even beyond experimental reproducibility, a nice side effect of declaring your environment is that you can distill your whole build process into one command. So, converting all your source files to HTML, optimizing all your images, and everything else, can just be done all in one go. Then that process can happen automatically in CI/CD (continuous integration / continuous deployment)—you can set it up so that GitHub Actions, or GitLab CI builds your dissertation on each commit, so you don’t have to.

9. It should be archive-ready

Your dissertation should be archived at a future-proof document repository. Many universities already have such a digital repository.

If yours doesn’t, I prefer Zenodo, since they accept Git repositories, and provide a DOI, which may represent the state of your dissertation at that time.

Template

Here is a template you can use. I’ve incorporated almost all of these recommendations, so far. Its features include:

Automatic generation of your bibliography, in whatever bibliographic format you want, using Pandoc’s Citeproc. You should never have to write these things out by hand.
Annotations using Hypothes.is.
Support for a powerful markdown derivative, Pandoc’s markdown, with tons of features useful to scholarly writing.
Layout in Tufte.css for beautiful typography.
Sidenotes by default, rather than footnotes or endnotes. (See Tufte for more on this.)
Support for figures and images, automatically numbered sequentially, with captions in the margins.
Support for LaTeX-style math and MathML, via MathJax.
Modern, standards-compliant CSS and HTML. 9. Excellent, machine-readable metadata for the semantic web, using Schema.org.
GitHub Actions and GitHub Pages integration, for automatic builds and deploys. Serve to the web at no cost to you.

References

Andrews, Richard. The Sage Handbook of Digital Dissertations and Theses. Sage Publication, 2012.

Fitzpatrick, Kathleen. “Dissertating in Public.” Shaping the Digital Dissertation: Knowledge Production in the Arts and Humanities, edited by V. Kuhn and A. Finger, Open Book Publishers, 2021, https://books.google.com/books?id=VY4tEAAAQBAJ.

Fox, Edward. Electronic Theses and Dissertations : A Sourcebook for Educators, Students, and Librarians. Marcel Dekker, 2004.

Kuhn, V., and A. Finger. Shaping the Digital Dissertation: Knowledge Production in the Arts and Humanities. Open Book Publishers, 2021, https://books.google.com/books?id=VY4tEAAAQBAJ.

Books with Question Titles

2023-06-14

I’ve been playing around with some of the Open Library book data, and made an interesting mistake along the way. I incorrectly constructed a regular expression, using a literal question mark instead of an escaped question mark, and ended up with a list of book titles which end in question marks. This was a happy accident, since the results are interesting, and often hilarious, book titles:

Isn’t that Lewis Carroll?
How Much Wine Will Fix My Broken Heart?
You’re the Professor, What Next?
What Counts as Mathematics?
Too Many Babies?
What do we know about Namibia’s competitiveness?
Enfants bandits?
Why Is That So Funny?
Do our people believe what our preachers preach?
Can we afford to waste municipal waste?
Mission accomplished?
IF YOU SAY MULTICULTURALISM IS THE WRONG ANSWER, THEN WHAT WAS THE QUESTION YOU ASKED? [Emphasis in the original.]
Is the key to successful breastfeeding all in the mind?
Why Grow Old?
We need to talk, but first, do you like my shoes?
But what if I don’t want to go to college?
How come I feel so disconnected if this is such a user-friendly world?
How does a czar eat potatoes?
What Do Cats Do for Halloween?
Did You Want To Talk To The Doctor In Charge Or The Nurse Who Knows What’s Going On?
Forty men are going to form the largest poultry farm in Canada, will you be one of them?
Is that lunch adequate?
What Color Is My Internet?
What’s Funny About Yorkshire?
Could You Please, Please Stop Singing?
Dinosaurs or dynamos?
A concise answer to the inquiry, who or what are the Shakers?
What’s With Modern Art?
Le Moyen Age pour quoi faire?
Elvis, is that you?
Mama amasa la masa?
Texas?
How Big Is Baby?
Where are the footnotes?
Your phone bill, fact or fiction?
What Is Western about the West?
I’m a Scientist Scary Isn’t It?
What’s so funny about computers?

Some patterns

I liked the first one, with the “what next?” tag question, so I looked for titles that end with that pattern, and found some 134 such titles, including these good ones:

Drama - what next?
Northern Ireland, What Next?
Helsinki, what next?
After age 16 - what next?

The last one, too, has an interesting form, so I searched for books beginning with “What’s so funny about.” There were 21, mostly about religion:

What’s so funny about faith?
What’s So Funny about God?
What’s So Funny about Being Catholic?
What’s so funny about microbiology?

Since there are several book titles which are addressed to a person, like “Elvis, is that you?” and “Jesus, is your Daddy mean?”, I tried searching for titles starting with “Jesus,” and there are 31. Here are a few:

Jesus, what’s for lunch?
Jesus, Were You Little?
Jesus, where are you taking us?
Jesus, Who Needs Him?

The books that begin with “Elivis,” and end with a question mark, are fewer:

Elvis, is that you?
Elvis, what happened?

Books that begin with “how come” mostly sound like children’s books. (“How come” feels like children’s language to me, for some reason. There are some interesting explanations on the English Stackexchange, and Garner says to “avoid it in serious writing,” understandably.)

How Come It’s Windy?
How come you’re not married?
How come holding hands feels so good?
How come elephants?
How come they always had the battles in national parks?
How come I’ve never seen a can of broccoli?

This makes for a funny tone in books that sound sound like they’re written for adults:

How Come Every Time I Get Stabbed in the Back My Fingerprints Are on the Knife?
How Come That Idiot’s Rich and I’m Not?

There are unfortunately no other books beginning “How Much Wine Will Fix…”

If you want to replicate this experiment, just grab the “works” dump from the Open Library data dumps site, and then run zgrep -io '"title": "Jesus, .*?"' ol_dump_works_latest.txt.gz or equivalent.

My Writing Workflow

2025-06-31

I thought I’d say a few things about my writing workflow, since I often get questions about my setup, and since there’s a blog carnival happening this month on the topic.

I write everything (books, dissertations, articles, blog posts, emails, code) using Emacs, the venerable old text editor which has been in active development since the 1970s. I write using an Emacs mode called Org-mode, which itself has been around since 2003. I arrived at Emacs and Org-mode after trying everything else, since they’re the only tools I’ve found which can handle the whole writing pipeline: reading, notetaking, writing, and publication, and can do it well. This is an unusual choice, which deserves some explanation. But before I describe the software stack and discuss all the technical details, I’ll first say a few words about my writing process.

Overview and Methodology

Writing starts with reading. Reading generates notes, which then become writing, and finally publication. I wanted a way to make this process as frictionless as possible. A great methodology for that is the Zettelkasten method, explained perhaps the mostly eloquently in Sönke Ahrens’s book, How to Take Smart Notes. In a nutshell, it’s about keeping short, atomic notes, and ensuring that they’re deeply interconnected. Notes are either fleeting notes, concept notes, or literature notes. New thoughts become fleeting notes, which are eventually reworked into a concept notes, (an evergreen note which is often being revised), which then become writing. Every book or article has a literature note, too, which keeps track of the bibliographic information, so that it’s easy to cite sources later. The genius of this system is that daily tasks as simple as keeping a commonplace book can eventually become articles, or even books, without much additional effort.

Emacs can function as a great implementation of this Zettelkasten system, using a number of plugins which work together to streamline the writing process: org-noter for reading, org-roam for notetaking, org-mode for writing, and pandoc for publication. But if Emacs seems like a strange choice, allow me to explain.

Why Emacs?

As a grad student in English Literature, I didn’t know too many people using Emacs for writing. There are so many shiny tools for writing these days, and Emacs is very much not shiny. But it’s the best tool for the job, by far.

I arrived at Emacs via Vim, an equally old and venerable text editor, famous for it modal editing style. If you’ve never written with a modal text editor, you’re missing out. It gives your keyboard two modes: one for entering text, and one for editing it. This is effectively an editing superpower, since all the pointing-and-clicking, all the selecting text, and everything else annoying about writing can be replaced with a couple of keystrokes. Want to delete three words? Type d3w, short for “delete three words.” Want to delete a paragraph? Type dap for “delete a paragraph.” Change a double-quoted sentence to a single-quoted one? Type cs'" for “change surrounding single-quotes to double-quotes.”

Once I got used to Vim, I wanted to use it to write everything. And I did, until I discovered Emacs. Emacs has modal editing functionality (cheekily called evil-mode) but can extend it to anything else in the Emacs ecosystem. Like Vim, it’s old (read: future-proof), open-source (read: free as in freedom), and extensible.

I write more prose than I do code, but it turns out a coder’s text editor is what I needed to write prose.

But the killer feature of Emacs is its built-in mode, Org-mode.

Org-mode

It’s hard to describe just how powerful org-mode is. It’s a lightweight markup language, outliner, calendar, organizational system, todo list tracker, notetaking tool, literate programming utility, and a lot more. It’s built into Emacs, but there is a growing number of Org-compatible programs available for other platforms now.

You can think of Org like a supercharged Markdown, but with an entire ecosystem of plugins built around it. Whereas Markdown has a minimal core by definition, and has led to a huge number of Markdown-based markup languages (CommonMark, GitHub Flavored Markdown, and MyST Markdown, just to name a few), Org-mode is comparatively stable, standardized, and has grown along with its primary editor, Emacs, ensuring a tight compatibility between the two. I write just about everything in org-mode files, including emails, scripts, literate Python notebooks, and this blog post.

Discovery and Capture: Org-capture and Org-protocol

If I come across something interesting on the web, I’ll capture it using org-protocol and org-capture. I have key combinations set up for my preferred web browser, Qutebrowser, which send the website URL, the website title, and any text I’ve selected to org-capture, which then can allow me to insert that link and text to any existing note (via org-refile) or to a new note. In this way, I’m basically using org-mode as a bookmark management utility.

Reading: Org-noter

Once I collect reading material, in the form of papers or books, .pdfs or .epubs, I usually read it with org-noter. Org-noter provides a two-pane interface for reading, with the text on the left and my Org-mode notes on the right. The notes stay in sync with the text, such that if I turn the page, the notes also jump to the appropriate page. I used to annotate PDFs and ePUBs by using PDF annotation programs, but these didn’t always make my annotations searchable.

If I read something which relates to a note I’ve already taken, I’ll simply type the title of the note, and org-roam will insert a link to it, ensuring that I’ll be able to find my literature notes from a concept note, and vice-versa.

Notetaking: Org-roam

Org-roam is a clone of Roam Research, but made for org-mode. It enables Zettlekasten-like notetaking from within org-mode. I like it because it allows me to do graphical notetaking, where notes are part of a decentralized network, and also hierarchical notetaking, where notes are part of a hierarchy. For example, I have a file movies.org where I keep track of movies I’ve seen, along with ratings I’ve given them and links to their IMDB pages. That enables me to disambiguate between books and movies with the same title, without resorting to the parentheses-based namespacing of the kind used by Wikipedia.

Org-roam also works well with org-mode citation tools like citar, using citar-org-roam.

I can even visualize my whole Org-roam graph, now at over 3K notes, using org-roam-ui.

Publishing: Pandoc

Pandoc is an amazing program for converting between lots of different file formats, including Org files. With a little finessing, it’s a very powerful system for publishing. It can generate bibliographies from citations, Bibtex files, and .CSL citation styles. It can handle numbering your figures and equations, adding captions, dealing with special blocks of text, and more.

My dissertation, for example, uses a custom Pandoc template along with build instructions in a Shakefile, for the Shake build system, to convert .org files to the HTML files that I then published to the web and the university archives. I wrote about this here, in the dissertation introduction.

Since Pandoc can handle so many different types of document formats, I can write in .org and then publish HTML to the web, or export to PDF, ePUB, or whatever.