Classifying

Classification

To classify data, let’s start by looking at one of the simplest classification goals: to classify an item into one of two buckets.

As an example work here with the same dataset that Mosteller and Wallace use in their initial authorship attribution study. The file here is an XML document that Matthew Jockers and Daniela Witten created for their 2010 article about authorship attribution; since it is XML, we can use the TEIdy package to read it in and lift the author and title information to each document.

The Federalist papers are a classic text of American government written during the period of debate over the ratification over the constitution. They were written as newspaper articles under pseudonyms (like “Publius”), and a few remained of disputed authorship into the 20th century.

The method we’ll explore here is called logistic regression. It is not the method used by Mosteller and Wallace, who applied far more sophisticated statistical methods than almost anything used in ‘data science’ today. But in general, more data allows you to use less complicated tools; and the the omnipresence of logistic regression means that you’re not so likely to have to explain why you chose it. It represents a widely used swiss-army knife tool to divide items into classes.

Mosteller and Wallace look specifically at function words for classification. So before we go into doing a classification, let’s look at how often each author uses each word, as a share of all words he uses?

Naive Bayes

(Incomplete: this is theoretically interesting to think about, but not widely used in practice.)

That new matrix gives us the chance of each author using any individual word. We just need to merge that in, and we can gather the overall probability for any author using the individual words that he or she does. For example, here are the probabilities of each author in our set using the word ‘upon’. If you take a random word, it’s very unlikely that it will be ‘upon’ for any of them; but it’s a lot less unlikely that it’s Hamilton than that it’s Jay or Madison.

Naive Bayesian classifiers by just repeating this over and over again by multiplying possibilities. Multiplying numbers lots of times is hard for computers to track, but this is another case where logarithms are helpful. Maybe you remember from high school math that logarithms let you use addition instead of multiplication. This means that instead of multiplying a string of numbers together, we can simply take the average of their logarithms to see how unlikely the average word is. (We need logarithms for this to be rigorous probability; averaging the percentages is not a good measure.)

Most data science courses talk extensively about linear regression. This is a technique for estimate quantities from other quantities–for instance, you can estimate the amount of money someone has based on their income. While this method is occasionally useful in some digital humanities fields–especially if you touch on economic history–the comparison of two types of quantities is fairly uncommon in digital humanities methods.

Another method, K-nearest-neighbor, predicts classes of items based on a vector space.

A more interpretable method are decision trees. rpart allows training these, but decisions trees are often not especially effective. A method that uses multiple decision trees called random forests is often quite effective, if relatively difficult to interpret. Especially when your data is categorical or difficult to scale, they may be worth exploring. The package randomForest provides random forests.

Logistic Regression

One species of linear regression, called logistic regression has become the overwhelmingly popular way of performing classifications in the digital humanities, perhaps because it does not generally rely on refined statistical assumptions about distributions.

We’ll explore it first through the authorship problem in the Federalist papers. We’ll do this, for the time being, by aiming to predict solely whether the unknown Federalists are by Hamilton or Madison.

First we’ll create a term document matrix as in the last chapter.

This gives us a data frame where two columns are the author and the title, and the rest are the shares of various different words.

Then we run the model. In R, logistic regression is contained inside the glm function, which allows you run many different types of models.

This returns a model object, which is quite different than the tidy data we’ve been working with, and more like the PCA models we saw before.

You can inspect the output of a model by calling the ‘summary’ function. The most important statistics here are the coefficients table; here you can read codes that tell you if the model is finding statistically significant patterns. The general standard for significance is a 95% chance that the result isn’t random, represented by a p value under 0.05. Here, the best score we get is 0.06; you could call that ‘weakly significant’, but fortunately this kind of equivocation is getting pushed out of the social sciences and has never become widespread in the humanities.

Note the special syntax of the glm function. The first argument is the phrase is_hamilton ~ there + by + on. is_hamilton is a logical value defined the code, and there + by + on are the wordcounts we will use to predict it. In between is a squiggly line, ~. We’ve already seen it a bit in other functions. In R you can read the squiggly line as meaning “in terms of” or “as a function of.”

More relevant is how powerful these features are. We can use the model to create a new row of predictions on the combination.

You may have heard that computers think in binary categories and humans think in subtle shades. With authorship, as with much else, the truth is precisely the opposite: while we can know that a particular paper is by Hamilton, a logistic model will only ever put out a probability.

These predictions are, yet again, a form of log transformation. They are, though, a special type of transformation known as the ‘log odds,’ which can be converted to a probability using the so-called ‘logistic function.’ They are ‘odds’ in the betting sense; the log-odds treat a bet like “3 to 1 odds” as a fraction and take the logarithm of it.

The raw output of a logistic regression are numbers of any size: a ‘0’ indicates that the model predicts an even chance of the label belonging to either class: as the numbers grow larger they asymptotically approach certainty that it belongs to the class used to train the model, and lower and lower negative numbers indicate progressively higher chances that the label does not apply.

You might say, for instance, that something has a “million-to-one odds”” of happening; since the natural log of one million is 13.8, that would be expressed as a chance of -13.8. Odds of 95 to 5–that traditional significance interval–are about -2.94; anything in between -2.94 and 2.94 is an uncertain guess.

The chart below shows the predictions of our logistic model in both of these dimensions: normally you’d likely only look at the log odds, because using probabilities tends to cluster things too tightly.

As you can see, our little model seems to be doing a pretty good job: most of the Hamilton has a high probability of being Hamilton-written, and most of the Madison is on the left.

The disputed papers are generally listed as being more likely by Madison, with one exception that looks kind of like a Hamilton-authored paper.

This looks great! But actually, this provides too much certainty, because we’re committing the cardinal sin of data analysis: mixing our test and training data. It’s not right to evaluate the performance of a model on the same data we trained it on; the model can easily be overfitted.

The right way to handle this is to break into test and training sets.

In the tidyverse, we can do this in two steps: first create a test set by sampling 1 in 4 documents randomly, then assigning the rest to a train set using an anti_join function. There are, of course, other ways; but the relational algebra one works well.

Then we fit a new model using glm again.

Thinking about errors requires more than just blindly sampling, though.

Consider predicting the party of an American president based on their stopword usage. We’ll run a classification of whether a president is a “Democrat” or not, allowing all the other various parties to blend together.

Again, we drop down to common words; and pivot out the counts to make them into columns

Once again, we can use predict on a model to see what we get. Remember that predicted values over 0 mean ‘TRUE’ and those below 0 mean FALSE. Here we can use the count() function to see how often we get the correct answer right.