Texts as Data

Loading texts into R

You can read in a text the same way you would a table. But the most basic way is to take a step down the complexity ladder and read it in as a series of lines.

As a reminder, you probably need to update the package:

      #devtools::install_github("HumanitiesDataAnalysis/HumanitiesDataAnalysis")

text = read_lines("SOTUS/2020.txt")

text %>% head(3)
    

If you type in “text,” you’ll see that we get the full 2015 State of the Union address, broken into multiple lines.

As structured here, the text is divided into paragraphs. For most of this text, we’re going to be interested instead in working with words.

Tokenization

Let’s pause and note a perplexing fact: although you probably think that you know what a word is, that conviction should grow shaky the more you think about it. “Field” and “fields” take up the same dictionary definition: is that one word or two?

In the field of corpus linguistics, the term “word” is generally dispensed with as too abstract in favor of the idea of a “token” or “type.”

Where a word is more abstract, a “type” is a concrete term used in actual language, and a “token” is the particular instance we’re interested in. The type-token distinction is used across many fields. Philosophers like Charles Saunders Pierce, who originated the term @williams_tokens_1936, use it to distinguish between abstract things (‘wizards’) and individual instances of the thing (‘Harry Potter.’) Numismatologist (scholars of coins) use it to characterize the hierarchy of coins: a single mint might produce many coins from the same mold, which is the ‘type’; each individual instance is a token of that type.

Breaking a piece of text into words is thus called “tokenization.” There are many ways to do it–coming up with creative tokenization methods will be helpful in the algorithms portion of this class. But the simplest is to simply to remove anything that isn’t a letter. Using regular expression syntax, the R function strsplit lets us do just this: split a string into pieces. We could use the regular expression [^A-Za-z] to say “split on anything that isn’t a letter between A and Z.” Note, for example, that this makes the word “Don’t” into two words.

      text %>%
  str_split("[^A-Za-z]") %>%
  head(2)
    

You’ll notice that now each paragraph is broken off in a strange way: each paragraph shows up nested. this is because we’re now looking at data structures other than a data.frame. This can be useful: but doesn’t let us apply the rules of tidy analysis we’ve been working with. We’ve been working with tibble objects from the tidyverse. This individual word, though, can be turned into a column in a tibble by using tibble function to create it.

      SOTU = tibble(text = text)
    

Now we can use the “tidytext” package to start to analyze the document.

The workhorse function in tidytext is unnest_tokens. It creates a new columns (here called ‘words’) from each of the individual ones in text. This is the same, conceptually, as the splitting above, but with a large number of hard-earned knowledge built in.

      library(tidytext)

tidied = SOTU %>%
  unnest_tokens(word, text)
    

You’ll notice, immediately, that this looks a little different: each of the words is lowercased, and we’ve lost all punctuation.

      list(
  SOTU %>% unnest_tokens(word, text) %>% select(lowercase = word),
  SOTU %>% unnest_tokens(word, text) %>% rowwise() %>% mutate(word = SnowballC::wordStem(word)) %>% select(stemmed = word),
  SOTU %>% unnest_tokens(word, text, to_lower = F) %>% select(uppercase = word),
  SOTU %>% unnest_tokens(word, text, to_lower = F, strip_punc = FALSE) %>% select(punctuations = word),
  SOTU %>% unnest_tokens(word, text, token = "ngrams", n = 2, to_lower = F) %>% select(bigrams = word)
) %>% map_dfc(~ .x %>% head(10))
    

      wordcounts = tidied %>%
  group_by(word) %>%
  summarize(n = n()) %>%
  arrange(-n)

wordcounts %>% head(5)
    

      wordcounts = wordcounts %>%
  mutate(rank = rank(-n)) %>%
  filter(n > 2, word != "")

ggplot(wordcounts) + aes(x = rank, y = n, label = word) + geom_text()
    

      ggplot(wordcounts) +
  aes(x = rank, y = n, label = word) +
  geom_point(alpha = .3, color = "grey") +
  geom_text(check_overlap = TRUE) +
  scale_x_continuous(trans = "log") +
  scale_y_continuous(trans = "log") +
  labs(title = "Zipf's Law",
       subtitle="The log-frequency of a term is inversely correlated with the logarithm of its rank.")
    

      nowadays = CESTA %>%
  arrange(-`2010`) %>% 
  mutate(rank = rank(-`2010`)) %>% 
  filter(rank < 1000)

ggplot(nowadays) + aes(x = rank, label = CityST, y = `2010`) +
  geom_point(alpha = 1, color = "grey") +
  geom_text(check_overlap = TRUE, adj = -.1) +
  scale_x_continuous(trans = "log") +
  scale_y_continuous(trans = "log")
    

      tibble(number = c(1, 2, 3, 4, 5)) %>%
  mutate(lag = lag(number, 1)) %>%
  mutate(lead = lead(number, 1))
    

      twoColumns = tidied %>% mutate(word2 = lead(word, 1))

twoColumns %>%
  group_by(word, word2) %>%
  summarize(count = n()) %>%
  arrange(-count) %>%
  head(10)
    

      multiColumn = tidied %>% 
  mutate(word2 = lead(word, 1),
         word3 = lead(word, 2),
         word4 = lead(word, 3),
         word5 = lead(word, 4))


multiColumn %>% count(word, word2, word3, word4, word5) %>% arrange(-n) %>% head(5)
    

      multiColumn %>% filter(word3 == "immigration")
    

Functions

That’s just one State of the Union. How are we to read them all in?

We could obviously type all the code in, one at a time. But that’s bad pogramming!

To work with this, we’re going to finally discuss a core programming concepts: functions.

We’ve used functions, continuously: but whenever you’ve written up a useful batch of code, you can bundle it into a function that you can then reuse.

Here, for example, is a function that will read the state of the union address for any year.

Note that we add one more thing to the end–a column that identifies the year.

      readSOTU = function(filename) {
  
  read_lines(filename) %>%
    tibble(text = .) %>%
    filter(text != "") %>%
    ## Add a counter for the paragraph number
    mutate(paragraph = 1:n()) %>%
    unnest_tokens(word, text) %>%
    mutate(filename = filename %>%
             str_replace(".*/", "") %>%
             str_replace(".txt", "")
           )
}

readSOTU("SOTUS/1899.txt") %>% slice(16:20)
    

We also need to know all the names of the files! R has functions for this built in natively–you can do all sorts of file manipulation programatically, from downloading to renaming and even deleting files from your hard drive.

      all_files = list.files("SOTUS", full.names = T)
    

Now we have a list of State of the Unions and a function to read them in. How do we automate this procedure? There are several ways.

The most traditional way, dating back to the programming languages of the 1950s, would be to write a for loop. This is OK to do, and maybe you’ll see it at some point. But it’s not the approach we take in this class.

It’s worth knowing how to recognize this kind of code if you haven’t seen it before. But we aren’t using it here. Good R programmers almost never write a for-loop; instead, they use functions that abstract up a level from working on a single item at a time.

But if you’re working in–say–python, this is almost certainly how you’d do it.

      allSOTUs = tibble()

for (fname in all_files) {
  allSOTUs = rbind(allSOTUs, readSOTU(fname))
}
    

One of the most basic ones is called “map”: it takes as an argument a list and a function, and applies the function to each element of the list. The word “map” can mean so many things that you could write an Abbot-and-Costello routine about it. It can mean cartography, it can mean a lookup dictionary, or it can mean a process of applying a function multiple times.

The tidyverse comes with a variety of special functions for performing mapping in this last sense, including one for combining dataframes by rows called map_dfr.

      allSOTUs = all_files %>%
  map_dfr(readSOTU)


allSOTUs %>%
  group_by(filename) %>%
  summarize(count = n()) %>%
  ggplot() + geom_line() + aes(x = filename %>% as.numeric(), y = count) +
  labs(title = "Number of words per State of the Union")
    

      allSOTUs = allSOTUs %>%
  mutate(year = as.numeric(filename)) %>%
  inner_join(presidents)
    

      allSOTUs %>%
  ggplot() +
  geom_bar(aes(x = year, fill = party))
    

        labs("SOTU lengths by year and party")
    

      allSOTUs %>%
  group_by(year, sotu_type) %>%
  summarize(`Word Count` = n()) %>%
  ggplot() +
  geom_point(aes(x = year, y = `Word Count`, color = sotu_type))
    

      allSOTUs %>%
  mutate(year = as.numeric(filename)) %>%
  group_by(word, president) %>%
  summarize(count = n()) %>%
  group_by(word) %>%
  filter(n() == 1) %>%
  arrange(-count) %>% head(10)
    

Subword tokenization with sentencepiece

Some notes on integrating more complicated tokenization with tidytext in R

Tidytext works well in English, but for some languages the assumption that a “word” can be created through straightforward tokenization is more problematic. In language like Turkish or Finnish, agglutination (the composition of words through suffixes) is common; in languages like Latin or Russian, words may appear in many different forms; and in Chinese, text may not have any whitespace delineation at all.

“Sentencepiece” is one of several strategies for so-called “subword encoding” that have become popular in neural-network based natural language processing pipelines. It works by looking at the statistical properties of a set of texts to find linguistically common breakpoints.

You tell it how many individual tokens you want it to output, and it divides words up into portions. These portions may be as long as a full word or as short as a letter.

Especially common words will usually be preserved as they are. In the R implementation, sequences that start a word are prefixed with ▁: so, for instance, the most common word is “▁to” . This includes not just the word “to” but also “tough”, “tolerance”, and other things that start with “to”.

The number I give here, 4000 tokens, is fairly small. If you are doing this on a substantially sized corpus, you might want to go as high as 16,000 or more tokens. But the choice you make will undoubtedly be inflected by the language you’re working in, so above all, look at whether the segments you get seem to be occasionally useful.

      if (!require(sentencepiece)) install.packages("sentencepiece")
library(sentencepiece)
    

      files = list.files("SOTUS", full.names = TRUE) %>% keep(~str_detect(.x, ".txt"))
model = sentencepiece(files, vocab_size = 4000 - 1, model_dir = "." )
    

      words = sentencepiece_encode(model, read_lines(files[210]))
table = data_frame(words) %>% unnest(words)
    

      table %>% count(encoded) %>% arrange(-n)
table %>% filter(lag(encoded, 1) == "▁in") %>% count(encoded) %>% arrange(-n)
    

Probabilities and Markov chains.

Let’s look at probabilities, which lets us apply our merging and functional skills in a fun way.

      bigrams = allSOTUs %>%
  mutate(word1 = word, word2 = lead(word, 1), word3 = lead(word, 2))

transitions = bigrams %>%
  group_by(word1) %>%
  # First, we store the count for each word.
  mutate(word1Count = n()) %>%
  group_by(word1, word2) %>%
  # Then we group1 by the second word to see what share of 
  # the first word is followed by the second.
  summarize(chance = n() / word1Count[1])
    

This gives a set of probabilities. What words follow “United?”

      transitions %>%
  filter(word1 == "united") %>%
  arrange(-chance) %>%
  head(5)
    

We can use the weight argument to sample_n. If you run this several times, you’ll see that you get different results.

      transitions %>%
  filter(word1 == "my") %>%
  sample_n(1, weight = chance)
    

So now, consider how we combine this with joins. We can create a new column from a seed word, join it in against the transitions frame, and then use the function str_c–which pastes strings together– to create a new column holding our new text.

      chain = data_frame(text = "my")
    

      chain = chain %>%
  mutate(word1 = text) %>%
  inner_join(transitions) %>%
  sample_n(1, weight = chance) %>%
  mutate(text = str_c(text, word2, sep = " "))
    

      chain
    

(Are you having fun yet?) Here’s what’s interesting. This gives us a column that’s called “text”; and we can select it, and tokenize it again.

Why is that useful? Because now we can combine this join and the wordcounts to keep doing the same process! We tokenize our text; take the very last token; and do the merge.

Try running this piece of code several different times.

      chain = chain %>%
  select(text) %>%
  unnest_tokens(word1, text, drop = FALSE) %>%
  tail(1) %>% inner_join(transitions) %>%
  sample_n(1, weight = chance) %>%
  mutate(text = str_c(text, word2, sep=" ")) %>%
  select(text)
    

      chain
    

You can do this until your fingers get tired: but we can also wrap it up as a function. In the tidyverse, the simplest functions start with a dot.

      add_a_word = . %>%
  select(text) %>%
  unnest_tokens(word1, text, drop = FALSE) %>%
  tail(1) %>% inner_join(transitions, by = "word1") %>%
  sample_n(1, weight = chance) %>%
  mutate(text = str_c(text, word2, sep=" ")) %>%
  select(text)
    

      tibble(text = "america") %>%
  add_a_word %>% add_a_word %>% add_a_word %>%
  add_a_word %>% add_a_word %>% add_a_word %>%
  add_a_word %>% add_a_word %>% add_a_word %>%
  add_a_word %>% add_a_word %>% add_a_word %>%
  add_a_word %>% add_a_word %>% add_a_word %>% pull(text)
    

      seed = tibble(text = "start")

for (i in 1:15) {
  seed = add_a_word(seed)
}

seed %>% pull(text)
    

A more head-scratching way is to use a technique called “recursion.”

Here the function calls itself if it has more than one word left to add. This would be something a programming class would dig into; here just know that functions might call themselves!

      add_n_words = function(seed_frame, n_words) {
  with_one_addition = add_a_word(seed_frame)
  if (n_words==1) {
    return(with_one_addition)
  } else {
    return(add_n_words(with_one_addition, n_words - 1))
  }
}

add_n_words(tibble(text = "chile"), 10)
    

      length_2_transitions = allSOTUs %>%
  mutate(word1 = word, word2 = lead(word, 1), word3 = lead(word, 2)) %>%
  group_by(word1, word2, word3) %>%
  summarize(chance = n()) %>% 
  ungroup
    

      length_2_transitions %>% sample_n(10)
    

      add_a_word = . %>%
  select(text) %>%
  unnest_tokens(word1, text, drop = FALSE) %>%
  tail(1) %>% inner_join(transitions, by = "word1") %>%
  sample_n(1, weight = chance) %>%
  mutate(text = str_c(text, word2, sep=" ")) %>%
  select(text)

add_a_bigram = . %>%
  unnest_tokens(word, text, drop=FALSE) %>%
  tail(2) %>% mutate(wordpos = str_c("word", 1:n())) %>%
  pivot_wider(values_from=word,names_from = wordpos) %>%
  inner_join(length_2_transitions) %>%
  sample_n(1, weight = chance) %>%
  mutate(text = str_c(text, word3, sep=" ")) %>%
  select(text)

start = tibble(text = "united states")

start = start %>% add_a_bigram
    

      start %>% head(10)
    

      library(tidyverse)
library(HumanitiesDataAnalysis)
library(tidytext)

extract_SOTUs()

# This code can be modified to work on *any* folder of text files.
allSOTUs = list.files("SOTUS", full.names = TRUE) %>%
  map_dfr(read_tokens)

# This one is state-of-the-union specific.
allSOTUs = allSOTUs %>%
  mutate(year = as.numeric(filename)) %>%
  inner_join(HumanitiesDataAnalysis::presidents)
    

      allSOTUs %>% 
  filter (sotu_type == "speech") %>%
  KEEP_GOING
    

      allSOTUs %>% 
  filter (sotu_type == "speech") %>%
  group_by(president, year) %>% 
  KEEP_GOING
    

      totals = KEEP_GOING()
    

      military_counts = allSOTUs %>%
  filter(word %in% c("army", "navy", "marines", "forces")) %>%
  KEEP_GOING
    

      all_my_files = list.files("SOTUS", full.names = TRUE) %>%
  map_dfr(read_tokens) %>%
  KEEP_GOING