Section 3.8 Working With Text
Beyond working with single strings and string literals, sometimes the information you’re analyzing is a whole body of text. This could be a speech, a novel, an article, or any other written document. In text analysis, the document(s) you’ve set out to analyze are referred to as a corpus. Linguists frequently analyze such types of data and doing so within R in a tidy data format has become simpler thanks to the
tidytext package and the package-accompanying book Text Mining with R.
To get started, the package must be installed and loaded in:
# install.packages("tidytext")
library(tidytext)
Subsection 3.8.1 Tidy Text Format
If we’re thinking about all the text in a novel, it’s pretty clear that it is not in a format that is easy to analyze computationally. To analyze the text in the novel computationally and say, determine what words are used most frequently, or what topics are discussed, we need to convert the text in the novel into a format that a computer can interpret. And, as with all types of data discussed in these courses, we want this to be a tidy format where (1) each observation is a row (2) each variable is a column, and (3) each observational unit is a table. So, how do we take text from a novel and store the information in a tidy format?
The tidy text format requires that the data frame will store one token per row. This requires knowing that a token is a meaningful unit of text. How you define that unit is up to you, the analyst and is driven by the question you’re asking. If you’re looking to identify the words used most frequently in this analysis, the unit of your token would be individual words. You would then utilize your computer to generate a data frame with each row containing data about a single word. However, your token could be two words (a bigram), a sentence, or a paragraph. Whatever you decide is meaningful for your analysis will be the unit for your token. Each row will contain a separate token.
Subsubsection 3.8.1.1 Tokenization
After determining what level of information you’re most interested in, you need a way to go from a wall of text (say, all the text in a novel) to a data frame of tokens (say, individual words). To do this, the
unnest_tokens() function is incredibly useful.
We’ll use a bare bones example to demonstrate how it works. Below is text from the Shel Silverstein poem "Carrots" stored as a character vector:
carrots <- c("They say that carrots are good for your eyes",
"They swear that they improve your sight",
"But I'm seein' worse than I did last night -",
"You think maybe I ain't usin' em right?")
carrots
## [1] "They say that carrots are good for your eyes" ## [2] "They swear that they improve your sight" ## [3] "But I'm seein' worse than I did last night -" ## [4] "You think maybe I ain't usin' em right?"
For analysis, we’d need to get this into a tidy data format. So, first things first, let’s get it into a data frame:
library(tibble)
text_df <- tibble(line = 1:4, text = carrots)
text_df
## # A tibble: 4 × 2 ## line text ## <int> <chr> ## 1 1 They say that carrots are good for your eyes ## 2 2 They swear that they improve your sight ## 3 3 But I'm seein' worse than I did last night - ## 4 4 You think maybe I ain't usin' em right?
At this point we have a tibble with each line of the poem in a separate row. Now, we want to convert this using
unnest_tokens() so that each row contains a single token, where, for this example, our token will be an individual word. This process is known as tokenization.
text_df %>%
unnest_tokens(word, text)
## # A tibble: 33 × 2 ## line word ## <int> <chr> ## 1 1 they ## 2 1 say ## 3 1 that ## 4 1 carrots ## 5 1 are ## 6 1 good ## 7 1 for ## 8 1 your ## 9 1 eyes ## 10 2 they ## # … with 23 more rows
Notice that the two arguments to the
unnest_tokens() function. The first (word in our example) is the name of the token column in the output. The second (text in our example) is the name of the column in the input data frame (text_df) that should be used for tokenization.
In the output we see that there is a single word (token) in each row, so our data are now in a tidy format, which makes further analysis simpler.
Finally, note that, by default
unnest_tokens() strips punctuation and converts the tokens to lowercase.
Subsection 3.8.2 Sentiment Analysis
Often, once you’ve tokenized your dataset, there is an analysis you want to do - a question you want to answer. Sometimes, this involves wanting to measure the sentiment of a piece by looking at the emotional content of the words in that piece.
To do this, the analyst must have access to or create a lexicon, a dictionary with the sentiment of common words. There are three single word-based lexicons available within the
tidytext package: afinn, bing, loughran and nrc. Each differs in how they categorize sentiment, and to get a sense of how words are categorized in any of these lexicon, you can use the get_sentiments() function.
However, this requires an additional package:
textdata. Be sure this has been installed before using the get_sentiments() function.
library(textdata)
# be sure textdata is installed
#install.packages("textdata", repos = 'http://cran.us.r-project.org')
# see information stored in NRC lexicon
get_sentiments('nrc')
## # A tibble: 13,901 × 2 ## word sentiment ## <chr> <chr> ## 1 abacus trust ## 2 abandon fear ## 3 abandon negative ## 4 abandon sadness ## 5 abandoned anger ## 6 abandoned fear ## 7 abandoned negative ## 8 abandoned sadness ## 9 abandonment anger ## 10 abandonment fear ## # … with 13,891 more rows
Note: The first time you use this function R will prompt you to verify that you want to download the lexicon.
In the output you’ll see words in the first column and the sentiment attached to each word in the
sentiment column. Notice that the same word can have multiple sentiments attached to it. All told, there are more than 13,000 word-sentiment pairs in this lexicon.
Let’s quantify the sentiment in the "Carrots" poem from above:
text_df %>%
unnest_tokens(word, text) %>%
inner_join(get_sentiments('nrc'))
## Joining, by = "word" ## # A tibble: 14 × 3 ## line word sentiment ## <int> <chr> <chr> ## 1 1 good anticipation ## 2 1 good joy ## 3 1 good positive ## 4 1 good surprise ## 5 1 good trust ## 6 2 swear positive ## 7 2 swear trust ## 8 2 improve anticipation ## 9 2 improve joy ## 10 2 improve positive ## 11 2 improve trust ## 12 3 worse fear ## 13 3 worse negative ## 14 3 worse sadness
Notice that the sentiments applied to each word are dependent upon the sentiments defined within the lexicon. Words that are missing or that are used differently than anticipated by those who generated the lexicon could be misclassified. Additionally, since we’re using single word tokens, qualifiers are removed from context. So in the carrots poem, the word good in "are good for your eyes" would be given the same sentiment as good if the phrase were "are not good for your eyes." Thus, a lot context and nuance is lost in this approach. It’s always important to consider the limitations of your analytical approach!
Above we found the sentiments for each token, but let’s summarize that by counting the number of times each sentiment appears.
text_df %>%
unnest_tokens(word, text) %>%
inner_join(get_sentiments('nrc')) %>%
count(sentiment, sort = TRUE)
## Joining, by = "word" ## # A tibble: 8 × 2 ## sentiment n ## <chr> <int> ## 1 positive 3 ## 2 trust 3 ## 3 anticipation 2 ## 4 joy 2 ## 5 fear 1 ## 6 negative 1 ## 7 sadness 1 ## 8 surprise 1
As we’re analyzing a short poem, we see that only a few sentiments show up multiple times; however, using sentiment analysis on this poem suggests that the poem is generally positive, including words that convey trust, anticipation, and joy.
Analyzing a four line poem, however, is not typically what one would do. They would instead analyze the text across chapters in a book or across multiple books. Here, we’ve just demonstrated the concepts behind how you would go about carrying out sentiment analysis.
Subsection 3.8.3 Word and Document Frequency
Beyond sentiment analysis, analysts of text are often interested in quantifying what a document is about. One could start by quantifying term frequency and looking at which terms occur most often; however, common words, such as the and and, are likely to appear most often. Those aren’t unique to the work and hardly explain the text’s topic. Often, these words, referred to as stop words are removed from analysis; however, these words are more important to some works relative to others. So, analysts tend to take a different approach: inverse document frequency (idf).
A document’s inverse document frequency (idf) weights each term by its frequency in a collection of documents. Those words that are quite common in a set of documents are down-weighted. The weights for words that are less common are increased. By combining idf with term frequency (tf) (through multiplication), words that are common and unique to that document (relative to the collection of documents) stand out.
To see an example of this, we’ll need a few more poems from Shel Silverstein for analysis. Here is Invitation:
library(tibble)
invitation <- c("If you are a dreamer, come in,",
"If you are a dreamer, a wisher, a liar",
"A hope-er, a pray-er, a magic bean buyer…",
"If you’re a pretender, come sit by my fire",
"For we have some flax-golden tales to spin.",
"Come in!",
"Come in!")
invitation <- tibble(line = 1:7, text = invitation, title = "Invitation")
invitation
## # A tibble: 7 × 3 ## line text title ## <int> <chr> <chr> ## 1 1 If you are a dreamer, come in, Invitation ## 2 2 If you are a dreamer, a wisher, a liar Invitation ## 3 3 A hope-er, a pray-er, a magic bean buyer… Invitation ## 4 4 If you’re a pretender, come sit by my fire Invitation ## 5 5 For we have some flax-golden tales to spin. Invitation ## 6 6 Come in! Invitation ## 7 7 Come in! Invitation
And, here is masks:
masks <- c("She had blue skin.",
"And so did he.",
"He kept it hid",
"And so did she.",
"They searched for blue",
"Their whole life through",
"Then passed right by—",
"And never knew")
masks <- tibble(line = 1:8, text = masks, title = "Masks")
masks
## # A tibble: 8 × 3 ## line text title ## <int> <chr> <chr> ## 1 1 She had blue skin. Masks ## 2 2 And so did he. Masks ## 3 3 He kept it hid Masks ## 4 4 And so did she. Masks ## 5 5 They searched for blue Masks ## 6 6 Their whole life through Masks ## 7 7 Then passed right by— Masks ## 8 8 And never knew Masks
We’ll combine all three poems into a single data frame for TF-IDF analysis. To do so, we’ll first add a column to our carrots example from above so that it has a column for
title:
# add title to carrots poem
carrots <- text_df %>% mutate(title = "Carrots")
# combine all three poems into a tidy data frame
poems <- bind_rows(carrots, invitation, masks)
Now that we have our three documents (poems) in a single data frame, we can tokenize the text by word and calculate each tokens frequency within the document (poem).
# count number of times word appwars within each text
poem_words <- poems %>%
unnest_tokens(word, text) %>%
count(title, word, sort = TRUE)
# count total number of words in each poem
total_words <- poem_words %>%
group_by(title) %>%
summarize(total = sum(n))
# combine data frames
poem_words <- left_join(poem_words, total_words)
poem_words
## Joining, by = "title" ## # A tibble: 82 × 4 ## title word n total ## <chr> <chr> <int> <int> ## 1 Invitation a 8 48 ## 2 Invitation come 4 48 ## 3 Carrots they 3 33 ## 4 Invitation if 3 48 ## 5 Invitation in 3 48 ## 6 Masks and 3 31 ## 7 Carrots i 2 33 ## 8 Carrots that 2 33 ## 9 Carrots your 2 33 ## 10 Invitation are 2 48 ## # … with 72 more rows
Note that there are a different number of total words in each document, which is important to consider when you’re comparing relative frequency between documents.
We could visualize the number of times a word appears relative to document length as follows:
library(ggplot2)
# visualize frequency / total words in poem
ggplot(poem_words, aes(n/total, fill = title)) +
geom_histogram(show.legend = FALSE, bins = 5) +
facet_wrap(~title, ncol = 3, scales = "free_y")
With most documents there are only a few words that show up infrequently in the tail off to the right (rare words), while most words show up a whole bunch of times.
What we’ve just visualized is term frequency. We can add this quantity to our data frame:
freq_by_rank <- poem_words %>%
group_by(title) %>%
mutate(rank = row_number(),
`term frequency` = n/total)
Notice that words that appear most frequently will have the largest term frequency. However, we’re not just interested in word frequency, as stop words (such as "a") have the highest term frequency. Rather, we’re interested in tf-idf - those words in a document that are unique relative to the other documents being analyzed.
To calculate tf-idf, we can use
bind_tf_idf(), specifying three arguments: the column including the token (word), the column specifying the document from which the token originated (title), and the column including the number of times the word appears (n):
poem_words <- poem_words %>%
bind_tf_idf(word, title, n)
# sort ascending
poem_words %>%
arrange(tf_idf)
## # A tibble: 82 × 7 ## title word n total tf idf tf_idf ## <chr> <chr> <int> <int> <dbl> <dbl> <dbl> ## 1 Carrots for 1 33 0.0303 0 0 ## 2 Invitation for 1 48 0.0208 0 0 ## 3 Masks for 1 31 0.0323 0 0 ## 4 Invitation by 1 48 0.0208 0.405 0.00845 ## 5 Carrots are 1 33 0.0303 0.405 0.0123 ## 6 Carrots did 1 33 0.0303 0.405 0.0123 ## 7 Carrots right 1 33 0.0303 0.405 0.0123 ## 8 Carrots you 1 33 0.0303 0.405 0.0123 ## 9 Masks by 1 31 0.0323 0.405 0.0131 ## 10 Masks right 1 31 0.0323 0.405 0.0131 ## # … with 72 more rows
If we sort this output in ascending order by
tf_idf, you’ll notice that the word "for" has a tf_idf of 0. The data indicates that this word shows up with equal frequency across all three poems. It is not a word unique to any one poem.
# sort descending
poem_words %>%
arrange(desc(tf_idf))
## # A tibble: 82 × 7 ## title word n total tf idf tf_idf ## <chr> <chr> <int> <int> <dbl> <dbl> <dbl> ## 1 Invitation a 8 48 0.167 1.10 0.183 ## 2 Masks and 3 31 0.0968 1.10 0.106 ## 3 Invitation come 4 48 0.0833 1.10 0.0916 ## 4 Masks blue 2 31 0.0645 1.10 0.0709 ## 5 Masks he 2 31 0.0645 1.10 0.0709 ## 6 Masks she 2 31 0.0645 1.10 0.0709 ## 7 Masks so 2 31 0.0645 1.10 0.0709 ## 8 Invitation if 3 48 0.0625 1.10 0.0687 ## 9 Invitation in 3 48 0.0625 1.10 0.0687 ## 10 Carrots i 2 33 0.0606 1.10 0.0666 ## # … with 72 more rows
Alternatively, here we see the words most unique to the individual poems. "a" and "come" are most unique to Invitation, while "and" and "blue" are most unique to Masks. If we had removed stop words, we would have lost the fact that some common words are really unique in one of these poems relative to the others.
Again, we’re looking at a limited amount of text here, but this analysis can be applied to novels, works by different authors, or articles written in a newspaper.
We can summarize these tf-idf results by visualizing the words with the highest tf-idf in each of these poems:
poem_words %>%
arrange(desc(tf_idf)) %>%
mutate(word = factor(word, levels = rev(unique(word)))) %>%
group_by(title) %>%
top_n(3) %>%
ungroup() %>%
ggplot(aes(word, tf_idf, fill = title)) +
geom_col(show.legend = FALSE) +
labs(x = NULL, y = "tf-idf") +
facet_wrap(~title, ncol = 3, scales = "free") +
coord_flip()
## Selecting by tf_idf
