Introduction to Data Science Version 3

The picture at the start of this chapter is a so called “word cloud” that was generated by examining all of the words returned from a Twitter search of the term “data science” (using a web application at http://www.jasondavies.com) These colorful word clouds are fun to look at, but they also do contain some useful information. The geometric arrangement of words on the figure is partly random and partly designed and organized to please the eye. Same with the colors. The font size of each word, however, conveys some measure of its importance in the “corpus” of words that was presented to the word cloud graphics program. Corpus, from the Latin word meaning “body,” is a word that text analysts use to refer to a body of text material, often consisting of one or more documents. When thinking about a corpus of textual data, a set of documents could really be anything: web pages, word processing documents on your computer, a set of Tweets, or government reports. In most cases, text analysts think of a collection of documents, each of which contains some natural language text, as a corpus if they plan to analyze all the documents together.

The word cloud on the previous page shows that “Data” and “Science” are certainly important terms that came from the search of Twitter, but there are dozens and dozens of less important, but perhaps equally interesting, words that the search results contained. We see words like algorithms, molecules, structures, and research, all of which could make sense in the context of data science. We also see other terms, like #christian, Facilitating, and Coordinator, that don’t seem to have the same obvious connection to our original search term “data science.” This small example shows one of the fundamental challenges of natural language processing and the closely related area of search: ensuring that the analysis of text produces results that are relevant to the task that the user has in mind.

In this chapter we will use some new R packages to extend our abilities to work with text and to build our own word cloud from data retrieved from Twitter. If you have not worked on the chapter “String Theory” that precedes this chapter, you should probably do so before continuing, as we build on the skills developed there.

Depending upon where you left off after the previous chapter, you will need to retrieve and pre-process a set of tweets, using some of the code you already developed, as well as some new code. At the end of the previous chapter, we have provided sample code for the TweetFrame() function, that takes a search term and a maximum tweet limit and returns a time-sorted dataframe containing tweets. Although there are a number of comments in that code, there are really only three lines of functional code thanks to the power of the twitteR package to retrieve data from Twitter for us. For the activities below, we are still working with the dataframe that we retrieved in the previous chapter using this command:

tweetDF <- TweetFrame("#solar",100)

This yields a dataframe, tweetDF, that contains 100 tweets with the hashtag #solar, presumably mostly about solar energy and related “green” topics. Before beginning our work with the two new R packages, we can improve the quality of our display by taking out a lot of the junk that won’t make sense to show in the word cloud. To accomplish this, we have authored another function that strips out extra spaces, gets rid of all URL strings, takes out the retweet header if one exists in the tweet, removes hashtags, and eliminates references to other people’s tweet handles. For all of these transformations, we have used string replacement functions from the stringr package that was introduced in the previous chapter. As an example of one of these transformations, consider this command, which appears as the second to last line of the CleanTweet() function:

tweets <- str_replace_all(tweets,"@[a-z,A-Z]*","")

You should feel pretty comfortable reading this line of code, but if not, here’s a little more practice. The left hand side is easy: we use the assignment arrow to assign the results of the right hand side expression to a data object called “tweets.” Note that when this statement is used inside the function as shown at the end of the chapter, “tweets” is a temporary data object, that is used just within CleanTweets() after which it disappears automatically.

The right hand side of the expression uses the str_replace_all() function from the stringr package. We use the “all” function rather than str_replace() because we are expecting multiple matches within each individual tweet. There are three arguments to the str_replace_all() function. The first is the input, which is a vector of character strings (we are using the temporary data object “tweets” as the source of the text data as well as its destination), the second is the regular expression to match, and the third is the string to use to replace the matches, in this case the empty string as signified by two double quotes with nothing between them. The regular expression in this case is the at sign, “@”, followed by zero or more upper and lowercase letters. The asterisk, “*”, after the stuff in the square brackets is what indicates the zero or more. That regular expression will match any screenname referral that appears within a tweet.

If you look at a few tweets you will find that people refer to each other quite frequently by their screennames within a tweet, so @SolarFred might occur from time to time within the text of a tweet. Here’s something you could investigate on your own: Can screennames contain digits as well as letters? If so, how would you have to change the regular expression in order to also match the digits zero through nine as part of the screenname? On a related note, why did we choose to strip these screen names out of our tweets? What would the word cloud look like if you left these screennames in the text data?

Whether you typed in the function at the end of this chapter or you plan to enter each of the cleaning commands individually, let’s begin by obtaining a separate vector of texts that is outside the original dataframe:

cleanText<-tweetDF$text
head(cleanText, 10)

There’s no critical reason for doing this except that it will simplify the rest of the presentation. You could easily copy the tweetDF$text data into another column in the same dataframe if you wanted to. We’ll keep it separate for this exercise so that we don’t have to worry about messing around with the rest of the dataframe. The head() command above will give you a preview of what you are starting with. Now let’s run our custom cleaning function:

cleanText<-CleanTweets(cleanText)
head(cleanText, 10)

Note that we used our “cleanText” data object in the first command above as both the source and the destination. This is an old computer science trick for cutting down on the number of temporary variables that need to be used. In this case it will do exactly what we want, first evaluating the right hand side of the expression by running our CleanTweets() function with the cleanText object as input and then taking the result that is returned by CleanTweets() and assigning it back into cleanText, thus overwriting the data that was in there originally. Remember that we have license to do whatever we want to cleanText because it is a copy of our original data, and we have left the original data intact (i.e., the text column inside the tweetDF dataframe).

The head() command should now show a short list of tweets with much of the extraneous junk filtered out. If you have followed these steps, cleanText is now a vector of character strings (in this example exactly 100 strings) ready for use in the rest of our work below. We will now use the “tm” package to process our texts. The “tm” in this case refers to “text mining,” and is a popular choice among the many text analysis packages available in R. By the way, text mining refers to the practice of extracting useful analytic information from corpora of text (corpora is the plural of corpus). Although some people use text mining and natural language processing interchangeably, there are probably a couple subtle differences worth considering. First, the “mining” part of text mining refers to an area of practice that looks for unexpected patterns in large data sets, or what some people refer to as knowledge discovery in databases. In contrast, natural language processing reflects a more general interest in understanding how machines can be programmed (or learn on their own) how to digest and make sense of human language. In a similar vein, text mining often focuses on statistical approaches to analyzing text data, using strategies such as counting word frequencies in a corpus. In natural language processing, one is more likely to hear consideration given to linguistics, and therefore to the processes of breaking text into its component grammatical pieces such as nouns and verbs. In the case of the “tm” add on package for R, we are definitely in the statistical camp, where the main process is to break down a corpus into sequences of words and then to tally up the different words and sequences we have found.