Section 14.2 The Term-Document Matrix
To begin, make sure that the tm package is installed and “library-ed” in your copy of R and R-Studio. You can use the graphic interface in R-Studio for this purpose or the EnsurePackage() function that we wrote in a previous chapter. Once the tm package is ready to use, you should be able to run these commands:
tweetCorpus<-Corpus(VectorSource(cleanText))
tweetCorpus
A corpus with 100 text documents
tweetCorpus<-tm_map(tweetCorpus, tolower)
tweetCorpus<-tm_map(tweetCorpus, removePunctuation)
tweetCorpus<-tm_map(tweetCorpus,removeWords,+
stopwords('english'))
In the first step above , we “coerce” our cleanText vector into a custom “Class” provided by the tm package and called a “Corpus,” storing the result in a new data object called “tweetCorpus.” This is the first time we have directly encountered a “Class.” The term “class” comes from an area of computer science called “object oriented programming.” Although R is different in many ways from object-oriented languages such as Java, it does contain many of the most fundamental features that define an object oriented language. For our purposes here, there are just a few things to know about a class. First, a class is nothing more or less than a definition for the structure of a data object. Second, classes use basic data types, such as numbers, to build up more complex data structures. For example, if we made up a new “Dashboard” class, it could contain one number for “Miles Per Hour,” another number for “RPM,” and perhaps a third one indicating the remaining “Fuel Level.” That brings up another point about Classes: users of R can build their own. In this case, the author of the tm package, Ingo Feinerer, created a new class, called Corpus, as the central data structure for text mining functions. (Feinerer is a computer science professor who works at the Vienna University of Technology in the Database and Artificial Intelligence Group.) Last, and most important for this discussion, a Class not only contains definitions about the structure of data, it also contains references to functions that can work on that Class. In other words, a Class is a data object that carries with it instructions on how to do operations on it, from simple things like add and subtract all the way up to complicated operations such as graphing.
In the case of the tm package, the Corpus Class defines the most fundamental object that text miners care about, a corpus containing a collection of documents. Once we have our texts stored in a Corpus, the many functions that the tm package provides to us are available. The last three commands in the group above show the use of the tm_map() function, which is one of the powerful capabilities provided by tm. In each case where we call the tm_map() function, we are providing tweetCorpus as the input data, and then we are providing a command that undertakes a transformation on the corpus. We have done three transformations here, first making all of the letters lowercase, then removing the punctuation, and finally taking out the so called “stop” words.
The stop words deserve a little explanation. Researchers who developed the early search engines for electronic databases found that certain words interfered with how well their search algorithms worked. Words such as “the,” “a,” and “at” appeared so commonly in so many different parts of the text that they were useless for differentiating between documents. The unique and unusual nouns, verbs, and adjectives that appeared in a document did a much better job of setting a document apart from other documents in a corpus, such that researchers decided that they should filter out all of the short, commonly used words. The term “stop words” seems to have originated in the 1960s to signify words that a computer processing system would throw out or “stop using” because they had little meaning in a data processing task. To simplify the removal of stop words, the tm package contains lists of such words for different languages. In the last command on the previous page we requested the removal of all of the common stop words.
At this point we have processed our corpus into a nice uniform “bag of words” that contains no capital letters, punctuation, or stop words. We are now ready to conduct a kind of statistical analysis of the corpus by creating what is known as a “term-document matrix.” The following command from the tm package creates the matrix:
tweetTDM<-TermDocumentMatrix(tweetCorpus)
tweetTDM
A term-document matrix (375 terms, 100 documents) Non-/sparse entries: 610/36890 Sparsity : 98% Maximal term length: 21 Weighting : term frequency (tf)
A term-document matrix, also sometimes called a document-term matrix, is a rectangular data structure with terms as the rows and documents as the columns (in other uses you may also make the terms as columns and documents as rows). A term may be a single word, for example, “biology,” or it could also be a compound word, such as “data analysis.” The process of determining whether words go together in a compound word can be accomplished statistically by seeing which words commonly go together, or it can be done with a dictionary. The tm package supports the dictionary approach, but we have not used a dictionary in this example. So if a term like “data” appears once in the first document, twice in the second document, and not at all in the third document, then the column for the term data will contain 1, 2, 0.
The statistics reported when we ask for tweetTDM on the command line give us an overview of the results. The TermDocumentMatrix() function extracted 375 different terms from the 100 tweets. The resulting matrix mainly consists of zeros: Out of 37,500 cells in the matrix, only 610 contain non-zero entries, while 36,890 contain zeros. A zero in a cell means that that particular term did not appear in that particular document. The maximal term length was 21 words, which an inspection of the input tweets indicates is also the maximum word length of the input tweets. Finally, the last line, starting with “Weighting” indicates what kind of statistic was stored in the term-document matrix. In this case we used the default, and simplest, option which simply records the count of the number of times a term appears across all of the documents in the corpus. You can peek at what the term-document matrix contains by using the inspect function:
inspect(tweetTDM)
Be prepared for a large amount of output. Remember the term “sparse” in the summary of the matrix? Sparse refers to the overwhelming number of cells that contain zero - indicating that the particular term does not appear in a given document. Most term document matrices are quite sparse. This one is 98% sparse because 36890/37500 = 0.98. In most cases we will need to cull or filter the term-document matrix for purposes of presenting or visualizing it. The tm package provides several methods for filtering out sparsely used terms, but in this example we are going to leave the heavy lifting to the word cloud package.
