Section 13.2 Basic String Operations
The first and most basic thing to do with strings is to see how long they are. The stringr package gives us the str_length() function to accomplish this task:
str_length(text)
[1] 130 136 136 128 98 75 131 139 85 157 107 49 75 139 136 136 [17] 136 72 73 136 157 123 160 142 142 122 122 122 122 134 82 87 [33] 89 118 94 74 103 91 136 136 151 136 139 135 70 122 122 136 [49] 123 111 83 136 137 85 154 114 117 98 125 138 107 92 140 119 [65] 92 125 84 81 107 107 73 73 138 63 137 139 131 136 120 124 [81] 124 114 78 118 138 138 116 112 101 94 153 79 79 125 125 102 [97] 102 139 138 153
There are all of the string lengths of the texts, reported to the command line. It is interesting to find that there are a few of them (like the very last one) that are longer than 140 characters:
tail(text,1)
[1] "RT @SolarFred: Hey, #solar & wind people. Tell @SpeakerBoehner and @Reuters that YOU have a green job and proud to be providing energy Independence to US"
As you can see, the tail() command works like the head() command except from the bottom up rather than the top down. So we have learned that under certain circumstances Twitter apparently does allow tweets longer than 140 characters. Perhaps the initial phrase “RT @SolarFred” does not count against the total. By the way “RT” stands for “retweet” and it indicates when the receiver of a tweet has passed along the same message to his or her followers.
We can glue the string lengths onto the respective rows in the dataframe by creating a new field/column:
tweetDF$textlen <- str_length(text)
After running this line of text, you should use the data browser in R-studio to confirm that the tweetDF now has a new column of data labeled “textlen”. You will find the new column all the way on the rightmost side of the dataframe structure. One peculiarity of the way R treats attached data is that you will not be able to access the new field without the $ notation unless you detach() and then again attach() the data frame. One advantage of grafting this new field onto our existing dataframe is that we can use it to probe the dataframe structure:
detach(tweetDF)
attach(tweetDF)
tweetDF[textlen>140, "text"]
[1] "RT @andyschonberger: Exciting (and tempting) to see #EVs all over the #GLS12 show. Combine EVs w #solar generation and we have a winner! http://t.co/NVsfq4G3"
We’ve truncated the output to save space, but in the data we are using here, there were nine tweets with lengths greater than 140. Not all of them had “RT” in them, though, so the mystery remains. An important word about the final command line above, though: We’re using the square brackets notation to access the elements of tweetDF. In the first entry, “textlen>140”, we’re using a conditional expression to control which rows are reported. Only those rows where our new field “textlen” contains a quantity larger than 140 will be reported to the output. In the second entry within square brackets, “text” controls which columns are reported onto the output. The square bracket notation is extremely powerful and sometimes a little unpredictable and confusing, so it is worth experimenting with. For example, how would you change that last command above to report all of the columns/fields for the matching rows? Or how would you request the “screenName” column instead of the “text” column? What would happen if you substituted the number 1 in place of “text” on that command?
The next common task in working with strings is to count the number of words as well as the number of other interesting elements within the text. Counting the words can be accomplished in several ways. One of the simplest ways is to count the separators between the words - these are generally spaces. We need to be careful not to over count, if someone has mistakenly typed two spaces between a word, so let’s make sure to take out doubles. The str_replace_all() function from stringr can be used to accomplish this:
tweetDF$modtext <- str_replace_all(text," "," ")
tweetDF$textlen2 <- str_length(tweetDF$modtext)
detach(tweetDF)
attach(tweetDF)
tweetDF[textlen != textlen2,]
The first line above uses the str_replace_all() function to substitute the one string in place of another as many times as the matching string appears in the input. Three arguments appear on the function above: the first is the input string, and that is tweetDF$text (although we’ve referred to it just as “text because the dataframe is attached). The second argument is the string to look for and the third argument is the string to substitute in place of the first. Note that here we are asking to substitute one space any time that two in a row are found. Almost all computer languages have a function similar to this, although many of them only supply a function that replaces the first instance of the matching string.
In the second command we have calculated a new string length variable based on the length of the strings where the substitutions have occurred. We preserved this in a new variable/field/column so that we can compare it to the original string length in the final command. Note the use of the bracket notation in R to address a certain subset of rows based on where the inequality is true. So here we are looking for a report back of all of the strings whose lengths changed. In the tweet data we are using here, the output indicated that there were seven strings that had their length reduced by the elimination of duplicate spaces.
Now we are ready to count the number of words in each tweet using the str_count() function. If you give it some thought, it should be clear that generally there is one more word than there are spaces. For instance, in the sentence, “Go for it,” there are two spaces but three words. So if we want to have an accurate count, we should add one to the total that we obtain from the str_count() function:
tweetDF$wordCount<-(str_count(modtext," ") + 1)
detach(tweetDF)
attach(tweetDF)
mean(wordCount)
[1] 14.24
In this last command, we’ve asked R to report the mean value of the vector of word counts, and we learn that on average a tweet in our dataset has about 14 words in it.
Next, let’s do a bit of what computer scientists (and others) call “parsing.” Parsing is the process of dividing a larger unit, like a sentence, into smaller units, like words, based on some kind of rule. In many cases, parsing requires careful use of pattern matching. Most computer languages accomplish pattern matching through the use of a strategy called “regular expressions.” A regular expression is a set of symbols used to match patterns. For example, [a-z] is used to match any lowercase letter and the asterisk is used to represent a sequence of zero or more characters. So the regular expression “[a-z]*” means, “match a sequence of zero or more lowercase characters.
If we wanted to parse the retweet sequence that appears at the beginning of some of the tweets, we might use a regular expression like this: “RT @[a-z,A-Z]*: ”. Each character up tot he square bracket is a “literal” that has to match exactly. Then the “[a-z,A-Z]*” lets us match any sequence of uppercase and lowercase characters. Finally, the “: ” is another literal that matches the end of the sequence. You can experiment with it freely before you commit to using a particular expression, by asking R to echo the results to the command line, using the function str_match() like this:
str_match(modtext,"RT @[a-z,A-Z]*: ")
Once you are satisfied that this expression matches the retweet phrases properly, you can commit the results to a new column/field/variable in the dataframe:
tweetDF$rt <- str_match(modtext,"RT @[a-z,A-Z]*: ")
detach(tweetDF)
attach(tweetDF)
Now you can review what you found by echoing the new variable “rt” to the command line or by examining it in R-studio’s data browser:
head(rt, 10)
[,1]
[1,] NA
[2,] NA
[3,] NA
[4,] NA
[5,] NA
[6,] NA
[7,] NA
[8,] "RT @SEIA: "
[9,] NA
[10,] "RT @andyschonberger: "
This may be the first time we have seen the value “NA.” In R, NA means that there is no value available, in effect that the location is empty. Statisticians also call this missing data. These NAs appear in cases where there was no match to the regular expression that we provided to the function str_match(). So there is nothing wrong here, this is an expected outcome of the fact that not all tweets were retweets. If you look carefully, though, you will see something else that is interesting.
R is trying to tell us something with the bracket notation. At the top of the list there is a notation of [,1] which signifies that R is showing us the first column of something. Then, each of the entries looks like [#,] with a row number in place of # and an empty column designator, suggesting that R is showing us the contents of a row, possibly across multiple columns. This seems a bit mysterious, but a check of the documentation for str_match() reveals that it returns a matrix as its result. This means that tweetDF$rt could potentially contain its own rectangular data object: In effect, the variable rt could itself contain than one column!
In our case, our regular expression is very simple and it contains just one chunk to match, so there is only one column of new data in tweetDF$rt that was generated form using str_match(). Yet the full capability of regular expressions allows for matching a whole sequence of chunks, not just one, and so str_match() has set up the data that it returns to prepare for the eventuality that each row of tweetDF$rt might actually have a whole list of results.
If, for some reason, we wanted to simplify the structure of tweetDF$rt so that each element was simply a single string, we could use this command:
tweetDF$rt <- tweetDF$rt[ ,1]
This assigns to each element of tweetDF$rt the contents of the first column of the matrix. If you run that command and reexamine tweetDF$rt with head() you will find the simplified structure: no more column designator.
For us to be able to make some use of the retweet string we just isolated, we probably should extract just the “screenname” of the individual whose tweet got retweeted. A screenname in Twitter is like a username, it provides a unique identifier for each person who wants to post tweets. An individual who is frequently retweeted by others may be more influential because their postings reach a wider audience, so it could be useful for us to have a listing of all of the screennames without the extraneous stuff. This is easy to do with str_replace(). Note that we used str_replace_all() earlier in the chapter, but we don’t need it here, because we know that we are going to replace just one instance of each string:
tweetDF$rt<-str_replace(rt, "RT @","")
tweetDF$rt<-str_replace(rt,": ","")
tail(rt, 1)
[,1]
[100,] "SolarFred"
tweetDF$rt <- tweetDF$rt[ ,1]
In the first command, we substitute the empty string in place of the four character prefix “RT @”, while in the second command we substitute the empty string in place of the two character suffix “: “. In each case we assign the resulting string back to tweetDF$rt. You may be wondering why sometimes we create a new column or field when we calculate some new data while other times we do not. The golden rule with data columns is never to mess with the original data that was supplied. When you are working ona “derived” column, i.e., one that is calculated from other data, it may require several intermediate steps to get the data looking the way you want. In this case, rt is a derived column that we extracted from the text field of the tweet and our goal was to reduce it to the bare screenname of the individual whose post was retweeted. So these commands, which successfully overwrite rt with closer and closer versions of what we wanted, were fair game for modification.
You may also have noticed the very last command. It seems that one of our steps, probably the use of str_match() must have “matrix-ized” our data again, so we use the column trick that appeared earlier in this chapter to flatten the matrix back to a single column of string data.
This would be a good point to visualize what we have obtained. Here we introduce two new functions, one which should seem familiar and one that is quite new:
