Introduction to Data Science Version 3

Now let’s get some data from Twitter. First, tell the twitteR package that you want to use your shiny new credentials:

registerTwitterOAuth(credential)

[1] TRUE

The return value of TRUE shows that the credential is working and ready to help you get data from Twitter. Subsequent commands using the twitteR package will pass through the authorized application interface.

The twitteR package provides a function called searchTwitter() that allows us to retrieve some recent tweets based on a search term. Twitter users have invented a scheme for organizing their tweets based on subject matter. This system is called "hashtags" and is based on the use of the hashmark character (#) followed by a brief text tag. For example, fans of Oprah Winfrey use the tag #oprah to identify their tweets about her. We will use the searchTwitter() function to search for hashtags about global climate change. The website hashtags.org lists a variety of hashtags covering a range of contemporary topics. You can pick any hashtag you like, as long as there are a reasonable number of tweets that can be retrieved. The searchTwitter() function also requires specifying the maximum number of tweets that the call will return. For now we will use 500, although you may find that your request does not return that many. Here’s the command:

tweetList <- searchTwitter("#climate", n=500)

As above, if you are on Windows, and you had to get new certificates, you may have to use this command:

tweetList <- searchTwitter("#climate", n=500, cainfo="cacert.pem")

Depending upon the speed of your Internet connection and the amount of traffic on Twitter’s servers, this command may take a short while for R to process. Now we have a new data object, tweetList, that presumably contains the tweets we requested. But what is this data object? Let’s use our R diagnostics to explore what we have gotten:

> mode(tweetList)
[1] "list"

Hmm, this is a type of object that we have not encountered before. In R, a list is an object that contains other data objects, and those objects may be a variety of different modes/types. Contrast this definition with a vector: A vector is also a kind of list, but with the requirement that all of the elements in the vector must be in the same mode/type. Actually, if you dig deeply into the definitions of R data objects, you may realize that we have already encountered one type of list: the dataframe. Remember that the dataframe is a list of vectors, where each vector is exactly the same length. So a dataframe is a particular kind of list, but in general lists do not have those two restrictions that dataframes have (i.e., that each element is a vector and that each vector is the same length).

So we know that tweetList is a list, but what does that list contain? Let’s try using the str() function to uncover the structure of the list:

str(tweetList)

Whoa! That output scrolled right off the screen. A quick glance shows that it is pretty repetitive, with each 20 line block being quite similar. So let’s use the head() function to just examine the first element of the list. The head() function allows you to just look at the first few elements of a data object. In this case we will look just at the first list element of the tweetList list. The command, also shown on the screen shot below, is:

str(head(tweetList,1))

Looks pretty messy, but is simpler than it may first appear. Following the line “List of 1,” there is a line that begins “$ :Reference class” and then the word ‘status’ in single quotes. In Twitter terminology a “status” is a single tweet posting (it supposedly tells us the “status” of the person who posted it). So the author of the R twitteR package has created a new kind of data object, called a ‘status’ that itself contains 10 fields. The fields are then listed out. For each line that begins with “..$” there is a field name and then a mode or data type and then a taste of the data that the field contains.

So, for example, the first field, called “text” is of type “chr” (which means character/text data) and the field contains the string that starts with, “Get the real facts on gas prices.” You can look through the other fields and see if you can make sense of them. There are two other data types in there: “logi” stands for logical and that is the same as TRUE/FALSE; “POSIXct” is a format for storing the calendar date and time. (If you’re curious, POSIX is an old unix style operating system, where the current date and time were stored as the number of seconds elapsed since 12 midnight on January 1, 1970.) You can see in the “created” field that this particular tweet was created on April 5, 2012 one second after 2:10 PM. It does not show what time zone, but a little detective work shows that all Twitter postings are coded with “coordinated universal time” or what is usually abbreviated with UTC.

One last thing to peek at in this data structure is about seven lines from the end, where it says, “and 33 methods...” In computer science lingo a “method” is an operation/activity/procedure that works on a particular data object. The idea of a method is at the heart of so called “object oriented programming.” One way to think of it is that the data object is the noun, and the methods are all of the verbs that work with that noun. For example you can see the method getCreated in the list: If you use the method getCreated on a reference object of class ‘status’, the method will return the creation time of the tweet.

If you try running the command:

str(head(tweetList,2))

you will find that the second item in the tweetList list is structured exactly like the first, with the only difference being the specific contents of the fields. You can also run:

length(tweetList)

to find out how many items are in your list. The list obtained for this exercise was a full 500 items long. So we have 500 complex items in our list, but every item had exactly the same structure, with 10 fields in it and a bunch of other stuff too. That raises a thought: tweetList could be thought of as a 500 row structure with 10 columns! That means that we could treat it as a dataframe if we wanted to (and we do, because this makes handling these data much more convenient as you found in the “Rows and Columns” chapter).

Happily, we can get some help from R in converting this list into a dataframe. Here we will introduce four powerful new R functions: as(), lapply(), rbind(), and do.call(). The first of these, as(), performs a type coercion: in other words it changes one type to another type. The second of these, lapply(), applies a function onto all of the elements of a list. In the command below, lapply(tweetList, as.data.frame), applies the as.data.frame() coercion to each element in tweetList. Next, the rbind() function “binds” together the elements that are supplied to it into a row-by-row structure. Finally, the do.call() function executes a function call, but unlike just running the function from the console, allows for a variable number of arguments to be supplied to the function. The whole command we will use looks like this:

tweetDF <- do.call("rbind", lapply(tweetList, as.data.frame))

You might wonder a few things about this command. One thing that looks weird is "rbind" in double quotes. This is the required method of supplying the name of the function to do.call(). You might also wonder why we needed do.call() at all. Couldn’t we have just called rbind() directly from the command line? You can try it if you want, and you will find that it does provide a result, but not the one you want. The difference is in how the arguments to rbind() are supplied to it: if you call it directly, lapply() is evaluated first, and it forms a single list that is then supplied to rbind(). In contrast, by using do.call(), all 500 of the results of lapply() are supplied to rbind() as individual arguments, and this allows rbind() to create the nice rectangular dataset that we will need. The advantage of do.call() is that it will set up a function call with a variable number of arguments in cases where we don’t know how many arguments will be supplied at the time when we write the code.

If you run the command above, you should see in the upper right hand pane of R-studio a new entry in the workspace under the heading of “Data.” For the example we are running here, the entry says, “500 obs. of 10 variables.” This is just what we wanted, a nice rectangular data set, ready to analyze. Later on, we may need more than one of these data sets, so let’s create a function to accomplish the commands we just ran:

# TweetFrame() - Return a dataframe based on a
#                search of Twitter
TweetFrame<-function(searchTerm, maxTweets)
{
twtList<-searchTwitter(searchTerm,n=maxTweets)
return(do.call("rbind",
               lapply(twtList,as.data.frame)))
}

There are three good things about putting this code in a function. First, because we put a comment at the top of the function, we will remember in the future what this code does. Second, if you test this function you will find out that the variable twtList that is created in the code above does not stick around after the function is finished running. This is the result of what computer scientists call “variable scoping.” The variable twtList only exists while the TweetFrame() function is running. Once the function is done, twtList evaporates as if it never existed. This helps us to keep our workspace clean and avoid collecting lots of intermediate variables that are not reused.

The last and best thing about this function is that we no longer have to remember the details of the method for using do.call(), rbind(), lapply(), and as.data.frame() because we will not have to retype these commands again: we can just call the function whenever we need it. And we can always go back and look at the code later. In fact, this would be a good reason to put in a comment just above the return() function. Something like this:

# as.data.frame() coerces each list element into a row
# lapply() applies this to all of the elements in twtList
# rbind() takes all of the rows and puts them together
# do.call() gives rbind() all the rows as individual elements

Now, whenever we want to create a new data set of tweets, we can just call TweetFrame() from the R console command line like this:

lgData <- TweetFrame("#ladygaga", 250)

This command would give us a new dataframe lgData all ready to analyze, based on the supplied search term and maximum number of tweets.