(LC1.3) What does any ONE row in this
flights dataset refer to?
-
A. Data on an airline
-
B. Data on a flight
-
C. Data on an airport
-
D. Data on multiple flights
Section 1.4 Explore your first datasets
Let’s put everything we’ve learned so far into practice and start exploring some real data! Data comes to us in a variety of formats, from pictures to text to numbers. Throughout this book, we’ll focus on datasets that are saved in “spreadsheet”-type format. This is probably the most common way data are collected and saved in many fields. Remember from Subsection 1.2.1 that these “spreadsheet”-type datasets are called data frames in R. We’ll focus on working with data saved as data frames throughout this book.
Let’s first load all the packages needed for this chapter, assuming you’ve already installed them. Read Section 1.3 for information on how to install and load R packages if you haven’t already.
library(nycflights23)
library(dplyr)
library(knitr)
At the beginning of all subsequent chapters in this book, we’ll always have a list of packages that you should have installed and loaded in order to work with that chapter’s R code.
Subsection 1.4.1 nycflights23 package
Many of us have flown on airplanes or know someone who has. Air travel has become an ever-present aspect of many people’s lives. If you look at the Departures flight information board at an airport, you will frequently see that some flights are delayed for a variety of reasons. Are there ways that we can understand the reasons that cause flight delays?
We’d all like to arrive at our destinations on time whenever possible. (Unless you secretly love hanging out at airports. If you are one of these people, pretend for a moment that you are very much anticipating being at your final destination.) Throughout this book, we’re going to analyze data related to all domestic flights departing from one of New York City’s three main airports in 2023: Newark Liberty International (EWR), John F. Kennedy International (JFK), and LaGuardia Airport (LGA). We’ll access this data using the
nycflights23 R package, which contains five datasets saved in five data frames:
-
flights: Information on all flights. -
airlines: A table matching airline names and their two-letter International Air Transport Association (IATA) airline codes (also known as carrier codes) for 14 airline companies. For example, “DL” is the two-letter code for Delta. -
planes: Information about each of the physical aircraft used. -
weather: Hourly meteorological data for each of the three NYC airports. This data frame has rows roughly corresponding to the \(365 \times 24 \times 3 = 26{,}280\) possible hourly measurements one can observe at three locations over the course of a year. -
airports: Names, codes, and locations of the domestic destinations.
The
nycflights23 package is an updated version of the classic nycflights13 R package. nycflights23 was authored by ModernDive co-author Chester Ismay using the anyflights R package developed by Simon Couch. Simon granted permission to the ModernDive team to create nycflights23 and submit the package to CRAN.
Subsection 1.4.2 flights data frame
We’ll begin by exploring the
flights data frame and get an idea of its structure. Run the following code in your console, either by typing it or by cutting-and-pasting it. It displays the contents of the flights data frame in your console. Note that depending on the size of your monitor, the output may vary slightly.
flights
# A tibble: 435,352 Ă— 19
year month day dep_time sched_dep_time dep_delay arr_time sched_arr_time
<int> <int> <int> <int> <int> <dbl> <int> <int>
1 2023 1 1 1 2038 203 328 3
2 2023 1 1 18 2300 78 228 135
3 2023 1 1 31 2344 47 500 426
4 2023 1 1 33 2140 173 238 2352
5 2023 1 1 36 2048 228 223 2252
6 2023 1 1 503 500 3 808 815
7 2023 1 1 520 510 10 948 949
8 2023 1 1 524 530 -6 645 710
9 2023 1 1 537 520 17 926 818
10 2023 1 1 547 545 2 845 852
# â„ą 435,342 more rows
# â„ą 11 more variables: arr_delay <dbl>, carrier <chr>, flight <int>,
# tailnum <chr>, origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>,
# hour <dbl>, minute <dbl>, time_hour <dttm>
Let’s unpack this output:
-
# A tibble: 435,352 Ă— 19: Atibbleis a specific kind of data frame in R. This particular data frame has-
435,352 rows corresponding to different observations. Here, each observation is a flight.
-
19 columns corresponding to variables describing each observation.
-
-
year,month,day,dep_time,sched_dep_time,dep_delay, andarr_timeare the different columns, in other words, the different variables of this dataset. -
We then have a preview of the first 10 rows of observations corresponding to the first 10 flights. R is only showing the first 10 rows, because if it showed all 435,352 rows, it would overwhelm your screen.
-
... with 435,342 more rows and 11 more variables:indicating to us that 435,342 more rows of data and 11 more variables could not fit in this screen.
Unfortunately, this output does not allow us to explore the data very well, but it does give a nice preview. Let’s look at some different ways to explore data frames.
Subsection 1.4.3 Exploring data frames
There are many ways to get a feel for the data contained in a data frame such as
flights. We present three functions that take as their “argument” (their input) a data frame and a fourth method for exploring one column of a data frame:
-
Using the
View()function, which brings up RStudio’s built-in data viewer. -
Using the
glimpse()function, which is included in thedplyrpackage. -
Using the
kable()function, which is included in theknitrpackage. -
Using the
$“extraction operator,” which is used to view a single variable.
1.
View():
Run
View(flights) in your console in RStudio, either by typing it or cutting-and-pasting it into the console pane. Explore this data frame in the resulting pop up viewer. You should get into the habit of viewing any data frames you encounter. Note the uppercase V in View(). R is case-sensitive, so you’ll get an error message if you run view(flights) instead of View(flights).
Checkpoint 1.4.1. Learning Check.
By running
View(flights), we can explore the different variables listed in the columns. Observe that there are many different types of variables. Some of the variables like distance, day, and arr_delay are what we will call quantitative variables. These variables are numerical in nature. Other variables here are categorical.
If you look in the leftmost column of the
View(flights) output, you’ll see a column of numbers. These are the row numbers of the dataset. Glancing across a row with the same number, say row 5, you can get an idea of what each row represents. This allows you to identify what object is being described in a given row by taking note of the values of the columns in that specific row. This is often called the observational unit. The observational unit in this example is an individual flight departing from New York City in 2023. You can identify the observational unit by determining what “thing” is being measured or described by each of the variables. We’ll talk more about observational units in Subsection 1.4.4 on identification and measurement variables.
2.
glimpse():
The second way we’ll cover to explore a data frame is using the
glimpse() function included in the dplyr package. Thus, you can only use the glimpse() function after you’ve loaded the dplyr package by running library(dplyr). This function provides us with an alternative perspective for exploring a data frame than the View() function:
glimpse(flights)
Rows: 435,352 Columns: 19 $ year <int> 2023, 2023, 2023, 2023, 2023, 2023, 2023, 2023, 2023, 2… $ month <int> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1… $ day <int> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1… $ dep_time <int> 1, 18, 31, 33, 36, 503, 520, 524, 537, 547, 549, 551, 5… $ sched_dep_time <int> 2038, 2300, 2344, 2140, 2048, 500, 510, 530, 520, 545, … $ dep_delay <dbl> 203, 78, 47, 173, 228, 3, 10, -6, 17, 2, -10, -9, -7, -… $ arr_time <int> 328, 228, 500, 238, 223, 808, 948, 645, 926, 845, 905, … $ sched_arr_time <int> 3, 135, 426, 2352, 2252, 815, 949, 710, 818, 852, 901, … $ arr_delay <dbl> 205, 53, 34, 166, 211, -7, -1, -25, 68, -7, 4, -13, -14… $ carrier <chr> "UA", "DL", "B6", "B6", "UA", "AA", "B6", "AA", "UA", "… $ flight <int> 628, 393, 371, 1053, 219, 499, 996, 981, 206, 225, 800,… $ tailnum <chr> "N25201", "N830DN", "N807JB", "N265JB", "N17730", "N925… $ origin <chr> "EWR", "JFK", "JFK", "JFK", "EWR", "EWR", "JFK", "EWR",… $ dest <chr> "SMF", "ATL", "BQN", "CHS", "DTW", "MIA", "BQN", "ORD",… $ air_time <dbl> 367, 108, 190, 108, 80, 154, 192, 119, 258, 157, 164, 1… $ distance <dbl> 2500, 760, 1576, 636, 488, 1085, 1576, 719, 1400, 1065,… $ hour <dbl> 20, 23, 23, 21, 20, 5, 5, 5, 5, 5, 5, 6, 5, 6, 6, 6, 6,… $ minute <dbl> 38, 0, 44, 40, 48, 0, 10, 30, 20, 45, 59, 0, 59, 0, 0, … $ time_hour <dttm> 2023-01-01 20:00:00, 2023-01-01 23:00:00, 2023-01-01 2…
Observe that
glimpse() will give you the first few entries of each variable in a row after the variable name. In addition, the data type (see Subsection 1.2.1) of the variable is given immediately after each variable’s name inside < >. Here, int and dbl refer to “integer” and “double,” which are computer coding terminology for quantitative/numerical variables. “Doubles” take up twice the size to store on a computer compared to integers.
In contrast,
chr refers to “character,” which is computer terminology for text data. In most forms, text data, such as the carrier or origin of a flight, are categorical variables. The time_hour variable is another data type: dttm. These types of variables represent date and time combinations. However, we won’t work with dates and times in this book; we leave this topic for other data science books like Data Science: A First Introduction by Tiffany-Anne Timbers, Melissa Lee, and Trevor Campbell or R for Data Science [1].
Checkpoint 1.4.2. Learning Check.
(LC1.4) What are some other examples in this dataset of categorical variables? What makes them different than quantitative variables?
3.
kable():
The final way to explore the entirety of a data frame is using the
kable() function from the knitr package. Let’s explore the different carrier codes for all the airlines in our dataset two ways. Run both of these lines of code in the console:
airlines
# A tibble: 14 Ă— 2 carrier name <chr> <chr> 1 9E Endeavor Air Inc. 2 AA American Airlines Inc. 3 AS Alaska Airlines Inc. 4 B6 JetBlue Airways 5 DL Delta Air Lines Inc. 6 F9 Frontier Airlines Inc. 7 G4 Allegiant Air 8 HA Hawaiian Airlines Inc. 9 MQ Envoy Air 10 NK Spirit Air Lines 11 OO SkyWest Airlines Inc. 12 UA United Air Lines Inc. 13 WN Southwest Airlines Co. 14 YX Republic Airline
At first glance, it may not appear that there is much difference in the outputs. However, when using tools for producing reproducible reports such as R Markdown, the latter code produces output that is much more legible and reader-friendly. You’ll see us use this reader-friendly style in many places in the book when we want to print a data frame as a nice table.
4.
$ operator
Lastly, the
$ operator allows us to extract and then explore a single variable within a data frame. For example, run the following in your console:
airlines$name
[1] "Endeavor Air Inc." "American Airlines Inc." "Alaska Airlines Inc." [4] "JetBlue Airways" "Delta Air Lines Inc." "Frontier Airlines Inc." [7] "Allegiant Air" "Hawaiian Airlines Inc." "Envoy Air" [10] "Spirit Air Lines" "SkyWest Airlines Inc." "United Air Lines Inc." [13] "Southwest Airlines Co." "Republic Airline"
We used the
$ operator to extract only the name variable and return it as a vector of length 14. We’ll only be occasionally exploring data frames using the $ operator, instead favoring the View() and glimpse() functions.
Subsection 1.4.4 Identification and measurement variables
There is a subtle difference between the kinds of variables that you will encounter in data frames. There are identification variables and measurement variables. For example, let’s explore the
airports data frame by showing the output of glimpse(airports):
glimpse(airports)
Rows: 1,255 Columns: 8 $ faa <chr> "AAF", "AAP", "ABE", "ABI", "ABL", "ABQ", "ABR", "ABY", "ACK", "… $ name <chr> "Apalachicola Regional Airport", "Andrau Airpark", "Lehigh Valle… $ lat <dbl> 29.72750, 29.72250, 40.65210, 32.41130, 67.10630, 35.04020, 45.4… $ lon <dbl> -85.02750, -95.58830, -75.44080, -99.68190, -157.85699, -106.609… $ alt <dbl> 20, 79, 393, 1791, 334, 5355, 1302, 197, 47, 516, 221, 75, 18, 7… $ tz <dbl> -5, -6, -5, -6, -9, -7, -6, -5, -5, -6, -8, -5, -10, -6, -9, -6,… $ dst <chr> "A", "A", "A", "A", "A", "A", "A", "A", "A", "A", "A", "A", "A",… $ tzone <chr> "America/New_York", "America/Chicago", "America/New_York", "Amer…
The variables
faa and name are identification variables that uniquely identify each airport. faa provides the airport’s unique FAA code, while name gives its official name. These variables are used to uniquely identify each row in a data frame. The remaining variables (lat, lon, alt, tz, dst, tzone) are often called measurement or characteristic variables: variables that describe properties of each observational unit. For example, lat and long describe the latitude and longitude of each airport.
Furthermore, sometimes a single variable might not be enough to uniquely identify each observational unit: combinations of variables might be needed. While it is not an absolute rule, for organizational purposes it is considered good practice to have your identification variables in the leftmost columns of your data frame.
Checkpoint 1.4.3. Learning Check.
(LC1.5) What properties of each airport do the variables
lat, lon, alt, tz, dst, and tzone describe in the airports data frame? Take your best guess.
Checkpoint 1.4.4. Learning Check.
(LC1.6) Provide the names of variables in a data frame with at least three variables where one of them is an identification variable and the other two are not.
Subsection 1.4.5 Help files
Another nice feature of R are help files, which provide documentation for various functions and datasets. You can bring up help files by adding a
? before the name of a function or data frame and then run this in the console. You will then be presented with a page showing the corresponding documentation if it exists. For example, let’s look at the help file for the flights data frame.
?flights
The help file should pop up in the Help pane of RStudio. If you have questions about a function or data frame included in an R package, you should get in the habit of consulting the help file right away.
Checkpoint 1.4.5. Learning Check.
(LC1.7) Look at the help file for the
airports data frame. Revise your earlier guesses about what the variables lat, lon, alt, tz, dst, and tzone each describe.
