Section 1.5 The Tidyverse Ecosystem
With a solid understanding of tidy data and how tidy data fit into the data science life cycle, we’ll take a bit of time to introduce you to the tidyverse and tidyverse-adjacent packages that we’ll be teaching and using throughout this specialization. Taken together, these packages make up what we’re referring to as the tidyverse ecosystem. The purpose for the rest of this course is not for you to understand how to use each of these packages (that’s coming soon!), but rather to help you familiarize yourself with which packages fit into which part of the data science life cycle.
Note that we will describe the official tidyverse packages, as well as other packages that are tidyverse-adjacent. This means they follow the same conventions as the official tidyverse packages and work well within the tidy framework and structure of data analysis.
Subsection 1.5.1 Reading Data into R
After identifying a question that can be answered using data, there are many different ways in which the data you’ll want to use may be stored. Sometimes information is stored within an Excel spreadsheet. Other times, the data are in a table on a website that needs to be scraped. Or, in a CSV file. Each of these types of data files has their own structure, but R can work with all of them. To do so, however, requires becoming familiar with a few different packages. Here, we’ll discuss these packages briefly. In later courses in the specialization we’ll get into the details of what characterizes each file type and how to use each packages to read data into R.
Subsubsection 1.5.1.1 tibble
While not technically a package that helps read data into R,
tibble is a package that re-imagines the familiar R data.frame. It is a way to store information in columns and rows, but does so in a way that addresses problems earlier in the pipeline.
From the
tibble website:
A tibble, or tbl_df, is a modern reimagining of the data.frame, keeping what time has proven to be effective, and throwing out what is not. Tibbles are data.frames that are lazy and surly: they do less (i.e. they don’t change variable names or types, and don’t do partial matching) and complain more (e.g. when a variable does not exist). This forces you to confront problems earlier, typically leading to cleaner, more expressive code. Tibbles also have an enhanced print() method which makes them easier to use with large datasets containing complex objects.
In fact, when working with data using the tidyverse, you’ll get very comfortable working with tibbles.
Subsubsection 1.5.1.2 readr
readr is a package that users of the tidyverse use all the time. It helps read rectangular data into R. If you find yourself working with CSV files frequently, then you’ll find yourself using readr regularly.
According to the
readr website:
The goal ofreadris to provide a fast and friendly way to read rectangular data (like csv, tsv, and fwf). It is designed to flexibly parse many types of data found in the wild, while still cleanly failing when data unexpectedly changes. If you are new to readr, the best place to start is the data import chapter in R for data science.
Subsubsection 1.5.1.3 googlesheets4
googlesheets4 is a brilliant tidyverse-adjacent package that allows users to "access and manage Google spreadsheets from R." As more and more data is saved in the cloud (rather than on local computers), packages like googlesheets4 become invaluable. If you store data on Google Sheets or work with people who do, this package will be a lifesaver.
Subsubsection 1.5.1.4 readxl
Another package for working with tabular data is
readxl, which is specifically designed to move data from Excel into R. If many of your data files have the .xls or .xlsx extension, familiarizing yourself with this package will be helpful.
From the
readxl website:
The readxl package makes it easy to get data out of Excel and into R. Compared to many of the existing packages (e.g.gdata,xlsx,xlsReadWrite)readxlhas no external dependencies, so it’s easy to install and use on all operating systems. It is designed to work with tabular data.
Subsubsection 1.5.1.5 googledrive
Similar to
googlesheets4, but for interacting with file on Google Drive from R (rather than just Google Sheets), the tidyverse-adjacent googledrive package is an important package for working with data in R.
Subsubsection 1.5.1.6 haven
If you are a statistician (or work with statisticians), you’ve likely heard of the statistical packages SPSS, Stata, and SAS. Each of these has data formats for working with data that are compatible only with their platform. However, there is an R package that allows you to use read stored in these formats into R. For this, you’ll need
haven.
Subsubsection 1.5.1.7 jsonlite & xml2
Data stored on and retrieved from the Internet are often stored in one of the two most common semi-structured data formats: JSON or XML. We’ll discuss the details of these when we discuss how to use
jsonlite and xml2, which allow data in the JSON and XML formats, respectively, to be read into R. jsonlite helps extensively when working with Application Programming Interfaces (APIs) and xml2 is incredibly helpful when working with HTML.
Subsubsection 1.5.1.8 rvest
If you are hoping to scrape data from a website,
rvest is a package with which you’ll want to become familiar. It allows you to scrape information directly from web pages on the Internet.
Subsubsection 1.5.1.9 httr
Companies that share data with users often do so using Application Programming Interfaces or APIs. We’ve mentioned that these data are often stored in the JSON format, requiring packages like jsonlite to work with the data. However, to retrieve the data in the first place, you’ll use
httr. This package helps you interact with modern web APIs.
Subsection 1.5.2 Data Tidying
There are loads of ways in which data and information are stored on computers and the Internet. We’ve reviewed that there are a number of packages that you’ll have to use depending on the type of data you need to get into R. However, once the data are in R, the next goal is to tidy the data. This process is often referred to as data wrangling or data tidying. Regardless of what you call it, there are a number of packages that will help you take the untidy data you just read into R and convert it into the flexible and usable tidy data format.
Subsubsection 1.5.2.1 dplyr
The most critical package for wrangling data is
dplyr. Its release completely transformed the way many R users write R code and work with data, greatly simplifying the process.
According to the
dplyr website:
dplyr is a grammar of data manipulation, providing a consistent set of verbs that help you solve the most common data manipulation challenges.
dplyr is built around five primary verbs (mutate, select, filter, summarize, and arrange) that help make the data wrangling process simpler. This specialization will cover these verbs among other functionality within the dplyr package.
Subsubsection 1.5.2.2 tidyr
Like
dplyr, tidyr is a package with the primary goal of helping users take their untidy data and make it tidy.
According to the
tidyr website:
The goal oftidyris to help you create tidy data. Tidy data is data where: each variable is in a column, each observation is a row, and each value is a cell. Tidy data describes a standard way of storing data that is used wherever possible throughout thetidyverse. If you ensure that your data is tidy, you’ll spend less time fighting with the tools and more time working on your analysis.
Subsubsection 1.5.2.3 janitor
In addition to
dplyr and tidyr, a common tidyverse-adjacent package used to clean dirty data and make users life easier while doing so is janitor.
According to the
janitor website:
janitor has simple functions for examining and cleaning dirty data. It was built with beginning and intermediate R users in mind and is optimized for user-friendliness. Advanced R users can already do everything covered here, but with janitor they can do it faster and save their thinking for the fun stuff.
Subsubsection 1.5.2.4 forcats
R is known for its ability to work with categorical data (called factors); however, they have historically been more of a necessary evil than a joy to work with. Due to the frustratingly hard nature of working with factors in R, the
forcats package developers set out to make working with categorical data simpler.
According to the
forcats website:
The goal of the forcats package is to provide a suite of tools that solve common problems with factors, including changing the order of levels or the values.
Subsubsection 1.5.2.5 stringr
Similar to
forcats, but for strings, the stringr package makes common tasks simple and streamlined. Working with this package becomes easier with some knowledge of regular expressions, which we’ll cover in this specialization.
According to the
string website:
Strings are not glamorous, high-profile components of R, but they do play a big role in many data cleaning and preparation tasks. The stringr package provide a cohesive set of functions designed to make working with strings as easy as possible.
Subsubsection 1.5.2.6 lubridate
The final common package dedicated to working with a specific type of variable is
lubridate (a tidyverse-adjacent package), which makes working with dates and times simpler. Working with dates and times has historically been difficult due to the nature of our calendar, with its varying number of days per month and days per year, and due to time zones, which can make working with times infuriating. The lubridate developers aimed to make working with these types of data simpler.
According to the
lubridate website:
Date-time data can be frustrating to work with in R. R commands for date-times are generally unintuitive and change depending on the type of date-time object being used. Moreover, the methods we use with date-times must be robust to time zones, leap days, daylight savings times, and other time related quirks, and R lacks these capabilities in some situations.Lubridatemakes it easier to do the things R does with date-times and possible to do the things R does not.
Subsubsection 1.5.2.7 glue
The
glue tidyverse-adjacent package makes working with interpreted string literals simpler. We’ll discuss this package in detail in this specialization.
Subsubsection 1.5.2.8 skimr
After you’ve got your data into a tidy format and all of your variable types have been cleaned, the next step is often summarizing your data. If you’ve used the
summary() function in R before, you’re going to love skimr, which summarizes entire data frames for you in a tidy manner.
skimrprovides a frictionless approach to summary statistics which conforms to the principle of least surprise, displaying summary statistics the user can skim quickly to understand their data.
Subsubsection 1.5.2.9 tidytext
While working with factors, numbers, and small strings is common in R, longer texts have historically been analyzed using approaches outside of R. However, once the
tidyverse-adjacent package tidytext was developed, R had a tidy approach to analyzing text data, such as novels, news stories, and speeches.
According to the
tidytext website:
In this package, we provide functions and supporting data sets to allow conversion of text to and from tidy formats, and to switch seamlessly between tidy tools and existing text mining packages.
Subsubsection 1.5.2.10 purrr
The final package we’ll discuss here for data tidying is
purrr, a package for working with functions and vectors in a tidy format. If you find yourself writing for loops to iterate through data frames, then purrr will save you a ton of time!
According to the
purrr website:
purrrenhances R’s functional programming (FP) toolkit by providing a complete and consistent set of tools for working with functions and vectors. If you’ve never heard of FP before, the best place to start is the family ofmap()functions which allow you to replace many for loops with code that is both more succinct and easier to read.
Subsection 1.5.3 Data Visualization
Data Visualization is a critical piece of any data science project. Once you have your data in a tidy format, you’ll first explore your data, often generating a number of basic plots to get a better understanding of your dataset. Then, once you’ve carried out your analysis, creating a few detailed and well-designed visualizations to communicate your findings is necessary. Fortunately, there are a number of helpful packages to create visualizations.
Subsubsection 1.5.3.1 ggplot2
The most critical package when it comes to plot generation in data visualization is
ggplot2, a package that allows you to quickly create plots and meticulously customize them depending on your needs.
According to the
ggplot2 website:
ggplot2is a system for declaratively creating graphics, based on The Grammar of Graphics. You provide the data, tell ggplot2 how to map variables to aesthetics, what graphical primitives to use, and it takes care of the details.
This package will be covered in a great amount of detail in this specialization, largely due to the fact that you’ll find yourself using it all the time. Having a strong foundation of how to use
ggplot2 is incredibly important for any data science project.
Subsubsection 1.5.3.2 kableExtra
While the ability to make beautiful and informative plots is essential, tables can be incredibly effective vehicles for the conveyance of information. Customizing plots can be done using the
tidyverse-adjacent kableExtra package, which is built on top of the knitr() function from the kable package, which generates basic tables. kableExtra allows complex and detailed tables to be built using a ggplot2-inspired syntax.
Subsubsection 1.5.3.3 ggrepel
Due to the popularity of
ggplot2, there are a number of tidyverse-adjacent packages built on top of and within the ggplot2 framework. ggrepel is one of theses packages that "provides geoms for ggplot2 to repel overlapping text labels."
Subsubsection 1.5.3.4 cowplot
cowplot is another tidyverse-adjacent package that helps you to polish and put finishing touches on your plots.
Thecowplotpackage provides various features that help with creating publication-quality figures, such as a set of themes, functions to align plots and arrange them into complex compound figures, and functions that make it easy to annotate plots and or mix plots with images.
Subsubsection 1.5.3.5 patchwork
The
patchwork package (also tidyverse-adjacent) is similar to cowplot and is an excellent option for combining multiple plots together.
Subsubsection 1.5.3.6 gganimate
Beyond static images, there are times when we want to display changes over time or other visualizations that require animation. The
gganimate package (also tidyverse-adjacent) enables animation on top of ggplot2 plots.
According to the
gganimate website:
gganimateextends the grammar of graphics as implemented byggplot2to include the description of animation. It does this by providing a range of new grammar classes that can be added to the plot object in order to customize how it should change with time.
Subsection 1.5.4 Data Modeling
Once data have been read in, tidied, and explored, the last step to answering your question and before communicating your findings is data modeling. In this step, you’re carrying out an analysis to answer your question of interest. There are a number of helpful suites of R packages.
Subsubsection 1.5.4.1 The tidymodels ecosystem
When it comes to predictive analyses and machine learning, there is a suite of packages called
tidymodels.
The
tidymodels ecosystem contains packages for data splitting, preprocessing, model fitting, performance assessment, and more.
The great advantage is that is allows for users to use predictive algorithms that were written across dozens of different R packages and makes them all usable with a standard syntax.
Subsubsection 1.5.4.2 Inferential packages: broom and infer
When carrying out inferential analyses the
tidymodels suite of tidyverse-adjacent packages is essential.
The
broom package which is part of the tidymodels suite takes statistical analysis objects from R (think the output of your lm() function) and converts them into a tidy data format. This makes obtaining the information you’re most interested in from your statistical models much simpler.
Similarly,
infer sets out to perform statistical inference using a standard statistical grammar. Historically, the syntax varied widely from one statistical test to the next in R. The infer package sets out to standardize the syntax, regardless of the test.
Subsubsection 1.5.4.3 tidyverts: tsibble, feasts, and fable
While many datasets are like a snapshot in time - survey data collected once or contact information from a business - time series data are unique. Time series datasets represent changes over time and require computational approaches that are unique to this fact. A suite of tidyverse-adjacent packages called
tidyverts have been developed to enable and simplify tidy time series analyses in R. Among these are tsibble, feasts, and fable.
A
tsibble is the time-series version of a tibble in that is provides the data.frame-like structure most useful for carrying out tidy time series analyses.
According to the
tsibble website:
The tsibble package provides a data infrastructure for tidy temporal data with wrangling tools. Adhering to the tidy data principles, tsibble is an explicit data- and model-oriented object.
The
feasts package is most helpful when it comes to the modeling step in time series analyses.
According to the
feasts website:
Feasts provides a collection of tools for the analysis of time series data. The package name is an acronym comprising of its key features: Feature Extraction And Statistics for Time Series.
The
fable package is most helpful when it comes to the modeling step in forecasting analyses.
According to the
fable website:
The R package fable provides a collection of commonly used univariate and multivariate time series forecasting models including exponential smoothing via state space models and automatic ARIMA modelling. These models work within the fable framework, which provides the tools to evaluate, visualize, and combine models in a workflow consistent with the tidyverse.
