distinct().
Keep only distinct rows from a data frame.
Section 2.5 Manipulating Data in tidyverse
We will now be utilizing another one of our packages that we installed and loaded, the
palmerpenguins package. It has the dataset penguins that we are going to use. If you use the helper function and write ?palmerpenguins or ?penguins you will find a description of the data (hint: it’s data related to penguins).
Each of the subsections touches on a different aspect. Let’s start by finding out more about our data.
Subsection 2.5.1 Distinct
We have a dataset that we don’t quite know too much about. Let’s first find out some information about it using what we learned from base R in the last chapter.
ncol(penguins) # Finding out the number of columns
nrow(penguins) # Finding out the number of rows
colnames(penguins) # Finding out the names of the columns
[1] 8 [1] 344 [1] "species" "island" "bill_length_mm" [4] "bill_depth_mm" "flipper_length_mm" "body_mass_g" [7] "sex" "year"
Great, we now know that there are 344 rows of data, 8 columns, and all the column names. We know that there are different types of penguins, and that the species is a column in this dataset. We could scroll through every single row and try to eyeball the distinct types of penguin species in this dataset. Or we could use the
distinct() function to get this information.
penguins |> distinct(species)
# A tibble: 3 × 1 species <fct> 1 Adelie 2 Gentoo 3 Chinstrap
That one line of code filtered through all 344 rows and figured out that there are only three different species of penguins in the
penguins dataset. But it doesn’t stop there - you can also use the distinct() command to see distinct combinations of data too. For example:
penguins |> distinct(species, island)
# A tibble: 5 × 2 species island <fct> <fct> 1 Adelie Torgersen 2 Adelie Biscoe 3 Adelie Dream 4 Gentoo Biscoe 5 Chinstrap Dream
Very interesting! We can see that the “Adelie” species is found on three different islands, whereas the other two species are only found on one island each (and different islands). With just a comma, you can find the distinct combinations between data in any of your columns.
Now, what if we don’t need all of our columns?
Subsection 2.5.2 Select
For now, let’s pretend like we only want to work with the columns species, island, bill_length_mm, and body_mass_g. How do we select for only those? We can use the
select() command!
select().
Changes which columns exist.
penguins |> select(species, island, bill_length_mm, body_mass_g)
# A tibble: 344 × 4 species island bill_length_mm body_mass_g <fct> <fct> <dbl> <int> 1 Adelie Torgersen 39.1 3750 2 Adelie Torgersen 39.5 3800 3 Adelie Torgersen 40.3 3250 4 Adelie Torgersen NA NA 5 Adelie Torgersen 36.7 3450 6 Adelie Torgersen 39.3 3650 7 Adelie Torgersen 38.9 3625 8 Adelie Torgersen 39.2 4675 9 Adelie Torgersen 34.1 3475 10 Adelie Torgersen 42 4250 # ℹ 334 more rows
Instead of using the actual column names, we can use the column indices - which number (in order). Below is an example using the column indexes (hint: it uses a comma and a colon.)
# This is saying let's take the first column, and then every column from 3 to 5
penguins |> select(1,3:5)
# A tibble: 344 × 4 species bill_length_mm bill_depth_mm flipper_length_mm <fct> <dbl> <dbl> <int> 1 Adelie 39.1 18.7 181 2 Adelie 39.5 17.4 186 3 Adelie 40.3 18 195 4 Adelie NA NA NA 5 Adelie 36.7 19.3 193 6 Adelie 39.3 20.6 190 7 Adelie 38.9 17.8 181 8 Adelie 39.2 19.6 195 9 Adelie 34.1 18.1 193 10 Adelie 42 20.2 190 # ℹ 334 more rows
Note that no matter if we specifically named the columns or used their index numbers, we get the same result. However using column numbers works, but can break if the dataset changes.
Now that we can select what columns we want to see, it is time to filter what rows we want. Below are some of the operators we are going to use to filter our data.
-
==: Equal to -
!=: Not equal to -
>=: Greater than or equal to -
<=: Less than or equal to -
>: Greater than -
<: Less than -
&: And -
|: Or -
%in%: Within
Subsection 2.5.3 Filter
When we want to select for specific columns in a dataset, we use the
select() command. But, when we want to filter the data inside a dataset, we utilize the filter() command.
filter().
Changes which rows exist.
Let’s say that we want to filter for only the penguin species Adelie. If you do not know what they look like, here is a picture:
We will filter for only rows that have the word Adelie inside the species column.
penguins |>
filter(species == "Adelie")
# A tibble: 152 × 8 species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g <fct> <fct> <dbl> <dbl> <int> <int> 1 Adelie Torgersen 39.1 18.7 181 3750 2 Adelie Torgersen 39.5 17.4 186 3800 3 Adelie Torgersen 40.3 18 195 3250 4 Adelie Torgersen NA NA NA NA 5 Adelie Torgersen 36.7 19.3 193 3450 6 Adelie Torgersen 39.3 20.6 190 3650 7 Adelie Torgersen 38.9 17.8 181 3625 8 Adelie Torgersen 39.2 19.6 195 4675 9 Adelie Torgersen 34.1 18.1 193 3475 10 Adelie Torgersen 42 20.2 190 4250 # ℹ 142 more rows # ℹ 2 more variables: sex <fct>, year <int>
With that piece of code, the output only returned rows where Adelie is the species.
Thankfully, we do not have to keep to only one filter condition at a time.
Subsubsection 2.5.3.1 Filtering with AND/OR
Inside the same
filter() command, we can have as many conditions as possible. One incredibly important decision is whether to use and/or.
In the case we only want to return rows where species is “Adelie” and the island is “Torgersen”, we will use the
& symbol.
Important: rows that return have to meet both criteria. If in that row either none, or even one, of the criteria are met, then that row will not return.
penguins |>
filter(species == "Adelie" & island == "Torgersen")
# A tibble: 52 × 8 species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g <fct> <fct> <dbl> <dbl> <int> <int> 1 Adelie Torgersen 39.1 18.7 181 3750 2 Adelie Torgersen 39.5 17.4 186 3800 3 Adelie Torgersen 40.3 18 195 3250 4 Adelie Torgersen NA NA NA NA 5 Adelie Torgersen 36.7 19.3 193 3450 6 Adelie Torgersen 39.3 20.6 190 3650 7 Adelie Torgersen 38.9 17.8 181 3625 8 Adelie Torgersen 39.2 19.6 195 4675 9 Adelie Torgersen 34.1 18.1 193 3475 10 Adelie Torgersen 42 20.2 190 4250 # ℹ 42 more rows # ℹ 2 more variables: sex <fct>, year <int>
That returned exactly what we wanted. But, what if we changed our mind. We want to return rows that either have “Adelie” as their species or the island is “Torgersen”. Instead of using the
& symbol, we need to use the | symbol.
penguins |>
filter(species == "Adelie" | island == "Torgersen")
# A tibble: 152 × 8 species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g <fct> <fct> <dbl> <dbl> <int> <int> 1 Adelie Torgersen 39.1 18.7 181 3750 2 Adelie Torgersen 39.5 17.4 186 3800 3 Adelie Torgersen 40.3 18 195 3250 4 Adelie Torgersen NA NA NA NA 5 Adelie Torgersen 36.7 19.3 193 3450 6 Adelie Torgersen 39.3 20.6 190 3650 7 Adelie Torgersen 38.9 17.8 181 3625 8 Adelie Torgersen 39.2 19.6 195 4675 9 Adelie Torgersen 34.1 18.1 193 3475 10 Adelie Torgersen 42 20.2 190 4250 # ℹ 142 more rows # ℹ 2 more variables: sex <fct>, year <int>
Now, instead of returning rows that have both “Adelie” and “Torgersen”, the code returns rows where either species is equal to “Adelie” or island is equal to “Torgersen”.
Subsubsection 2.5.3.2 Logical Filters
We do not have to just use
==, as we have other options. Below are two options: the first filters for any body mass over 4,000 grams, and the second filters for species that are either Adelie or Gentoo.
penguins |> filter(body_mass_g >= 4000)
# A tibble: 177 × 8 species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g <fct> <fct> <dbl> <dbl> <int> <int> 1 Adelie Torgersen 39.2 19.6 195 4675 2 Adelie Torgersen 42 20.2 190 4250 3 Adelie Torgersen 34.6 21.1 198 4400 4 Adelie Torgersen 42.5 20.7 197 4500 5 Adelie Torgersen 46 21.5 194 4200 6 Adelie Dream 39.2 21.1 196 4150 7 Adelie Dream 39.8 19.1 184 4650 8 Adelie Dream 44.1 19.7 196 4400 9 Adelie Dream 39.6 18.8 190 4600 10 Adelie Dream 42.3 21.2 191 4150 # ℹ 167 more rows # ℹ 2 more variables: sex <fct>, year <int>
penguins |> filter(species %in% c("Adelie", "Gentoo"))
# A tibble: 276 × 8 species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g <fct> <fct> <dbl> <dbl> <int> <int> 1 Adelie Torgersen 39.1 18.7 181 3750 2 Adelie Torgersen 39.5 17.4 186 3800 3 Adelie Torgersen 40.3 18 195 3250 4 Adelie Torgersen NA NA NA NA 5 Adelie Torgersen 36.7 19.3 193 3450 6 Adelie Torgersen 39.3 20.6 190 3650 7 Adelie Torgersen 38.9 17.8 181 3625 8 Adelie Torgersen 39.2 19.6 195 4675 9 Adelie Torgersen 34.1 18.1 193 3475 10 Adelie Torgersen 42 20.2 190 4250 # ℹ 266 more rows # ℹ 2 more variables: sex <fct>, year <int>
Subsubsection 2.5.3.3 Filtering NA values
In any dataset, NA values can be a pain. Depending what to do with them is always situational, but it is important to learn how to work with them. We can utilize
is.na() to filter for which are NA and the drop_na() to remove all rows that have at least one NA value.
penguins |> filter(is.na(body_mass_g)) # Filter for rows that have missing data
# A tibble: 2 × 8 species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g <fct> <fct> <dbl> <dbl> <int> <int> 1 Adelie Torgersen NA NA NA NA 2 Gentoo Biscoe NA NA NA NA # ℹ 2 more variables: sex <fct>, year <int>
penguins |> drop_na(body_mass_g) # Filters for rows that don't have missing data
# A tibble: 342 × 8 species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g <fct> <fct> <dbl> <dbl> <int> <int> 1 Adelie Torgersen 39.1 18.7 181 3750 2 Adelie Torgersen 39.5 17.4 186 3800 3 Adelie Torgersen 40.3 18 195 3250 4 Adelie Torgersen 36.7 19.3 193 3450 5 Adelie Torgersen 39.3 20.6 190 3650 6 Adelie Torgersen 38.9 17.8 181 3625 7 Adelie Torgersen 39.2 19.6 195 4675 8 Adelie Torgersen 34.1 18.1 193 3475 9 Adelie Torgersen 42 20.2 190 4250 10 Adelie Torgersen 37.8 17.1 186 3300 # ℹ 332 more rows # ℹ 2 more variables: sex <fct>, year <int>
As a cautionary tale, always be careful with how you handle NA values, as different situations really do require different handling of NA values.
Now, we will move on to how we want our data to be organized.
Subsection 2.5.4 Arrange
Organization of data is just as important as anything else. Most likely, data is not prearranged in any particular order. Luckily, the
arrange() command allows us to organize our data very nicely. There are only two options: ascending or descending order. By default, arrange() orders the data in ascending order.
arrange().
orders the rows of a data frame.
penguins |> arrange(body_mass_g) # ascending order
# A tibble: 344 × 8 species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g <fct> <fct> <dbl> <dbl> <int> <int> 1 Chinstrap Dream 46.9 16.6 192 2700 2 Adelie Biscoe 36.5 16.6 181 2850 3 Adelie Biscoe 36.4 17.1 184 2850 4 Adelie Biscoe 34.5 18.1 187 2900 5 Adelie Dream 33.1 16.1 178 2900 6 Adelie Torgers… 38.6 17 188 2900 7 Chinstrap Dream 43.2 16.6 187 2900 8 Adelie Biscoe 37.9 18.6 193 2925 9 Adelie Dream 37.5 18.9 179 2975 10 Adelie Dream 37 16.9 185 3000 # ℹ 334 more rows # ℹ 2 more variables: sex <fct>, year <int>
penguins |> arrange(desc(body_mass_g)) # descending order
# A tibble: 344 × 8 species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g <fct> <fct> <dbl> <dbl> <int> <int> 1 Gentoo Biscoe 49.2 15.2 221 6300 2 Gentoo Biscoe 59.6 17 230 6050 3 Gentoo Biscoe 51.1 16.3 220 6000 4 Gentoo Biscoe 48.8 16.2 222 6000 5 Gentoo Biscoe 45.2 16.4 223 5950 6 Gentoo Biscoe 49.8 15.9 229 5950 7 Gentoo Biscoe 48.4 14.6 213 5850 8 Gentoo Biscoe 49.3 15.7 217 5850 9 Gentoo Biscoe 55.1 16 230 5850 10 Gentoo Biscoe 49.5 16.2 229 5800 # ℹ 334 more rows # ℹ 2 more variables: sex <fct>, year <int>
Subsection 2.5.5 Mutate
Last chapter, we learned that in order to create a new column, we could do so using the
$ symbol. In tidyverse, we can also create new columns, but instead of using the $, we can use the mutate() command.
mutate().
Changes or creates columns.
# Base R
penguins$bill_ratio <- penguins$bill_length_mm / penguins$bill_depth_mm
penguins <- penguins |>
mutate(bill_ratio = bill_length_mm / bill_depth_mm)
Both of these examples are the same in the sense that a new column, bill_ratio, is created.
If we want to get a little more complicated and add some criteria, we can use either the
if_else() or case_when() command.
Subsection 2.5.6 If Else
In the scenario where we wanted to create a new column called size_category. If the body mass of the penguin is greater than 3,500 grams, then we want it to be considered “Big”; otherwise, it should be considered “Small”. To do this, we can use the
if_else() where we first add the criteria we want to build a column using, the value if it fits the criteria, and the value if it does not fit the criteria.
penguins |>
mutate(size_category = if_else(body_mass_g >= 3500, "Big","Small"))
# A tibble: 344 × 10 species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g <fct> <fct> <dbl> <dbl> <int> <int> 1 Adelie Torgersen 39.1 18.7 181 3750 2 Adelie Torgersen 39.5 17.4 186 3800 3 Adelie Torgersen 40.3 18 195 3250 4 Adelie Torgersen NA NA NA NA 5 Adelie Torgersen 36.7 19.3 193 3450 6 Adelie Torgersen 39.3 20.6 190 3650 7 Adelie Torgersen 38.9 17.8 181 3625 8 Adelie Torgersen 39.2 19.6 195 4675 9 Adelie Torgersen 34.1 18.1 193 3475 10 Adelie Torgersen 42 20.2 190 4250 # ℹ 334 more rows # ℹ 4 more variables: sex <fct>, year <int>, bill_ratio <dbl>, # size_category <chr>
That code creates the size_category, with the code saying:
-
if body_mass_g >= 3500 then make the value in size_category “Big”
-
if body_mass_g is not >= 3500 then make the value in size_category “Small”
But what if there are multiple criteria we want and not binary like big and small?
Subsubsection 2.5.6.1 Case When
When there are only two different criteria, then we can use
if_else(). But, when there are three or more criteria, we can use the case_when() command. Instead of having just “Big” and “Small”, let’s add the category “Gigantic”.
penguins |>
mutate(size_category = case_when(
body_mass_g <= 3500 ~ "Small",
body_mass_g > 3500 & body_mass_g <= 4000 ~ "Big",
body_mass_g > 4000 ~ "Gigantic",
TRUE ~ "Unknown" # catch-all for NAs or anything else
))
# A tibble: 344 × 10 species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g <fct> <fct> <dbl> <dbl> <int> <int> 1 Adelie Torgersen 39.1 18.7 181 3750 2 Adelie Torgersen 39.5 17.4 186 3800 3 Adelie Torgersen 40.3 18 195 3250 4 Adelie Torgersen NA NA NA NA 5 Adelie Torgersen 36.7 19.3 193 3450 6 Adelie Torgersen 39.3 20.6 190 3650 7 Adelie Torgersen 38.9 17.8 181 3625 8 Adelie Torgersen 39.2 19.6 195 4675 9 Adelie Torgersen 34.1 18.1 193 3475 10 Adelie Torgersen 42 20.2 190 4250 # ℹ 334 more rows # ℹ 4 more variables: sex <fct>, year <int>, bill_ratio <dbl>, # size_category <chr>
Now, still using the body_mass_g column, we still create the size_category column, just with three different values inside instead of two.
Subsection 2.5.7 Renaming Columns
Sometimes, for whatever reason, the column names that are originally in your data are not the ones that you want. That could be for aesthetic purposes or more practical reasons. Either way, to change the name of a column, you can use the
rename() command.
The formula to use when renaming columns using the rename() command is:.
\(New Name = OldName\)
As an example, let’s change the name of the species column to penguin_types.
penguins |> rename(penguin_types = species)
# A tibble: 344 × 9 penguin_types island bill_length_mm bill_depth_mm flipper_length_mm <fct> <fct> <dbl> <dbl> <int> 1 Adelie Torgersen 39.1 18.7 181 2 Adelie Torgersen 39.5 17.4 186 3 Adelie Torgersen 40.3 18 195 4 Adelie Torgersen NA NA NA 5 Adelie Torgersen 36.7 19.3 193 6 Adelie Torgersen 39.3 20.6 190 7 Adelie Torgersen 38.9 17.8 181 8 Adelie Torgersen 39.2 19.6 195 9 Adelie Torgersen 34.1 18.1 193 10 Adelie Torgersen 42 20.2 190 # ℹ 334 more rows # ℹ 4 more variables: body_mass_g <int>, sex <fct>, year <int>, # bill_ratio <dbl>
With all we can do, it is time to put
tidyverse to the test and perform our functions together!
Subsection 2.5.8 Putting them all together
What is fantastic about R is that we can put all these different functions together. We don’t just have to filter, or just have to select. As long as it makes sense, we can do as many functions together as we would like.
Let’s do exactly that and create a new data frame that selects, filters, mutates, and arranges data.
penguins_new <- penguins |>
select(species, island, bill_length_mm, body_mass_g) |>
filter(species == "Adelie") |>
mutate(size_category = if_else(body_mass_g >= 3500, "Big","Small")) |>
arrange(desc(body_mass_g)) |>
rename(penguin_types = species)
Here is the step by step breakdown of what exactly the code above does:
-
penguins_newis the name of the new dataset. -
It is created using the
<-command -
We are telling R that the source of our data is
penguins -
We are selecting for only the species, island, bill_length_mm, and body_mass_g columns
-
We are then filtering for only species that are “Adelie”
-
We are then creating a new column called size_category using the body_mass_g column.
-
We are then arranging the data in descending order of body_mass_g.
All of this code looks familiar, as it is all taken from previous example! Mind you, we could have taken any of the code from above and mixed and matched. In R, you are able to build whatever you can imagine.
Note: Order is incredibly important here. Imagine a case where you did not select for species and then went to rename the column to penguin_types. That code above would not work because in that code, species wouldn’t have been selected for.
Important:.
In tidyverse pipelines, the order of functions matters. Each step depends on the output of the previous step.
Now that we’ve put it all together, let’s check out some more things we can do in
tidyverse.
