Summary Statistics

Section 1.6 Summary Statistics

A statistic is a number derived from a dataset, usually intended to quantify some aspect of the data. Examples include the count, mean, variance, and standard deviation.

🔗

A Series object has a count method that returns the number of values that are not nan.

🔗

Listing 1.6.1. Python Code

🔗

$ weights = preg["totalwgt_lb"]
n = weights.count()
n
np.int64(9038)

It also provides a sum method that returns the sum of the values — we can use it to compute the mean like this.

🔗

Listing 1.6.2. Python Code

🔗

$ mean = weights.sum() / n
mean
np.float64(7.265628457623368)

But as we’ve already seen, there’s also a mean method that does the same thing.

🔗

Listing 1.6.3. Python Code

🔗

$ weights.mean()
np.float64(7.265628457623368)

In this dataset, the average birth weight is about 7.3 pounds.

🔗

Variance is a statistic that quantifies the spread of a set of values. It is the mean of the squared deviations, which are the distances of each point from the mean.

🔗

Listing 1.6.4. Python Code

🔗

squared_deviations = (weights - mean) ** 2

We can compute the mean of the squared deviations like this.

🔗

Listing 1.6.5. Python Code

🔗

$ var = squared_deviations.sum() / n
var
np.float64(1.983070989750022)

As you might expect, Series provides a var method that does almost the same thing.

🔗

Listing 1.6.6. Python Code

🔗

$ weights.var()
np.float64(1.9832904288326545)

The result is slightly different because when the var method computes the mean of the squared deviations, it divides by n-1 rather than n. That’s because there are two ways to compute the variance of a sample, depending on what you are trying to do. I’ll explain the difference in Chapter 8 — but in practice it usually doesn’t matter. If you prefer the version with n in the denominator, you can get it by passing ddof=0 as a keyword argument to the var method.

🔗

Listing 1.6.7. Python Code

🔗

$ weights.var(ddof=0)
np.float64(1.983070989750022)

In this dataset, the variance of the birth weights is about 1.98, but that value is hard to interpret — for one thing, it is in units of pounds squared. Variance is useful in some computations, but not a good way to describe a dataset. A better option is the standard deviation, which is the square root of variance. We can compute it like this.

🔗

Listing 1.6.8. Python Code

🔗

$ std = np.sqrt(var)
std
np.float64(1.40821553384062)

Or, we can use the std method.

🔗

Listing 1.6.9. Python Code

🔗

$ weights.std(ddof=0)
np.float64(1.40821553384062)

In this dataset, the standard deviation of birth weights is about 1.4 pounds. Informally, values that are one or two standard deviations from the mean are common — values farther from the mean are rare.

🔗

Prev Top Next