Effect Size

Section 2.5 Effect Size

A difference like this is sometimes called an "effect". There are several ways to quantify the magnitude of an effect. The simplest is to report the difference in absolute terms — in this example, the difference is 0.078 weeks.

🔗

Another is to report the difference in relative terms. For example, we might say that first pregnancies are 0.2% longer than others, on average.

🔗

Listing 2.5.1. Python Code

🔗

$ diff / live["prglngth"].mean() * 100
np.float64(0.20237586646738304)

Another option is to report a standardized effect size, which is a statistic intended to quantify the size of an effect in a way that is comparable between different quantities and different groups.

🔗

Standardizing means we express the difference as a multiple of the standard deviation. So we might be tempted to write something like this.

🔗

Listing 2.5.2. Python Code

🔗

$ diff / live["prglngth"].std()
np.float64(0.028877623375210403)

But notice that we used both groups to compute the standard deviation. If the groups are substantially different, the standard deviation when we put them together is larger than in either group, which might make the effect size seem small.

🔗

An alternative is to use the standard deviation of just one group, but it’s not clear which. So we could take the average of the two standard deviations, but if the groups are different sizes, that would give too much weight to one group and not enough to the other.

🔗

A common solution is to use pooled standard deviation, which is the square root of pooled variance, which is the weighted sum of the variances in the groups. To compute it, we’ll start with the variances.

🔗

Listing 2.5.3. Python Code

🔗

group1, group2 = firsts["prglngth"], others["prglngth"]

Listing 2.5.4. Python Code

🔗

v1, v2 = group1.var(), group2.var()

Here is the weighted sum, with the group sizes as weights.

🔗

Listing 2.5.5. Python Code

🔗

n1, n2 = group1.count(), group2.count()
pooled_var = (n1 * v1 + n2 * v2) / (n1 + n2)

Finally, here is the pooled standard deviation.

🔗

Listing 2.5.6. Python Code

🔗

$ np.sqrt(pooled_var)
np.float64(2.7022108144953862)

The pooled standard deviation is between the standard deviations of the groups.

🔗

Listing 2.5.7. Python Code

🔗

$ firsts["prglngth"].std(), others["prglngth"].std()
(np.float64(2.7919014146687204), np.float64(2.6158523504392375))

A standardized effect size that uses pooled standard deviation is called Cohen’s effect size. Here’s a function that computes it.

🔗

Listing 2.5.8. Python Code

🔗

def cohen_effect_size(group1, group2):
    diff = group1.mean() - group2.mean()

    v1, v2 = group1.var(), group2.var()
    n1, n2 = group1.count(), group2.count()
    pooled_var = (n1 * v1 + n2 * v2) / (n1 + n2)

    return diff / np.sqrt(pooled_var)

And here’s the effect size for the difference in mean pregnancy lengths.

🔗

Listing 2.5.9. Python Code

🔗

$ cohen_effect_size(firsts["prglngth"], others["prglngth"])
np.float64(0.028879044654449834)

In this example, the difference is 0.029 standard deviations, which is small. To put that in perspective, the difference in height between men and women is about 1.7 standard deviations.

🔗

Prev Top Next