Section 5.1 Reliability vs Validity
In criminal justice, we often aim to study variables or concepts that are symbolic or abstract in nature. For instance, it’s not possible to precisely measure the levels of disadvantage in society as we would measure weight using a scale. Instead, we rely on multiple indicators, such as poverty, low education, or employment status, to capture the degree of disadvantage. Ensuring the accuracy and consistency of these measures is critical, as invalid and unreliable measures can undermine the quality of our analysis.
The reliability of a scale refers to its consistency and freedom from measurement error. Essentially, it reflects how stable or consistent a measure, test, or observation is internally and across repeated uses. When multiple measurements of the same subject under the same conditions produce highly similar results, we consider the instrument reliable. On the other hand, the validity of a scale is about whether an instrument or test accurately measures what it’s supposed to, or how free it is from systematic error. Validity requires reliability, meaning that a measure must be consistent to be valid. However, it’s important to note that reliability alone does not guarantee validity (Vogt & Johnson, 2011).
Let me provide an example to illustrate reliability. Imagine you step on your weight scale, and it shows 160 pounds. You step off and on again, and it reads 150 pounds this time. Repeating the process again, it shows 140 pounds. In this case, your scale wouldn’t be considered reliable because it gives inconsistent readings.
However, your scale would be reliable if you stepped on the scale multiple times and consistently got the same reading, say 170 pounds each time. Even if the reading isn’t accurate (maybe you weigh 160 pounds), the scale is reliable if it consistently gives the same result.
Determining the validity of a scale is often more challenging than estimating its reliability because validity assessment requires more nuanced judgments about the accuracy and appropriateness of those measurements. Reliability can be assessed using statistical methods and quantifiable measures of consistency, but validity is a multifaceted concept involving various dimensions (e.g., content, criterion, and construct validity). In this chapter, we will only focus on the reliability of a scale.
Many social constructs in social science are abstract, and it is very challenging to measure consistently, unlike our physical weight, which is an objective physical property. Thankfully, there is a statistical way to examine the reliability of a scale for crime analysts. In this chapter, we will use the data from the National Crime Victimization Survey (NCVS) to compute the reliability of a scale.
Subsection 5.1.1 National Crime Victimization Survey
I noted that crimes that go unreported to the police are often called the “dark figure of crime” because they remain undiscovered and unreported, making them hidden from official records. Estimating unreported crimes is crucial for the criminal justice system to function properly.
To address the limitations of official crime measurement methods, the National Crime Victimization Survey (NCVS) was initiated. Conducted by the United States Census Bureau for the Bureau of Justice Statistics, the NCVS surveys over 200,000 individuals from 150,000 randomly selected households. The goal is to generate findings representative of the entire U.S. population. The survey, conducted every six months for three years, asks respondents about their experiences with crime, including whether they have been assaulted and whether they have reported the crime to the police. The survey aims to uncover the true extent of unreported and undiscovered crimes.
Despite its benefits, the NCVS has limitations. One major challenge is that respondents may not always be forthcoming about their victimization experiences. For example, individuals may feel uncomfortable discussing sexual offenses openly. Additionally, the accuracy of respondents’ memories may be affected since they are asked about victimization events every six months, leading to potential confusion about timelines. Furthermore, the survey design primarily focuses on street crimes, with limited coverage of other offenses. This narrow focus may result in the underrepresentation of certain types of crimes in the data, such as Part 2 offenses. Nonetheless, the NCVS provides valuable insights into unreported crime.
