Chapter 9 (PART) Collecting data

Objectives

So far, you have learnt to ask an RQ and design the study. In this chapter, you will learn to:

record the important steps in data collection.
describe study protocols.
ask questionnaire questions.

A diagram showing the six steps of research, arranged in a cycle: Ask, Design, Collect, Classify and summarise, Analyse, Report. Step 3 (Collect) is highlighted, captioned “Collecting data”.

Section 9.1 Introduction

If the RQ is well-constructed, terms are clearly defined, and the study is well-designed and explained, then the process for collecting the data should be easy to describe. Data collection is often time-consuming, tedious and expensive, so collecting the data correctly first time is important, hence an accurate description of the data collection process is essential.

Note 9.1.

Data collection is often tedious, time-consuming and expensive: you usually get one chance to collect data. In contrast, data (once collected) can be analysed as many times as necessary. Design the study properly the first time!

Section 9.2 Protocols

Before collecting the data, a plan should be established and documented that explains exactly how the data will be obtained, which will include operational definitions (Section 2.10). This plan is called a protocol, and allows results to be confirmed and compared.

Definition 9.2. Protocol.

A protocol is a procedure documenting the details of the design and implementation of studies, and for data collection.

A protocol usually has at least three components that describe:

how individuals are chosen from the population (i.e., external validity).
how data are collected from the individuals (i.e., internal validity).
the types of analyses and software (including version) used.

Data collection often encounters problems or challenges, which should be documented also.

Example 9.3. Protocol.

[230] made cookies using puréed green peas in place of margarine (to increase the nutritional value of cookies). They assessed the acceptance of these cookies to college students.

The protocol discussed how the individuals were chosen (p. 4):

...through advertisement across campus from students attending a university in the southeastern United States.

This voluntary sample comprised \(80.6\)% women, a higher percentage than in the general population, and in the college population. (Other extraneous variables were also recorded.)

Exclusion criteria were also applied, excluding people ’with an allergy or sensitivity to an ingredient used in the preparation of the cookies’ (p. 5). The researchers also described how the data were obtained, including these steps (p. 5):

tasters sat at individual tables, so they could not be influenced by other tasters.
cookies were presented individually, on individual plates.
the cookies were presented in a randomised order (fat substituted with green pea puree at \(25\)%, \(0\)%, \(50\)%, \(100\)% and \(75\)%).
between tastes, tasters ’cleansed their palates’ by drinking distilled water at \(25^{\circ}\text{C}\text{.}\)
the tasters recorded (p. 5):

characteristics of color, smell, moistness, flavor, aftertaste, and overall acceptability, for each sample of cookies...

Thus, internal validity was addressed using random allocation (to manage confounding), blinding individuals (to partially manage the Hawthorne effect), and washouts (to manage the carryover effect). Details are also given of how the cookies were made, and how objective measurements (such as moisture content) were determined. Subjects were not blinded to being in a study, but were blinded to which substitution percentage was in each cookie.

The type of analyses and software used were also given.

Example 9.4. Operational definitions.

In a study where the sex of a person is the explanatory variable, an operational definition for the sex of the person is needed. That is, a description is needed for how the researchers determine the sex of the person.

One option is to ask each individual their sex. Another option is to make an informed assessment, based on clothing and body shape, for instance. A third option is to have subjects indicate their sex in a multiple-choice question.

Different types of studies may require different means for determining the sex of a person. The study protocol should explain how the sex of the person will be established.

Unforeseen complications are not unusual, so often a pilot study (or a practice run) is conducted before the actual data collection, to:

determine the feasibility of the data collection protocol.
identify unforeseen challenges.
obtain data to determine appropriate sample sizes (Section 32.1).
identify ways to potentially save time and money.

The pilot study may suggest changes to the protocol.

Definition 9.5. Pilot study.

A pilot study is a small test run of the study used to check that the protocol is appropriate and practical, and to identify (and hence fix) possible problems with the research design or protocol.

Image.

The data can be collected once the protocol has been finalised. Protocols ensure studies are reproducible (Section 5.3), so others can confirm or compare results, and others can understand exactly what was done, and how. Protocols should indicate how design aspects (such as blinding the individuals, random allocation of treatments, etc.) will happen. The final protocol, without pedantic detail, should be reported. Diagrams can be useful to support explanations. All studies should have a well-established protocol for describing how the study was done.

Example 9.6.

Consider this partial protocol, which shows honesty in describing a protocol:

Fresh cow dung was obtained from free-ranging, grass fed, and antibiotic-free Milking Shorthorn cows (Bos taurus) in the Tilden Regional Park in Berkeley, CA. Resting cows were approached with caution and startled by loud shouting, whereupon the cows rapidly stood up, defecated, and moved away from the source of the annoyance. Dung was collected in ZipLoc bags (\(1\) gallon), snap-frozen and stored at \(-80\) C.
―[262], p. 10

Section 9.3 Collecting data using questionnaires

Subsection 9.3.1 Writing questions

Collecting data using questionnaires is common for both observational and experimental studies. Questionnaires are very difficult to do well: question wording is crucial, and surprisingly difficult [255]. Pilot testing questionnaires is crucial.

Definition 9.7. Questionnaire.

A questionnaire is a set of questions for respondents to answer.

Note 9.8.

A questionnaire is a set of questions to obtain information from individuals. A survey is an entire methodology, that includes gathering data using a questionnaire, finding a sample, and other components.

Questions in a questionnaire may be open-ended (respondents can write their own answers) or closed (respondents select from a small number of possible answers, as in multiple-choice questions). Open-ended and closed questions both have advantages and disadvantages. Answers to open-ended questions more easily lend themselves to qualitative analysis and closed questions more to quantitative research. This section briefly discusses writing questions.

Example 9.9. Open and closed questions.

German students were asked a series of questions about microplastics [104], including:

Name sources of microplastics in the household.
In which ecosystems are microplastics in Germany? Tick the answer (multiple ticks are possible). Options: (a) sea; (b) rivers; (c) lakes; (d) groundwater.
Assess the potential danger posed by microplastics. Options: (a) very dangerous; (b) dangerous; (c) hardly dangerous; (d) not dangerous.

The first question is open-ended: respondents provide their own answers. The second question is closed, where multiple options can be selected. The third question is closed, where only one option can be selected.

Writing good questionnaire questions is difficult. Good practice requires:

avoiding leading questions that may lead respondents to answer a certain way.
avoiding ambiguity by avoiding unfamiliar or technical terms.
avoiding asking the uninformed; avoid asking respondents about issues they don’t know about. Many people will give a response even if they do not understand (and such responses are worthless). For example, people may give directions to places that do not even exist [200].
avoiding complex and double-barrelled questions, which are hard to understand and the answers hard to interpret.
avoiding problems with ethics, such as questions about people breaking laws, or revealing confidential or private information. In special cases and with justification, ethics committees may allow such questions.
ensuring clarity in question wording.
(for closed questions) ensuring options are mutually exclusive (responses fit into only one category) and exhaustive (categories cover all options).

Example 9.10. Poor question wording.

Consider a questionnaire asking these questions:

Because bottles from bottled water create enormous amounts of non-biodegradable landfill and hence threaten native wildlife, do you support banning bottled water?
Do you drink more water now?
Are you more concerned about Coagulase-negative Staphylococcus or Neisseria pharyngis in bottled water?
Do you drink water in plastic and/or glass bottles?
Do you have a water tank installed illegally, without permission?
Do you avoid purchasing water in plastic bottles unless it is carbonated, unless the bottles are plastic but not necessarily if the lid is recyclable?

Question 1 is leading because the expected response is obvious. Better would be: ’Do you support or not support banning bottled water?’

Question 2 is ambiguous: it is unclear what ’more water now’ is being compared to.

Question 3 is unlikely to give sensible answers, as most people will be uninformed. Many people will still give an opinion, but the data will be effectively useless (though the researcher may not realise).

Question 4 is double-barrelled, and would be better asked as two separate questions (one asking about plastic bottles, and one about glass bottles).

Question 5 is unlikely to be given ethical approval or to obtain truthful answers, as respondents are unlikely to admit to breaking rules.

Question 6 is unclear, since knowing what a yes or no answer means is confusing.

Example 9.11. Question wording.

Question wording can be important. In the 2014 General Social Survey (https://gss.norc.org), when white Americans were asked for their opinion of the amount America spends on welfare, \(58\)% of respondents answered ’Too much’. However, when white Americans were asked for their opinion of the amount America spends on assistance to the poor, only \(16\)% of respondents answered ’Too much’ [296].

Example 9.12. Mutually exclusive options.

In a study to determine the time doctors spent with patients (from [198]), doctors were given the options:

\(0\)–\(5\text{.}\)
\(5\)–\(10\text{.}\)
more than \(10\text{.}\)

This is a poor question, because a respondent does not know which option to select for an answer of ’\(5\)’. The options are not mutually exclusive.

The following (humourous) video shows how questions can be manipulated by those not wanting to be ethical: Video: Yes Prime Minister on manipulative questioning.

Subsection 9.3.2 Challenges using questionnaires

Using questionnaires presents myriad challenges.

Non-response bias (Section 6.7): non-response bias is common with questionnaires, as they are often used with voluntary-response samples (Subsection 6.4.2). The people who do not respond to the survey may be different from those who do respond.
Response bias (Section 6.7): people do not always answer truthfully; for example, what people say may not correspond with what people do (Example 8.11). Sometimes this is unintentional (e.g., poor questions wording), due to embarrassment, or because questions are controversial. Sometimes, respondents repeatedly provide the same answer (without reading the question) to a series of multiple-choice questions (i.e., always select ’Always’).
Recall bias: people may not be able to accurately recall past events clearly, or recall when they happened.
Question order: the order of the questions can influence the responses.
Interpretation: phrases and words such as ’Sometimes’ and ’Somewhat disagree’ may have different meanings to different people.

Many of these can be managed with careful questionnaire design, but discussing the methods are beyond the scope of this book.

Example 9.13.

[181] studied the knowledge, attitude, and level of confidence of paramedics when managing patients with visual or hearing problems. They used a questionnaire, which was sent to (p. 5)

... \(372\) potential participants with the expectation that at least \(310\) questionnaires [would] be returned (\(83\)% assumed return rate). However, only \(97\) out of \(372\) participants completed the questionnaire, resulting in actual return rate of \(26\)%.

Response rates from questionnaires are often very low (and unrepresentative).

Subsection 9.3.3 Preparing software for questionnaire data

Care is needed when preparing software for data collected using a questionnaire. Sometimes, of course, the data are collected by computer (e.g., online) and are supplied to the researchers already formatted and in electronic format.

Data from open questions are usually text-based (such as words, sentences or paragraphs of text). These can generally be included in the data worksheet (though there may be a limit to the length of such data), but cannot be analysed using the quantitative methods described in this book.

Closed questions are easily included in a data worksheet. In closed questions where respondents can select one option only, one column is needed for the question that records which option was selected. In closed questions where respondents can select all options that apply, each option requires its own column that records each respondents’ answer for that option.

Example 9.14. Open and closed questions: software.

In Example 9.9, three questionnaire questions are given that were asked of German students about microplastics [104]. Some (artificial) data are shown entered in Figure 9.15.

The first question requires open-ended, text-based answers (Sources). For the second (closed) question, students could select multiple options, so each option needs one column in the data worksheet (WhereSeas to WhereGroundwater). The third (closed) question required students to select one option from a given list, so one column (Danger) is needed to record responses. As usual (Section 2.12), each row represents one unit of analysis (student).

Section 9.4 Chapter summary

Having a detailed procedure for collecting the data (the protocol) is important. Using a pilot study to trial the protocol can reveal unexpected changes necessary for a good protocol. Creating good questionnaires questions is difficult, but important.

Reading Questions 9.5 Quick review questions

1.

What is the biggest problem with this question: ’Do you have bromodosis?’

It is double-barrelled.
It is a leading question.
It uses language that may not be understood.
It is ambiguous.

Solution.

It uses language that may not be understood.

2.

What is the biggest problem with this question: ’Do you spend too much time connected to the internet?’

It is double-barrelled.
It is a leading question.
It uses language that may not be understood.
It is ambiguous.

Solution.

It is ambiguous.

3.

What is the biggest problem with this question: ’Do you eat fruits and vegetables?’

It is double-barrelled.
It is a leading question.
It uses language that may not be understood.
It is ambiguous.

Solution.

It is double-barrelled.

4.

True or false: A well-defined protocol allows the researchers to make the study externally valid.

Solution.

FALSE.

5.

True or false: This question is likely to be a leading question: ’Do you support a ban on drinks sold in unrecyclable plastic bottles?’

Solution.

TRUE.

Exercises 9.6 Exercises

Answers to odd-numbered exercises are given at the end of the book.

1.

What is the problem with this question?

What is your age? (Select one option)

Under \(18\text{.}\)

Over \(18\text{.}\)

2.

What is the problem with this question?

How many children do you have? (Select one option)

None.

\(1\) or \(2\text{.}\)

\(2\) or \(3\text{.}\)

More than \(4\text{.}\)

3.

Which of these questionnaire questions is better? Why?

Should concerned cat owners vaccinate their pets?
Should domestic cats be required to be vaccinated or not?
Do you agree that pet-owners should have their cats vaccinated?

4.

Which of these questionnaire questions is better? Why?

Do you own an environmentally-friendly electric vehicle?
Do you own an electric vehicle?
Do you own or do you not own an electric vehicle?

5.

[213] studied sunscreen use, and asked participants questions, including these:

how often do you sun bathe with the intention to tan during the summer in Sweden? (Possible answers: never, seldom, sometimes, often, always).
how long do you usually stay in the sun between \(11\)am and \(3\)pm, during a typical day-off in the summer (June–August)? (Possible answers: \(< 30\text{,}\) \(30\)–\(1\text{,}\) \(1\)–\(2\text{,}\) \(2\)–\(3\text{,}\) \(>3\)).

Critique these questions. What biases may be present?

6.

[288] studied primary-school children’s knowledge of their natural environment. They were asked three questions:

Do you usually visit Guadaira Park?
- No, I don’t like parks.
- No, I don’t usually visit it.
- Yes, once per week.
- Yes, more than once a week.
🔗
How many times have you visited nature (the beach, countryside, mountains, etc.) in the last month?
- Never.
- Once.
- Two to three times.
- More than three times.
🔗
Which is your favourite natural place?
- Write a story.
- Draw a picture.
🔗

Which questions are open and which are closed? Which questions will produce qualitative data? Critique the questions.

Prev Top Next