ZeePedia

PILOT TESTING OF THE QUESTIONNAIRE:Discovering errors in the instrument

<< TOOLS FOR DATA COLLECTION:Guidelines for Questionnaire Design
INTERVIEWING:The Role of the Interviewer, Terminating the Interview >>
img
Research Methods ­STA630
VU
Lesson 24
PILOT TESTING OF THE QUESTIONNAIRE
Pilot testing also called pre-testing means small scale trial run of a particular component; here we are
referring to pilot testing of the questionnaire.
Conventional wisdom suggests that pre-testing not only is an established practice for discovering errors
but also is useful for extra training the research team. Ironically, professionals who have participated in
scores of studies are more likely to pretest an instrument than is a beginning researcher hurrying to
complete a project.  Revising questions five or more times is not unusual.  Yet inexperienced
researchers often underestimate the need to follow the design-test-revise process.
It is important to pilot test the instrument to ensure that the questions are understood by the respondents
and there are no problems with the wording or measurement. Pilot testing involves the use of a small
number of respondents to test the appropriateness of the questions and their comprehension. Usually, the
draft questionnaire is tried out on a group that is selected on a convenience and that is similar in makeup
to the one that ultimately will be sampled. Making a mistake with 25 or so subjects can avert the
disaster of administering an invalid questionnaire to several hundred individuals. Hence the main
purpose of pilot testing is to identify potential problems with the methods, logistics, and the
questionnaire.
Administering a questionnaire exactly as planned in the actual study often is not possible. For example,
mailing out a questionnaire might require several weeks. Pre-testing a questionnaire in this manner
might provide important information on response rate, but it may not point out why questions were
skipped or why respondents found certain questions ambiguous or confusing. The ability of personal
interviewer to record requests for additional explanation and to register comments indicating
respondent's difficulty with question sequence or other factors is the primary reason why interviewers
are often used for pretest work.
What aspects to be evaluated during pilot testing?
1. Reactions of Respondents:
The reactions of the respondents can be looked at from different angles. The researcher may be familiar
with the local culture; still getting the first hand experience is always useful. Going to the field,
contacting the people, and their reactions to the different aspects of research may be a learning
experience.
·
Availability of study population timing. In case we are doing interviewing then pre-testing might
help to find out the most appropriate time when the respondent shall be available. The researcher
can plan the interviewing accordingly.
·
Acceptability of the questions asked.  An important purpose of pre-testing is to discover
participants' reaction to the questions. If the participants do not find the experience stimulating
when an interviewer is physically present, how will they react on the phone, or in the self
administered mode? Pre-testing should help to discover where repetitiveness or redundancy is
bothersome or what topics were not covered that the participant expected. An alert interviewer will
look for questions or even sections that the participant perceives to be sensitive or threatening or
topics about which the participant knows nothing.
·
Pre-testing will also provide the opportunity to see the acceptability of the wording of the questions
in the local cultural context. Some of the issues may be discussed openly while for others people
use a disguised language. If people consider the use of certain phrases as offensive, then it is high
time to change the wording.
·
Willingness of the respondents to co-operate. Field testing of the questionnaire will give the idea
about the level of cooperation the research team is likely to get from the respondents, particularly if
they have to interview them.
80
img
Research Methods ­STA630
VU
2. Discovering errors in the instrument:
·
Do the tools provide you the information? Reliability. Suitability for analysis. Tabulation
of the results /of a pretest helps determine whether the questionnaire will meet the objectives
of the research. A preliminary analysis often illustrates that although respondents can easily
comprehend and answer a given question, it is an inappropriate question because it does not
help solving the issue. The information may not be suitable for analysis.
·
Time taken/needed to interview/conduct the observation. Pre-testing can indicate the time
taken for interview or to conduct the observation. Too long questionnaires may not be
recommended and, therefore, need modification. It can also help in estimating average time
being taken to collect information form a respondent. Such an exercise can help in budget
estimations.
·
If there is any need to revise the format of the tool. Question arrangement can play a
significant role in the success of the instrument. May be we should start with stimulating
questions and place sensitive questions last. Such a situation might be handled through pre-
testing.  Therefore, pre-testing may help in putting questions in proper sequence, using
acceptable wording, doing appropriate translation, question spacing, structuring of
answers, coding system, and needing instructions for interviewers (probing).
3. Sampling procedure can be checked:
·
The extent to which instructions given are followed. Field functionaries are given the
instructions for following a sampling procedure. Depending upon the type of sampling to be
followed, the field worker must follow the guidelines otherwise the quality of the study will be
hampered. During the pre-testing one could see not only the extent to which the instructions
are being followed but also locate the problems in carrying out those instructions. Also what
could be the solutions to those problems?
·
How much time is needed to locate the respondents? By following the instructions how
easy it is to locate the respondents, and how much time is needed to do that activity. It could
help in calculating the overall time for data collection, having relevancy for budgeting thee
resources.
4. Staffing and activities of research team can be checked:
·
How successful the training has been? Pre-testing can be seen as a period of extra training.
The pre-testing exercise can provide a good opportunity to make an evaluation of the
achievement of the objectives of training. For any deficiencies additional training may be
provided.
·
What is the work output of each member? The researcher can calculate the average output
of each fieldworker and accordingly calculate the number of workers needed to finish the
work on time. It can also help in making the budget estimates.
·
How well the research team works together? It is a good opportunity to observe the kind of
coordination the research team has. The integrated work is likely affect the efficiency of the
team. Any shortcomings could be looked after.
·
Is the logistical support adequate? Of course we are leaving the field functionaries in
isolation. They shall be in need of other logistical support like the transportation, boarding,
lodging, guidance and supervision. Some of these aspects could also be appraised during the
pre-testing
5. Procedure for data processing and analysis can be evaluated:
·
Make dummy tables. See how can we tabulate the data and use the appropriate statistics for
purposes of interpretations
81
Table of Contents:
  1. INTRODUCTION, DEFINITION & VALUE OF RESEARCH
  2. SCIENTIFIC METHOD OF RESEARCH & ITS SPECIAL FEATURES
  3. CLASSIFICATION OF RESEARCH:Goals of Exploratory Research
  4. THEORY AND RESEARCH:Concepts, Propositions, Role of Theory
  5. CONCEPTS:Concepts are an Abstraction of Reality, Sources of Concepts
  6. VARIABLES AND TYPES OF VARIABLES:Moderating Variables
  7. HYPOTHESIS TESTING & CHARACTERISTICS:Correlational hypotheses
  8. REVIEW OF LITERATURE:Where to find the Research Literature
  9. CONDUCTING A SYSTEMATIC LITERATURE REVIEW:Write the Review
  10. THEORETICAL FRAMEWORK:Make an inventory of variables
  11. PROBLEM DEFINITION AND RESEARCH PROPOSAL:Problem Definition
  12. THE RESEARCH PROCESS:Broad Problem Area, Theoretical Framework
  13. ETHICAL ISSUES IN RESEARCH:Ethical Treatment of Participants
  14. ETHICAL ISSUES IN RESEARCH (Cont):Debriefing, Rights to Privacy
  15. MEASUREMENT OF CONCEPTS:Conceptualization
  16. MEASUREMENT OF CONCEPTS (CONTINUED):Operationalization
  17. MEASUREMENT OF CONCEPTS (CONTINUED):Scales and Indexes
  18. CRITERIA FOR GOOD MEASUREMENT:Convergent Validity
  19. RESEARCH DESIGN:Purpose of the Study, Steps in Conducting a Survey
  20. SURVEY RESEARCH:CHOOSING A COMMUNICATION MEDIA
  21. INTERCEPT INTERVIEWS IN MALLS AND OTHER HIGH-TRAFFIC AREAS
  22. SELF ADMINISTERED QUESTIONNAIRES (CONTINUED):Interesting Questions
  23. TOOLS FOR DATA COLLECTION:Guidelines for Questionnaire Design
  24. PILOT TESTING OF THE QUESTIONNAIRE:Discovering errors in the instrument
  25. INTERVIEWING:The Role of the Interviewer, Terminating the Interview
  26. SAMPLE AND SAMPLING TERMINOLOGY:Saves Cost, Labor, and Time
  27. PROBABILITY AND NON-PROBABILITY SAMPLING:Convenience Sampling
  28. TYPES OF PROBABILITY SAMPLING:Systematic Random Sample
  29. DATA ANALYSIS:Information, Editing, Editing for Consistency
  30. DATA TRANSFROMATION:Indexes and Scales, Scoring and Score Index
  31. DATA PRESENTATION:Bivariate Tables, Constructing Percentage Tables
  32. THE PARTS OF THE TABLE:Reading a percentage Table
  33. EXPERIMENTAL RESEARCH:The Language of Experiments
  34. EXPERIMENTAL RESEARCH (Cont.):True Experimental Designs
  35. EXPERIMENTAL RESEARCH (Cont.):Validity in Experiments
  36. NON-REACTIVE RESEARCH:Recording and Documentation
  37. USE OF SECONDARY DATA:Advantages, Disadvantages, Secondary Survey Data
  38. OBSERVATION STUDIES/FIELD RESEARCH:Logic of Field Research
  39. OBSERVATION STUDIES (Contd.):Ethical Dilemmas of Field research
  40. HISTORICAL COMPARATIVE RESEARCH:Similarities to Field Research
  41. HISTORICAL-COMPARATIVE RESEARCH (Contd.):Locating Evidence
  42. FOCUS GROUP DISCUSSION:The Purpose of FGD, Formal Focus Groups
  43. FOCUS GROUP DISCUSSION (Contd.):Uses of Focus Group Discussions
  44. REPORT WRITING:Conclusions and recommendations, Appended Parts
  45. REFERENCING:Book by a single author, Edited book, Doctoral Dissertation