Research Methods STA630
PILOT TESTING OF THE QUESTIONNAIRE
Pilot testing also called pre-testing means small scale trial run of a particular component; here we are
referring to pilot testing of the questionnaire.
Conventional wisdom suggests that pre-testing not only is an established practice for discovering errors
but also is useful for extra training the research team. Ironically, professionals who have participated in
scores of studies are more likely to pretest an instrument than is a beginning researcher hurrying to
complete a project. Revising questions five or more times is not unusual. Yet inexperienced
researchers often underestimate the need to follow the design-test-revise process.
It is important to pilot test the instrument to ensure that the questions are understood by the respondents
and there are no problems with the wording or measurement. Pilot testing involves the use of a small
number of respondents to test the appropriateness of the questions and their comprehension. Usually, the
draft questionnaire is tried out on a group that is selected on a convenience and that is similar in makeup
to the one that ultimately will be sampled. Making a mistake with 25 or so subjects can avert the
disaster of administering an invalid questionnaire to several hundred individuals. Hence the main
purpose of pilot testing is to identify potential problems with the methods, logistics, and the
Administering a questionnaire exactly as planned in the actual study often is not possible. For example,
mailing out a questionnaire might require several weeks. Pre-testing a questionnaire in this manner
might provide important information on response rate, but it may not point out why questions were
skipped or why respondents found certain questions ambiguous or confusing. The ability of personal
interviewer to record requests for additional explanation and to register comments indicating
respondent's difficulty with question sequence or other factors is the primary reason why interviewers
are often used for pretest work.
What aspects to be evaluated during pilot testing?
1. Reactions of Respondents:
The reactions of the respondents can be looked at from different angles. The researcher may be familiar
with the local culture; still getting the first hand experience is always useful. Going to the field,
contacting the people, and their reactions to the different aspects of research may be a learning
Availability of study population timing. In case we are doing interviewing then pre-testing might
help to find out the most appropriate time when the respondent shall be available. The researcher
can plan the interviewing accordingly.
Acceptability of the questions asked. An important purpose of pre-testing is to discover
participants' reaction to the questions. If the participants do not find the experience stimulating
when an interviewer is physically present, how will they react on the phone, or in the self
administered mode? Pre-testing should help to discover where repetitiveness or redundancy is
bothersome or what topics were not covered that the participant expected. An alert interviewer will
look for questions or even sections that the participant perceives to be sensitive or threatening or
topics about which the participant knows nothing.
Pre-testing will also provide the opportunity to see the acceptability of the wording of the questions
in the local cultural context. Some of the issues may be discussed openly while for others people
use a disguised language. If people consider the use of certain phrases as offensive, then it is high
time to change the wording.
Willingness of the respondents to co-operate. Field testing of the questionnaire will give the idea
about the level of cooperation the research team is likely to get from the respondents, particularly if
they have to interview them.
Research Methods STA630
2. Discovering errors in the instrument:
Do the tools provide you the information? Reliability. Suitability for analysis. Tabulation
of the results /of a pretest helps determine whether the questionnaire will meet the objectives
of the research. A preliminary analysis often illustrates that although respondents can easily
comprehend and answer a given question, it is an inappropriate question because it does not
help solving the issue. The information may not be suitable for analysis.
Time taken/needed to interview/conduct the observation. Pre-testing can indicate the time
taken for interview or to conduct the observation. Too long questionnaires may not be
recommended and, therefore, need modification. It can also help in estimating average time
being taken to collect information form a respondent. Such an exercise can help in budget
If there is any need to revise the format of the tool. Question arrangement can play a
significant role in the success of the instrument. May be we should start with stimulating
questions and place sensitive questions last. Such a situation might be handled through pre-
testing. Therefore, pre-testing may help in putting questions in proper sequence, using
acceptable wording, doing appropriate translation, question spacing, structuring of
answers, coding system, and needing instructions for interviewers (probing).
3. Sampling procedure can be checked:
The extent to which instructions given are followed. Field functionaries are given the
instructions for following a sampling procedure. Depending upon the type of sampling to be
followed, the field worker must follow the guidelines otherwise the quality of the study will be
hampered. During the pre-testing one could see not only the extent to which the instructions
are being followed but also locate the problems in carrying out those instructions. Also what
could be the solutions to those problems?
How much time is needed to locate the respondents? By following the instructions how
easy it is to locate the respondents, and how much time is needed to do that activity. It could
help in calculating the overall time for data collection, having relevancy for budgeting thee
4. Staffing and activities of research team can be checked:
How successful the training has been? Pre-testing can be seen as a period of extra training.
The pre-testing exercise can provide a good opportunity to make an evaluation of the
achievement of the objectives of training. For any deficiencies additional training may be
What is the work output of each member? The researcher can calculate the average output
of each fieldworker and accordingly calculate the number of workers needed to finish the
work on time. It can also help in making the budget estimates.
How well the research team works together? It is a good opportunity to observe the kind of
coordination the research team has. The integrated work is likely affect the efficiency of the
team. Any shortcomings could be looked after.
Is the logistical support adequate? Of course we are leaving the field functionaries in
isolation. They shall be in need of other logistical support like the transportation, boarding,
lodging, guidance and supervision. Some of these aspects could also be appraised during the
5. Procedure for data processing and analysis can be evaluated:
Make dummy tables. See how can we tabulate the data and use the appropriate statistics for
purposes of interpretations
Table of Contents: