Pretesting the questionnaire.
As my project’s viability relies heavily on a questionnaire, I did research into the questionnaire as a tool. This is a new process to me, and I was initially struck by the importance of the design and testing of a questionnaire in the early stages of it’s development. I began to understand that as a researcher, you are either an ‘insider’ or an ‘outsider’ (Bayeck, R. 2022), while also operating as a designer of sorts, and that the questionnaire goes through a series of iterations, to see how it looks and sounds, see how people react to it, and then cut again, and try again. (Converse, Presser. 2011)
In order to enhance the effectiveness of the questionnaire, I carried out a series of iterative ‘pretests’, by approaching participants who are aware that they are participating in a pretest, so in effect testing the effectiveness of the questionnaire, and feeding back on areas that may not be as effective as I had hoped.
I decided to approach colleagues that I work with and a family member, as opposed to students, as I wanted to ensure that there would not be any pressure to conform to the perceived intentions behind the questions. I am aware that it is possible that students can perform to the perceived expectations when completing such a task.
Furthermore, non students, I understand that this approach could produce more useful responses due to their level of confidence, and their interest in the activity itself, thereby resulting in a willingness to give up some time to help with my research. “Narrowed down to this subset, investigators may find themselves relying on that familiar source of forced labour – colleagues, friend and family.” (Converse, Presser. 2011)
The pretests were communicated to the initial participants as a way to test:
- Effectiveness of the questions
- Clarity of the questions
- Balance of the question, in terms of allowing a broad range of responses.
- Neutrality of the questions, in order to allow the participants to respond freely and not feel that they are being directed to a desired response.
The resulting feedback from the pretest participants were helpful in allowing me to edit the questionnaire. Examples of feedback comments are:
I think it would work better if you had both prompt for ‘yes’ and ‘no’ in the same slide, like: If your answered yes, what do you think the benefits of the breakout space could be? If no, please explain why. This would streamlined the process and also would avoid students having to respond to a question that does not apply to their previous answer.
A few questions could be more specific to help participants understand what kind of response you’re looking for. For example, instead of asking “Would a breakout space be useful?”, you could ask “In what ways might a breakout space support your learning or wellbeing?” This gives participants a clearer frame of reference.
Some questions might unintentionally suggest a preferred answer. For example, “Would a breakout space improve your wellbeing?” could be rephrased to “What impact, if any, do you think a breakout space could have on your wellbeing?” This keeps the question neutral and open-ended.
I would like to thank Marysa Dowling, Kirsten Leigh Zullo and Giacomo Occhipinti for their constructive feedback in responding to the pretest questions.
References:
Converse, J, Presser, S. (2011) Survey Questions. Sage Research Methods.