Developing valid and reliable questions
Contact Institutional Effectiveness and Assessment staff if you need additional advice or assistance in preparing a course feedback questionnaire or developing individual questions to be added to an existing form.
- Preserve anonymity. Do not include any questions that would allow you to identify the responses of individual students. Limit the number of demographic questions you ask so that you cannot identify students, even unintentionally.
- Ask about things students can reasonably be expected to remember, describe, and/or evaluate. Some aspects of course content or instructional quality are outside the range of students’ direct experience or knowledge. For example, students are rarely able to evaluate an instructor’s disciplinary or subject-matter expertise. They can provide information on how effectively the faculty member promotes student understanding of, or interest in, that area.
- Focus each question on a single topic or issue. Folding two or more questions into one will compromise the accuracy of the responses. Asking “Was the faculty member’s feedback on student work timely and helpful?” will pose problems if the students felt the feedback was timely but not particularly helpful, or helpful but not sufficiently timely. If a student responds “no,” the instructor won’t know if the problem was lack of timeliness, lack of helpfulness, or both.
- Keep questions simple and direct. Long questions, or questions that include negatives, are more subject to misinterpretation.
- Avoid questions that are likely to evoke simple “yes” or “no” answers. Instead of asking “Did the faculty member encourage the expression of different points of view?” phrase the question to elicit more judgment and/or description: “To what extent did the faculty member encourage…” or “How effectively did the faculty member encourage…” Adding “Please elaborate” or “In what ways?” at the end of the question is another way to elicit more information.
- Avoid leading questions. For example, asking “In what ways does the faculty member promote active learning?” appears to presume that the faculty member does promote active learning. Rewording the question more neutrally will elicit more valid responses: “To what extent does the faculty member promote active learning? Please elaborate.”
- Avoid words or phrases that might mean different things to different students. For example, asking “To what extent did the faculty member evaluate student work fairly?” is problematic because students might assign different meanings to “fairly.” Some may interpret the question to mean, “Did the faculty member spell out the criteria for evaluation in advance and adhere to them in assigning grades?” Others may interpret the question to mean, “Did the faculty member show favoritism in assigning grades?” (Notice that this latter interpretation also falls prey to the problem described in #2 above. Students can only evaluate favoritism if they know what grades other students received and why.)
- Avoid replicating previous questions. This is particularly important in preparing questions to be added to a standard form. The purpose of the supplementary questions is to elicit information that probably would not be elicited by the questions that precede them. Redundancy is confusing to respondents (“I thought I already answered that!”) and can also lower response rates by making questionnaires unnecessarily long.