Comprehensive Reappointment Reviews: Sampling

    Guidelines for completing a sampling request form

    In all comprehensive reappointment reviews, IE&A selects the sample of students who will be asked to complete a Student Review of Teaching questionnaire.  The selection process begins when IE&A sends a sampling request form to the department chair with SIS information about the candidate’s teaching already entered.  The chair then confers with the candidate to complete the remainder of the form, following the guidelines below, and returns it to IE&A.

    Here are some suggestions for preparing a sampling request that yields a representative sample of invited students and promotes a high response rate.  Departments are also welcome to contact IE&A for additional information and advice.

    1. Determine whether any courses should be excluded from sample selection and prepare a brief rationale for any exclusions.  Normally, the sampling time frame for a first comprehensive review will include all instructional experiences offered during the three academic terms prior to the semester of the review, and the time frame for a second comprehensive review stretches back five academic terms. (Summer sessions are generally excluded.)  These time frames can be adjusted for faculty whose comprehensive review schedule is affected by a decision to count previous years of service toward their probationary period, or for faculty who were not teaching during one or more of the semesters that would otherwise be selected.
      • IE&A will select from all SIS-listed courses taught (including studio lessons, lab sections and independent study/research courses) during the terms included in the sample unless specific courses are identified on the sampling request form as courses to be excluded.
      • Generally, exclusions should be avoided, but occasionally there may be good reason to exclude a specific course. For example, if a course offered two years ago was team-taught, it may be difficult for students to distinguish the contributions of one instructor from another. Courses taught under unusual circumstances, or with unusual content or pedagogy, might also be candidates for exclusion. The rationale for exclusion will be incorporated into IE&A’s description of the sample in the candidate’s dossier.
    1. If desired, identify any additional students the faculty member has instructed in settings not recorded in SIS during the terms included in the sample.  Departments may want the sample to include students who were instructed by the faculty member in settings that do not involve registration for a course, and whose work with the faculty member would therefore not be recorded in the faculty member’s profile information on SIS.  Examples include collaborative undergraduate research experiences, theater productions, music ensembles, athletic organizations, non-credited internships, or other settings.
      • If the department wants these students to be eligible for inclusion in the sample, please identify the type of setting in which instruction was provided, and list the names and student ID numbers of the instructed students.
    1. Indicate whether the sample should include alumni.  Normally, IE&A recommends that the sample for more comprehensive reviews exclude alumni and students studying off-campus, because current on-campus students are more likely to participate and to do so more quickly. However, if the courses from which the sample is to be selected included a high percentage of students who have graduated prior to the year of the review, it may be advisable to include alumni in the sample.
      • If alumni are to be included, departments should allow additional time to obtain contact information from the St. Olaf Online Alumni Directory and to administer the evaluation, since alumni returns are slower than returns for current students.
    1. Indicate the procedure to be used in selecting the final sample of students.  The goal of the selection procedure is to obtain a representative sample of the students who have been instructed by the faculty member during the academic terms and instructional settings included in the sample. The sampling request form asks departments to indicate the type of selection procedure to be used.
      • The simplest and most straightforward approach is a simple random sample drawn from all the students enrolled in the courses and other instructional groups included in the sample.
      • However, in some cases, a different sampling procedure may produce a more representative sample, so long as there is still an element of randomness included in the final selection of individual students.  For example:
        • If half the faculty member’s courses are upper-division courses, 50% of the sample could be drawn from upper-division courses and 50% from lower-division courses. Since upper-division courses typically have smaller enrollments, students from upper division courses would thus be “over-represented” in the sample as a percentage of the total number of students the faculty member taught, but the final sample would be more representative of the faculty member’s teaching assignments.
        • If the faculty member has offered instructional experiences not listed in SIS and wishes to have students from these experiences in the sample, it is important not only to identify them (as described in #2 above) but also to specify the selection procedure for them.  The students could be simply added to the pool of students in SIS-listed teaching assignments and sampled in the same way as the SIS-identified students.  Alternatively, a department may request that a fixed percentage of the final sample consist of non-SIS-listed students, or (if it is a small number but an important teaching responsibility) that all non-SIS-listed students be selected.
        • If a faculty member has regularly offered two courses a year through an interdisciplinary program outside his or her department (e.g., Environmental Studies or Asian Studies), 33% of the sample could be drawn from the interdisciplinary courses and 66% from the departmental courses.
        • If a faculty member’s teaching load includes courses in different areas of his/her field (e.g., historical, theoretical, applied), the courses could be categorized by area and proportionate random samples could be selected from each category.
        • If a faculty member has offered team-taught courses, the department may wish to under-sample students from those courses, because students often have difficulty distinguishing the contributions of individual members of the team

    Departments are welcome to consult with the Office of Institutional Effectiveness and Assessment at any point in the process of preparing a sampling request.