Brad Hartlaub Nuts and Bolts of Classroom Assessment
1. What is your process of creating an exam? That is, how do you decide on the topics to cover, types of questions, number of questions, use of technology, use of external aids, re-use of questions, etc.?
I am fortunate to be teaching in a computer equipped classroom so the students use statistical software (Minitab, StatCrunch, applets, etc.) on a regular basis. Since they get very comfortable with the software, I have decided not to take the software away from during exams. This enables me to ask a variety of questions with real data, but I try very hard not to construct exams where students who are fast with the mouse and the keyboard have an advantage.
The necessary data sets are posted to our network immediately before the exam begins. Exams usually contain a mix of short answer (matching, multiple choice, true or false, and/or fill in the blank) and open ended (free response questions). My mix of computational and conceptual questions almost always places more emphasis on the conceptual parts. Essay questions have also become a regular component of my exams.
The topics included on the exam are based on reading assignments, homework exercises, and classroom activities. However, I never ask questions that the students have completed for homework. I know that some instructors prefer to include some assigned exercises on the exam, typically after changing a few numbers, but I have not used that strategy.
Instead, I will compile exercises from other textbooks or test banks. My search usually begins by looking for data sets that will be of interest to the current group of students. This list of questions is usually much longer than the students can complete in 50 minutes, so I begin to combine and eliminate problems. Occasionally, I have field tested questions for ETS.
Deciding on the appropriate number of questions and parts is usually the most difficult part of the process for me. Early in my career I had two mathematicians in my department who were interested in learning more about statistics and helping to alleviate enrollment pressures by teaching additional sections of our introductory statistics course. These individuals attended my class, during different semesters, on a daily basis and provided feedback. One of the most memorable aspects of these experiences was the exam preparation. I would prepare a draft of the exam and my colleagues would take the exam in their office a day or two before the students. They provided detailed notes and suggestions, including the amount of time they spent on each problem. The feedback was clearI made the exams too long. I think my experience over the last fifteen years has helped me do a better job with estimating the appropriate number of questions for an exam, but my students may have a different opinion!
Student feedback on my course evaluations suggests that they really like the policy that allows them to create and use a note sheet during the exam. The size of the sheet must be 8.5x11 and the students are only allowed to include statistical terms, formulas, and examples.
I usually do not re-use old exam questions, especially since I post some of them on the course web page. However, I have been known to surprise them. One time I even pulled the old trick that one of my professors used on me. On the day the exam was scheduled, my philosophy professor walked into class and started talking about new material. One of my classmates in the front row stopped him after about five minutes and asked him when he was going to give us the exam. He acted a bit surprised and then proceeded to tell us that he scheduled the exam so that we would study the material. Since he was confident that we had prepared for the exam he decided that our time would be better spent learning new material. (About half of the class was furious and the other half was elated!) I have not had the courage to try exactly what my professor did, but I showed up at a final exam a few years ago and told the students that if they were happy with their overall percentage then they could leave. They had no clue this was coming, but every semester since this incident one of my students asks me if I am going to use the old exam trick again. Moral of the story: Be unpredictable and keep them guessing!
2. How do you grade exams? How much feedback do you give? How do you decide on partial credit? How much time to you spend grading? Any tips to reduce grading time?
I do not use rubrics like we do to grade the AP Statistics Exams! My strategy is basically to go through the stack of exams one question at a time. For each question, I sort out all of the perfect solutions and put them aside. Then I go through the remainder of the exams and sort them according to mistakes. Finally I classify the severity of the mistakes from minor arithmetic errors to serious conceptual errors (p-value of -1.34) and decide on point deductions. Partial credit is possible, but not for a probability less than zero!
Unfortunately, I spend a considerable amount of time grading. I provide detailed feedback or a suggestion for each incorrect solution. I also write notes on the exam encouraging students with low scores to come and get help during office hours.
Providing copies of my handwritten solutions to the class has reduced my grading time in the last few years, but I still spend too much time grading. You can easily reduce grading time by using certain types of assessment items, but you must be happy using these items as one of your primary forms of assessment.
3. Besides exams/quizzes, what types of assessment do you use in your courses? What is your method of creating and grading these assessments (e.g., learning goals, expectations)?
Projects Data sets have been obtained from friends and colleagues for explanatory data analysis and modeling projects. Other times, each student or group was asked to find a data set and conduct an appropriate analysis. Simulation activities have also been used for group and individual projects.
Papers Topics vary from semester to semester, but the two most common assignments are:
· Design a simple comparative experiment
· Summarize and critique an article dealing with statistical inference
Poster Sessions Students were asked to prepare a poster to present their work on a quantitative research question of interest to them.
Presentations Problem sessions, conducted by the students, were held occasionally throughout semester. These sessions were designed to improve understanding of important concepts and enhance mathematical maturity by requiring a clear, detailed presentation of the material to their peers. For each problem a student was responsible for solving an assigned problem and presenting the solution to the rest of the class. Answering all questions about the solution was a required part of the presentation.
I always grade these assessments myself, but in the last few years I have started to incorporate the opinions of others. For example, another statistician who provided data for projects has volunteered to read one-page executive summaries written by the students. The students submit electronic copies of their executive summaries to me, and then I forward them to him in one e-mail message. He provides feedback to me, and I forward his unedited comments directly to each group or student. The students really appreciate hearing honest and direct feedback from another statistician who worked on the project. I have been very happy with this revision to my standard grading system.
Another recent change to my grading is the use of peer review. This process has worked very well for paper assignments because I remove the name and then randomly assign the paper to another student in the class. I really do use a double-blind process just like you are familiar with in refereeing manuscripts for a journal.
Students have also provided comments and scores for presentations and poster sessions. The correlations between the student scores and my own independent assessments have been very high.
4. How much weight (toward the course grade) is each assessment piece worth in your class? Do you use any classroom assessment techniques that are not graded?
The categories used to determine the overall percentage for each student, and the weights assigned to each category, have varied considerably over the years. Two examples taken from my syllabi during different semesters are provided below. I have been using overall percentages to determine course grades since I arrived at Kenyon College fifteen years ago.
Fall 2004
· Homework and Labs (15%)
· Short Papers (10%)
· Exam 1 (25%)
· Exam 2 (25%)
· Final Exam (25%)
Spring 2003
· Homework and Projects (20%)
· Short Papers (20%)
· Quiz 1 (10%)
· Quiz 2 (10%)
· Quiz 3 (10%)
· Quiz 4 (10%)
· Final Project (20%)
I have regularly added a statement to my syllabus indicating that class participation will be used to help make borderline decisions. Occasionally, I will award extra points, typically to homework, for attending a department colloquium or the Summer Science Poster Session.
5. How do you support the students in preparation for assessment (e.g., review session, drafts, practice problems)?
I used to hold optional review sessions, but I was not happy with them. The students who really needed the review did not attend these sessions. The students who did attend the sessions asked good questions, but they really wanted to see more examples and get feedback on what they had done incorrectly on homework problems. I thought this could be done more effectively one-on-one or in small groups so I decided to extend my office hours before exams.
My colleagues always know when I am giving an exam because the traffic volume increases considerably. Our office suites were designed so that thee faculty offices are adjacent to a quiet work/study area with comfortable chairs, white boards, and computers. Interacting with students outside of class has always been enjoyable, but this new design has really improved the environment for professional and social interaction.
Approximately one week before an exam, I will post two, three, or sometimes four old exams to the course web page. I suggest that the students study first and then use the old exams as practice exams. For example, take them in a one hour period using the same resources they will have available during the exam. Some students take this advice, and others work these problems like homework problems.
I will post partial solutions to the web page and other times I will make complete solutions available on the wall outside of my office. As you might expect, the students always prefer to get the complete solutions.
Almost every semester a student will ask me, typically in the last week or two before the final exam, if they can complete an extra credit project. I have never been a fan of extra credit projects so my response has always been no. However, I have recently implemented a policy which allows students to submit another paper if they are not happy with their grade on the first submission. I continue to provide comments on drafts during office hours, but I do not collect and grade outlines or drafts before the final paper is due.