A distinctive feature of the Lawrence Fellows in the Liberal Arts and Sciences program is that program evaluation is built in from the beginning, creating a longitudinal study of the program's outcomes and benefits for Fellows, students, faculty, and the University. Thus, the Fellows program combines innovation with systematic assessment, both to document the effects of the program and to refine how the program is run based on results of formal study (rather than informal impressions). This effort takes place within the context of a new institutional commitment to becoming a self-reflective campus.

Whenever possible the program evaluation: (1) involves multiple measures to assess goals and outcomes, (2) uses repeated measurements over time (e.g., to establish progress made by Fellows), (3) reliesy on objective indicators as well as subjective self-reports, (4) when subjective assessments are used, employs the same measuring instrument, independently completed by different people, to control for individual biases (e.g., the same teaching evaluation form will be completed by students, mentors, independent observers, and Fellows - as a self-assessment - to yield a reliable and valid assessment of Fellows' teaching), (5) uses appropriate comparison groups (e.g., comparing student ratings of Fellows' teaching to student ratings of tenure-track faculty) to assess the program, (6) is conceived and administered by faculty (with administrative support) rather than solely by administrators.

Program evaluation addresses all of the overarching goals of the Fellows program (articulated below). The outcomes of this evaluation are used for general reports about the program (e.g., to University constituencies such as faculty and trustees, to funding agencies), to improve how the program is run, and to support the development of individual Fellows by providing feedback to them (e.g., on others' evaluations of their teaching). We intend to publish the results of the evaluation in journals or other outlets in the higher education community so that other institutions that are considering or have post-doctoral fellow programs may benefit from assessments of Lawrence's experience. All measures developed to assess the program will be made available (e.g., on the internet, as appendices to reports) for adoption by other institutions who are interested in using them. Because, to our knowledge, no other systematic study has been conducted on the outcomes of post-doctoral fellowships, Lawrence's study may be of special interest and significance to the higher education community.

Because many of the program's goals involve the impact of the program on specific constituencies -- students, Fellows, faculty, departments -- many outcomes are necessarily assessed by surveying individuals in these categories. As noted above, however, measuring independent perspectives on specific outcomes enhances the reliability, validity, and credibility of these assessments. Subjective assessments are also supplemented by objective or documented indicators. For example, the impact of the Fellows program on research opportunities for students is assessed by reports from Fellows, from students, from Department Chairs, but also through such documented outcomes as independent study and tutorial courses, joint publications, and conference presentations involving Fellows and students. Narrative reports are used to inform and deepen the interpretation of these more objective indicators. For example, a student's own assessment of how a research opportunity affected his or her skills and career plans enriches understanding of the quality of outcomes and supplement such evidence as joint papers or presentations.

Specific assessment forms for each of the relevant constituencies are appended to this document. Teaching Evaluation Form (for students, for Fellows), Fellows Annual Report, Mentor/Chair Annual Report, Survey of Majors, Graduate Advisor Survey, and Brief Faculty Survey.

The program evaluation also assesses long-term outcomes for the Fellows. Because a central aim of the program is to provide the Fellows with a transition between graduate school and an academic career that prepares them to be successful, it is important to assess Fellows' progress and accomplishments not only while they are at Lawrence, but long after they leave. These assessments will occur at intervals of increasing length (1 year, 3 years, 7 years). Success will be measured by such indicators as the quality of the academic jobs Fellows obtain, continued scholarly productivity, awards, and ability to gain tenure.

Desired Outcomes and Measures

The desired outcomes of the Fellows program are listed below, accompanied by summaries of the indicators that will assess how well these goals are achieved. Goals are divided by the various constituencies they are most strongly associated with (though, of course, many of the goals address multiple constituencies).

Outcomes for Fellows and Academia

Developing Fellows' Excellence as Teachers

The Fellows program includes not only the opportunity for Fellows to gain teaching experience in a liberal arts setting, but rather than simply throwing Fellows into the classroom to sink or swim, we aim to support and mentor them. The support takes the form of peer discussion groups, mentoring by established Lawrence faculty members, opportunities for the Fellows' classes to be observed by others , and co-teaching with Lawrence faculty members.

Assessment of Teaching Excellence:

Fellows' initial goal-setting at the beginning of the year are used as the basis for self-evaluation and evaluation by mentors regarding how well the Fellows have progressed toward each of their self-designed goals. Ratings and narrative reflections on each of the Fellow's goals are requested as part of the Fellows Annual Report, and Mentors/Chairs Annual Report.

As is done for all Lawrence faculty, Lawrence's form for student evaluation of teaching (recently revised based on research into how teaching is best assessed) is administered to students at the end of the term in each class taught by a Fellow. This form includes both Likert-style numerical ratings as well as narrative questions. Use of the standard evaluation form allow s for comparisons of Fellows' ratings over time (e.g., first and second year) as well as to other faculty (e.g., tenure-track faculty in their first few years, adjunct faculty, tenured faculty, etc.).

Additionally, the standard teaching evaluation form is used for ratings of Fellows made by their assigned faculty mentors and by department Chairs (as part of their annual reports). Questions from this form are also incorporated into the Classroom Observation Form, to be completed each time the mentor observes a Fellow's class Fellows will also complete the standard course evaluation at the end of each term.

Thus, although subjective assessments of teaching are a major source for documenting the quality of Fellows' teaching and their progress over their two year appointments, consistent with the program evaluation's general principles, these assessments will be made on the same forms by independent raters representing various constituencies and levels of expertise. Developing Fellows' Excellence in Scholarship/Performance

Developing Fellows' Excellence in Scholarship/Performance

The Fellows program seeks to provide a position that promotes the development of Fellows as independent scholars and performers by allowing them sufficient time to devote to professional activities and, in many cases, by linking them to established faculty scholars at Lawrence who share similar interests. In contrast, tenure-track hires at small colleges typically face a much higher teaching load that forces them to devote most of their energies to developing classes at a crucially important time in their scholarly development (after having left the resources, support, and specialized focus of graduate training). A lower teaching load, absence of formal academic advising duties, lessened pressure to teach lower-level classes (that intersect less with scholarly interests), and absence of committee assignments are structural features of the Fellows position designed to provide fertile soil for Fellows' development as scholars and performers. Further, the Fellows Program encourages overlap of scholarly interests with current Lawrence faculty (a luxury that cannot typically be afforded for tenure-track hires in the small departments of liberal arts colleges) affording more possibilities of collegial support and mentorship, as well as scholarly collaboration.

Assessment of Scholarly Excellence:

A primary means of assessing scholarly excellence is through documentation of what the Fellows have produced (e.g., published articles, books, performances) and the quality of those contributions.. In addition to the quantity of what the Fellows produce, its quality can also be judged, in part, by the prestige of the outlets for Fellows' work. Accepted ratings of a journal's or publisher's quality (e.g., impact and citation ratings of journals, prestige rankings of publishing houses) or of the standards of a recording label provide indices of quality. Additionally, the annual reports document any grants or awards or similar indicators of scholarly recognition and contribution.

Additionally, in their annual reports, Fellows and their mentors are each asked to rate and give narrative evaluations of Fellows' progress toward their self-defined scholarly goals, based on the goal-setting forms completed at the beginning of the academic year.

Fostering Fellows' Long-term Career Success and Contributions to Academia

The ultimate goal of the program with respect to Fellows and the wider academic world is to foster the success of promising young academics so that they become excellent contributors to their fields and the institutions that subsequently hire them. The Fellows position has been designed to promote success by helping Fellows to make a smoother, less frustrating transition into an academic career (especially at liberal arts colleges). Thus, an important measure of the overall success of the program is the long-term success of the Fellows as academics and a lower “attrition rate” (leaving the academy either due to frustration with the demands of academic jobs or inability to secure a tenured position).

Assessment of Fellows' Long-term Career Success and Contributions as Academics:

We follow the careers of Fellows after they leave Lawrence, asking them occasionally to update us on their current status, much in the way that Alumni Offices track alumni. At intervals of 1, 3, and 7 years, we will attempt to obtain current academic vitas, which would be easy for former Fellows to email as attachments, are usually kept up to date by academics, and contain a wealth of information on the status of the individual's career, contributions, and recognitions. We believe that while it is not realistic to expect former Fellows to complete long questionnaires after they have left Lawrence, they will be willing to respond to an occasional email requesting their current vitas. Vitas ought to document such important indicators as reappointments and promotions, publications and performances, quality of the institutions at which Fellows get jobs, whether Fellows have remained in academia, academic honors (e.g., prizes for scholarly contributions, teaching awards), and whether Fellows continue to teach courses they developed at Lawrence.

Outcomes for Lawrence Students

Enrichment of the Curriculum

The Fellows program provides an added pool of teaching faculty on the Lawrence campus whose courses will enrich the curriculum. Not only are the Fellows expected to have a direct effect by offering courses and topics that otherwise might not be taught, but their presence ought to have indirect effects by freeing up tenure-track faculty to offer courses that they may not otherwise have been able to teach (when a Fellow takes over an existing course). Fellows are also expected to sponsor individualized instruction opportunities (in addition to their prescribed course loads) in Independent Studies (ISs) and tutorial classes. The end result ought to be greater curricular diversity and, as another benefit, an ability to maintain small class sizes across the University (given the net increase in courses taught) than would otherwise be the case. (Because the Fellows program at Lawrence has begun at a time of sharply increased overall enrollment, its contribution may be to maintain lower class sizes when they otherwise would have risen; statistical analyses will therefore control for overall enrollments when examining the impact of the program.)

Assessment of Impact on the Curriculum:

The course schedule is all that is required to document the courses Fellows offer, which can be considered to be net additions to the curriculum.

Individualized instruction can be documented by reports from the Registrar on ISs and tutorials sponsored by Fellows. Fellows will also be asked in their annual reports to note any individualized instruction they offered that was not offered for credit and therefore not "on the books."

These data on course offerings can be enriched with narratives by selected samples of students, perhaps junior and senior majors in the relevant department, including students who completed courses or took ISs or tutorials offered by Fellows (Advanced Majors Survey). These students can speak to the impact of the Fellows' offerings (e.g., how a course that otherwise would not have been offered allowed them to develop crucial skills for graduate school).

Finally, to assess the overall benefit of Fellows' presence on class sizes at Lawrence, data on enrollment kept by the Registrar can be use to track trends in class sizes within departments and across the University. The additional courses offered by Fellows ought to reduce overall class sizes. Departments who have Fellows may service more students, but hopefully with fewer students per class than would otherwise be the case. Regression analyses will be used to control for the effect of the overall size of the student body to obtain a more accurate estimate of the effects of presence of Fellows on class size.

Increased Research and Performance Opportunities

Fellows are expected to increase the number of opportunities students have to engage in research or performance collaborations. Some of these collaborations may occur through the ISs and tutorials mentioned above.

Assessing Research and Performance Opportunities:

Advising for and Connections with Graduate Programs

Because the Fellows will all be recent PhDs, we expect them to serve as informal advisors to Lawrence students about graduate school (where students ought to apply to best serve their interests, how to structure a successful application, how to assess which graduate school is the best fit for the individual, etc.). Additionally, because Fellows will still have strong connections to and, probably, frequent contact with faculty from their graduate programs, it is likely that students will become connected to those programs through the Fellows (e.g., a student may be given personal recommendations by Fellows as someone their mentor might want to work with over the summer, alerted to internship possibilities at the graduate institution, introduced to the graduate mentor informally, etc.)

Assessing Advising for and Connections with Graduate Programs:

The survey of advanced majors within departments that have a Fellow specifically asks about whether the Fellows have served as informal advisors about graduate school. Students are also asked about any specific connections they made at research institutions that were due to a Fellow's interventions.

The student surveys are supplemented by questions on Chairs' and mentors' annual reports because they may also be aware of ways in which Fellows promoted connections between students and graduate schools.

Outcomes for Lawrence University

Academic Connections and Recognition

Fellows' continuing connections to their graduate programs ought to enhance and help to create additional connections between Lawrence and research universities (e.g., increase the number of Lawrence students who attend graduate school or find summer internships at research universities, increase research collaborations and joint grants involving Lawrence faculty and research schools). Those connections and any attention garnered by the Fellows program (e.g., articles in the higher education press, greater awareness of Lawrence among faculty at graduate schools) will enhance recognition of Lawrence.

Assessing Academic Connections and Recognition:

Department Chairs' and mentors' annual reports also provide information about connections to research universities or recognition of Lawrence. Chairs are asked about their own subjective assessments about whether Fellows have increased students' interest in and success at applying to graduate programs and to document any increased connections to graduate schools (e.g., faculty collaborations, talks given at Lawrence by a graduate program's faculty or by Lawrence faculty at a graduate program).

The survey of advanced majors also includes questions about connections they have personally made with graduate programs due to the Fellows. This will include research connections, internships obtained, personal contacts made with faculty at the graduate schools, degree of (and changes in) interest in graduate school generally or the specific program the Fellow attended.

Outcomes for Faculty

Faculty-Fellow Research and Teaching Collaborations

Because it is likely that there will be greater overlap of professional interests between Fellows and Lawrence faculty than commonly occurs among the faculty. This ought to encourage a greater likelihood of professional collaborations that not only add to the intellectual atmosphere of the campus, but can reinvigorate Lawrence faculty, keeping them more connected to their own fields and expanding their professional activities and contributions. Similarly, with overlap between Fellows' and faculty members' interests, co-teaching is a likely outcome (and is also encouraged to help develop Fellows' teaching skills by working with an experienced member of the faculty).

Assessing Faculty-Fellow Research and Teaching Collaborations:

Research and teaching collaborations between Fellows and faculty is documented through the annual reports of Fellows, department Chairs, and mentors, which specifically ask about these collaborations. Joint publications and performances by faculty and Fellows provide objective indicators of the amount of professional collaboration (which could be compared to the frequency of such collaborations among tenure-track faculty). Given that we encourage a greater degree of overlap of interests between Fellows and faculty than we do among the tenure-track faculty, there ought to be a significantly greater number of Fellow-faculty collaborative efforts than is normally the case among the faculty. The presence of Fellows as collaborators may also increase the production of scholarship among Lawrence faculty - which can be assessed by comparing current productivity to average productivity from previous years (from information on annual faculty reports).

Subjective assessments of the impact and quality of these teaching and research collaborations are also solicited from the Fellows, faculty members, and students involved in them (as well as from faculty more generally -- Brief Faculty Survey.Narrative comments concerning these collaborations from the individuals involved in them l provide insight into the effects of these collaborations.

Faculty Members' Opportunities to Teach Advanced Courses and to Provide Individualized Instruction

The presence of the Fellows ought to create more flexibility in terms of what Lawrence faculty are able to teach (by providing an extra pool of teaching faculty). This, in turn, may enable Lawrence faculty to teach advanced courses they otherwise would not be able to make room for in their schedules or to devote more time to individualized instruction.

Assessing Faculty Members' Opportunities to Teach Advanced Courses and to Provide Individualized Instruction:

Documentation of changes in what departmental faculty have been able to teach (including ISs and tutorials) due to the presence of Fellows will come from department Chairs' reports (because Chairs understand what courses faculty members would have had to teach if a Fellow had not been present). This question can also be asked in a brief faculty survey. As noted above, the beginning of the Fellows program coincides with a large increase in overall enrollment, mainly due to a much larger incoming Freshman class. Without the Fellows, it is likely that many advanced classes would not be taught as faculty are forced by enrollment pressures to switch to lower-level courses.

End of Year Evaluation and Retreat

At the end of each academic year, Fellows, mentors, and department Chairs will be asked, as part of their annual reports to assess the success of the Fellows program. The strengths and weaknesses of the program will be discussed at a Bjorklunden retreat weekend toward the end of the academic year, attended by Fellows, mentors, and Fellows Committee members. Possible improvements to the program that can be implemented during the next academic year will be discussed.

Summary of Evaluation Plan Goals and Measures

Benefits to Fellows and Wider Academic Community

Developing Excellent Teachers

Developing Excellent Scholars/Artists

Fostering Long-Term Career Success and Contributions

Benefits to Students

Enrichment of Curriculum

Increased Research and Performance Opportunities

Advising for and Connections to Graduate Programs

Benefits to Lawrence University

Academic Reputation and Recognition

Intellectual Excitement on campus

Benefits to Faculty

Research and Teaching Collaborations

Faculty Opportunity to Teach Advanced and Individualized Courses