A distinctive feature of the Lawrence Fellows in the Liberal Arts and Sciences program is that program evaluation is built in from the beginning, creating a longitudinal study of the program's outcomes and benefits for Fellows, students, faculty, and the University. Thus, the Fellows program will combine innovation with systematic assessment, both to document the effects of the program and to refine how the program is run based on results of formal study (rather than informal impressions). This effort takes place within the context of a new institutional commitment to becoming a self-reflective campus. The general approach to assessment at Lawrence will be faculty-driven, using the faculty's intimate and informed understanding of the goals of the institution and of particular programs (as well as the talents of those faculty with expertise as social science researchers). Assessment will not be imposed from without but designed from within, creating greater faculty buy-in. Further, as much as possible, assessment will take advantage of and be folded into existing structures and procedures so that it does not create an added burden for a faculty that is already spread thin.

The program evaluation will, whenever possible: (1) involve multiple measures to assess goals and outcomes, (2) use repeated measurements over time (e.g., to establish progress made by Fellows), (3) rely on objective indicators as well as subjective self-reports, (4) when subjective assessments are used, employ the same measuring instrument, independently completed by different people, to control for individual biases (e.g., the same teaching evaluation form will be completed by students, mentors, independent observers, and Fellows - as a self-assessment - to yield a reliable and valid assessment of Fellows' teaching), (5) use appropriate comparison groups (e.g., comparing student ratings of Fellows' teaching to student ratings of tenure-track faculty) to assess the program, (6) be conceived and administered by faculty (with administrative support) rather than solely by administrators.

Program evaluation will address all of the overarching goals of the Fellows program (articulated below). The outcomes of this evaluation will be used for general reports about the program (e.g., to University constituencies such as faculty and trustees, to funding agencies), to improve how the program is run, and to support the development of individual Fellows by providing feedback to them (e.g., on others' evaluations of their teaching). We intend to publish the results of the evaluation in journals or other outlets in the higher education community so that other institutions that are considering or have post-doctoral fellow programs may benefit from assessments of Lawrence's experience. All measures developed to assess the program will be made available (e.g., on the internet, as appendices to reports) for adoption by other institutions who are interested in using them. Because, to our knowledge, no other systematic study has been conducted on the outcomes of post-doctoral fellowships, Lawrence's study may be of special interest and significance to the higher education community.

Because many of the program's goals involve the impact of the program on specific constituencies -- students, Fellows, faculty, departments -- many outcomes will necessarily be assessed by surveying individuals in these categories. As noted above, however, measuring independent perspectives on specific outcomes will enhance the reliability, validity, and credibility of these assessments. Subjective assessments will also be supplemented by objective or documented indicators. For example, the impact of the Fellows program on research opportunities for students will be assessed by reports from Fellows, from students, from Department Chairs, but also through such documented outcomes as independent study and tutorial courses, joint publications, and conference presentations involving Fellows and students. Narrative reports will be used to inform and deepen the interpretation of these more objective indicators. For example, a student's own assessment of how a research opportunity affected his or her skills and career plans will enrich understanding of the quality of outcomes and supplement such evidence as joint papers or presentations.

Specific assessment forms for each of the relevant constituencies are appended to this document. These include the Videotape/Classroom Observation Form (for Fellows' self-observation, mentors, independent observers), Teaching Evaluation Form (for students, for Fellows), Fellows Annual Report, Mentor/Chair Annual Report, Survey of Majors, Graduate Advisor Survey, and Brief Faculty Survey.

The program evaluation will also assess long-term outcomes for the Fellows. Because a central aim of the program is to provide the Fellows with a transition between graduate school and an academic career that prepares them to be successful, it will be important to assess Fellows' progress and accomplishments not only while they are at Lawrence, but long after they leave. These assessments will occur at intervals of increasing length (1 year, 3 years, 7 years). Success will be measured by such indicators as the quality of the academic jobs Fellows obtain, continued scholarly productivity, awards, and ability to gain tenure.

Desired Outcomes and Measures

The desired outcomes of the Fellows program are listed below, accompanied by summaries of the indicators that will assess how well these goals are achieved. Goals are divided by the various constituencies they are most strongly associated with (though, of course, many of the goals address multiple constituencies).

Outcomes for Fellows and Academia

Developing Fellows' Excellence as Teachers

The Fellows program includes not only the opportunity for Fellows to gain teaching experience in a liberal arts setting, but rather than simply throwing Fellows into the classroom to sink or swim, we aim to support and mentor them. The support takes the form of peer discussion groups, mentoring by established Lawrence faculty members, opportunities for the Fellows' classes to be observed by others (as well as recorded for self-observation), and co-teaching with Lawrence faculty members.

Assessment of Teaching Excellence:

Fellows' initial goal-setting at the beginning of the year will be used as the basis for self-evaluation and evaluation by mentors regarding how well the Fellows have progressed toward each of their self-designed goals. Ratings and narrative reflections on each of the Fellow's goals will be requested as part of the Fellows Annual Report, and Mentors/Chairs Annual Report.

As is done for all Lawrence faculty, Lawrence's form for student evaluation of teaching (recently revised based on research into how teaching is best assessed) will be administered to students at the end of the term in each class taught by a Fellow. This form includes both Likert-style numerical ratings as well as narrative questions. Use of the standard evaluation form will allow for comparisons of Fellows' ratings over time (e.g., first and second year) as well as to other faculty (e.g., tenure-track faculty in their first few years, adjunct faculty, tenured faculty, etc.).

Additionally, the standard teaching evaluation form will be used for ratings of Fellows made by their assigned faculty mentors and by department Chairs (as part of their annual reports). Questions from this form are also incorporated into the Video/Classroom Observation Form, to be completed each time the mentor observes a Fellow's class and each time a Fellow self-reviews a videotaped class session (both expected once per term). Fellows will also complete the standard course evaluation at the end of each term.

Having samples of Fellows' teaching on videotape (if they choose to release these tapes) makes a more ambitious study of Fellows' performance and improvement possible, using expert observers (e.g., professors who are acknowledged to be excellent teachers) to independently evaluate the videotapes. Experimental controls, such as having recordings from different time periods rated in random order, would allow for rigorous assessment (in this example, of improvement over time without the potential influence on raters of knowing which recordings are earlier or later).

Thus, although subjective assessments of teaching will be a major source for documenting the quality of Fellows' teaching and their progress over their 2-3 year appointments, consistent with the program evaluation's general principles, these assessments will be made on the same forms by independent raters representing various constituencies and levels of expertise. Further, the evaluations (excluding self-assessments) will be compiled by the Provost's Office into an official teaching assessment that Fellows can choose to have sent to any colleges and universities to which they apply for tenure-track jobs.

Developing Fellows' Excellence in Scholarship/Performance

The Fellows program seeks to provide a position that promotes the development of Fellows as independent scholars and performers by allowing them sufficient time to devote to professional activities and, in many cases, by linking them to established faculty scholars at Lawrence who share similar interests. In contrast, tenure-track hires at small colleges typically face a much higher teaching load that forces them to devote most of their energies to developing classes at a crucially important time in their scholarly development (after having left the resources, support, and specialized focus of graduate training). A lower teaching load, absence of formal academic advising duties, lessened pressure to teach lower-level classes (that intersect less with scholarly interests), and absence of committee assignments are structural features of the Fellows position designed to provide fertile soil for Fellows' development as scholars and performers. Further, the Fellows Program encourages overlap of scholarly interests with current Lawrence faculty (a luxury that cannot typically be afforded for tenure-track hires in the small departments of liberal arts colleges) affording more possibilities of collegial support and mentorship, as well as scholarly collaboration.

Assessment of Scholarly Excellence:

A primary means of assessing scholarly excellence will be through documentation of what the Fellows have produced (e.g., published articles, books, performances) and the quality of those contributions. Fellows' annual report forms will ask for full documentation of these contributions, including copies of all articles, recordings of performances, etc. In addition to the quantity of what the Fellows produce, its quality can also be judged, in part, by the prestige of the outlets for Fellows' work. Accepted ratings of a journal's or publisher's quality (e.g., impact and citation ratings of journals, prestige rankings of publishing houses) or of the standards of a recording label provide indices of quality. Additionally, the annual reports will document any grants or awards or similar indicators of scholarly recognition and contribution.

Collecting copies of articles, books, recordings, etc., will allow the possibility of having outside reviewers assess quality. A possible comparison group for Fellows could be tenure-track faculty at Lawrence in their first few years to see if the Fellows position does, in fact, provide a boost to productivity by lessening the teaching and community demands they face in comparison to tenure-track faculty in similar fields. If so, this will also provide an index of the “cost” of those demands to tenure-track faculty early in their careers, which may suggest changes to how their positions are structured in those initial years. Understanding how to promote new faculty members' growth as scholars is of particular importance given the increased role of scholarship in tenure decisions at many liberal arts colleges.

Additionally, in their annual reports, Fellows and their mentors will each be asked to rate and give narrative evaluations of Fellows' progress toward their self-defined scholarly goals, based on the goal-setting forms completed at the beginning of the academic year.

Fostering Fellows' Long-term Career Success and Contributions to Academia

The ultimate goal of the program with respect to Fellows and the wider academic world is to foster the success of promising young academics so that they become excellent contributors to their fields and the institutions that subsequently hire them. The Fellows position has been designed to promote success by helping Fellows to make a smoother, less frustrating transition into an academic career (especially at liberal arts colleges). Thus, an important measure of the overall success of the program is the long-term success of the Fellows as academics and a lower “attrition rate” (leaving the academy either due to frustration with the demands of academic jobs or inability to secure a tenured position).

Assessment of Fellows' Long-term Career Success and Contributions as Academics:

We will follow the careers of Fellows after they leave Lawrence, asking them occasionally to update us on their current status, much in the way that Alumni Offices track alumni. At intervals of 1, 3, and 7 years, we will attempt to obtain current academic vitas, which would be easy for former Fellows to email as attachments, are usually kept up to date by academics, and contain a wealth of information on the status of the individual's career, contributions, and recognitions. We believe that while it is not realistic to expect former Fellows to complete long questionnaires after they have left Lawrence, they will be willing to respond to an occasional email requesting their current vitas. Vitas ought to document such important indicators as reappointments and promotions, publications and performances, quality of the institutions at which Fellows get jobs, whether Fellows have remained in academia, academic honors (e.g., prizes for scholarly contributions, teaching awards), and whether Fellows continue to teach courses they developed at Lawrence.

Additionally, during an exit interview and at the 3 and 7 year follow-ups we will invite Fellows to complete a short on-line survey concerning the degree to which they believe the Fellows program affected their career paths and whether the program, in their view, helped or hindered their careers. These surveys will, in addition to a short set of questions that can be responded to with numerical ratings, have a narrative portion on which Fellows can reflect on the impact that the program had on them.

Outcomes for Lawrence Students

Enrichment of the Curriculum

The Fellows program provides an added pool of teaching faculty on the Lawrence campus whose courses will enrich the curriculum. Not only are the Fellows expected to have a direct effect by offering courses and topics that otherwise might not be taught, but their presence ought to have indirect effects by freeing up tenure-track faculty to offer courses that they may not otherwise have been able to teach (when a Fellow takes over an existing course). Fellows are also expected to sponsor individualized instruction opportunities (in addition to their prescribed course loads) in Independent Studies (ISs) and tutorial classes. The end result ought to be greater curricular diversity and, as another benefit, an ability to maintain small class sizes across the University (given the net increase in courses taught) than would otherwise be the case. (Because the Fellows program at Lawrence has begun at a time of sharply increased overall enrollment, its contribution may be to maintain lower class sizes when they otherwise would have risen; statistical analyses will therefore control for overall enrollments when examining the impact of the program.)

Assessment of Impact on the Curriculum:

The course schedule is all that is required to document the courses Fellows offer, which can be considered to be net additions to the curriculum. Additionally annual department reports by department Chairs will be used to assess how Fellows affected departmental offerings. Chairs will be asked to explain not only the direct, but the indirect effects that Fellow's offerings had on the curriculum (e.g., Did other department faculty alter what they offered due to the presence of the Fellow? If so, how?). Chairs will specifically be asked to list areas that were covered (either by Fellows or faculty) that would not have been covered without the Fellow's presence (to create an index of the impact of the Fellows on the richness of the curriculum), noting the level (introductory, intermediate, advanced) at which these added offerings were taught. As in other instances in which Chairs will be asked to assess the program, the anticipated benefits articulated in the initial departmental recommendations to interview and hire Fellows (which involved an extensive and standard set of questions) can be used as a basis for reflection on whether goals were met and to make salient any unanticipated benefits that occur.

Individualized instruction can be documented by reports from the Registrar on ISs and tutorials sponsored by Fellows. Fellows will also be asked in their annual reports to note any individualized instruction they offered that was not offered for credit and therefore not "on the books."

These data on course offerings can be enriched with narratives by selected samples of students, perhaps junior and senior majors in the relevant department, including students who completed courses or took ISs or tutorials offered by Fellows (Advanced Majors Survey). These students can speak to the impact of the Fellows' offerings (e.g., how a course that otherwise would not have been offered allowed them to develop crucial skills for graduate school).

Finally, to assess the overall benefit of Fellows' presence on class sizes at Lawrence, data on enrollment kept by the Registrar can be use to track trends in class sizes within departments and across the University. The additional courses offered by Fellows ought to reduce overall class sizes. Departments who have Fellows may service more students, but hopefully with fewer students per class than would otherwise be the case. Regression analyses will be used to control for the effect of the overall size of the student body to obtain a more accurate estimate of the effects of presence of Fellows on class size.

Increased Research and Performance Opportunities

Fellows are expected to increase the number of opportunities students have to engage in research or performance collaborations. Some of these collaborations may occur through the ISs and tutorials mentioned above.

Assessing Research and Performance Opportunities:

Fellows will be asked to document research and performance collaborations with students as part of their annual reports and to include any finished products (e.g., co-authored articles) as appendices to those reports. The survey of advanced majors in the departments that have Fellows (as noted in the prior section) will include questions that specifically address research and performance collaborations with Fellows and the impact that they have had on the students who have engaged in them.

Advising for and Connections with Graduate Programs

Because the Fellows will all be recent PhDs, we expect that they will serve as informal advisors to Lawrence students about graduate school (where students ought to apply to best serve their interests, how to structure a successful application, how to assess which graduate school is the best fit for the individual, etc.). Additionally, because Fellows will still have strong connections to and, probably, frequent contact with faculty from their graduate programs, it is likely that students will become connected to those programs through the Fellows (e.g., a student may be given personal recommendations by Fellows as someone their mentor might want to work with over the summer, alerted to internship possibilities at the graduate institution, introduced to the graduate mentor informally, etc.)

Assessing Advising for and Connections with Graduate Programs:

The survey of advanced majors within departments that have a Fellow will specifically ask about whether the Fellows have served as informal advisors about graduate school. Students will also be asked about any specific connections they made at research institutions that were due to a Fellow's interventions.

The student surveys will be supplemented by questions on Chairs' and mentors' annual reports because they may also be aware of ways in which Fellows promoted connections between students and graduate schools.

As part of their annual reports, Fellows will also be asked to report on any informal advising or promotion of students' connections to graduate programs. Additionally, they will be asked to list participation in any formal talks or discussion sessions on graduate school (e.g., participation in a departmental advising workshop on graduate work), any contacts they established between students and graduate programs, and how many students for whom they have written recommendation letters for applications to graduate school.

Outcomes for Lawrence University

Academic Connections and Recognition

Fellows' continuing connections to their graduate programs ought to enhance and help to create additional connections between Lawrence and research universities (e.g., increase the number of Lawrence students who attend graduate school or find summer internships at research universities, increase research collaborations and joint grants involving Lawrence faculty and research schools). Those connections and any attention garnered by the Fellows program (e.g., articles in the higher education press, greater awareness of Lawrence among faculty at graduate schools) will enhance recognition of Lawrence.

Assessing Academic Connections and Recognition:

Media coverage of the Fellows program (which the Communications Office will systematically track) offers one index of increased recognition of Lawrence due to the program. Fellows Annual Report forms will request documentation of any connections that Fellows have initiated between their graduate program and Lawrence (including new cooperative programs, research and performance collaborations, student connections, etc.). Fellows will be encouraged to develop such connections, including inviting faculty from their graduate programs to talk at Lawrence (another way to increase recognition and another item to be listed in Fellows' annual reports). Additionally, any publications or presentations, awards or honors that individual Fellows produce or receive in their fields will, due to their association with Lawrence, increase recognition of the institution. (Fellows will be asked to include their Lawrence affiliation on any publications produced wholly or in part while at Lawrence; such public acknowledgments of Lawrence can be counted as another indicator of increased recognition.)

Department Chairs' and mentors' annual reports will also provide information about connections to research universities or recognition of Lawrence. Chairs will be asked about their own subjective assessments about whether Fellows have increased students' interest in and success at applying to graduate programs and to document any increased connections to graduate schools (e.g., faculty collaborations, talks given at Lawrence by a graduate program's faculty or by Lawrence faculty at a graduate program).

The survey of advanced majors will also include questions about connections they have personally made with graduate programs due to the Fellows. This will include research connections, internships obtained, personal contacts made with faculty at the graduate schools, degree of (and changes in) interest in graduate school generally or the specific program the Fellow attended.

Another possibility is to email Fellows' graduate school advisors (Graduate Advisor Survey) requesting that they take a very brief internet-based survey concerning their impressions of Lawrence and of the Fellows program based on the connection established by having one of their former students in this program (perhaps after the Fellow's first year at Lawrence). This brief survey might be a springboard for fostering connections (e.g., a prelude to an invitation for the respondent to come and speak at Lawrence). The survey would assess whether respondents' knowledge about and evaluation of Lawrence has changed (e.g., are they more interested in having Lawrence students apply to their program?) and whether the program has fostered greater interest in connections to Lawrence. (This survey will be brief to elicit participation, but would make use of the Fellows' names as an entrée.)

Intellectual Excitement on Campus

Overall, the Fellows program ought to enrich the intellectual atmosphere on campus by bringing recent PhDs who are deeply engaged researchers and performers who will share their interests and activities with students and faculty at Lawrence. Fellows will not only bring new classes on new topics, but will also give talks, performances (in existing forums, such as Science Hall Colloquia), and engage in collaborations with students and faculty. All of this ought to enhance the vibrancy of scholarship at Lawrence and enhance the intellectual climate of the University, both directly and indirectly.

Assessing Intellectual Excitement:

Fellows will be asked in their annual reports to document the ways in which they have contributed to the intellectual life of the campus, including presentations and performances, research and performance collaborations, and discussion groups they have participated in or organized. Similar questions will be asked of department Chairs and mentors in their reports.

In the survey of department majors, questions will address their subjective assessment of how the Fellows have affected their enthusiasm about their academic areas and their intellectual development. Students will also be asked to provide examples of specific instances that illustrate contributions Fellows have made to the intellectual life of the campus. Additionally a brief, internet survey of faculty could ask for subjective assessments of the contribution of the Fellows program to the intellectual life of the University.

Outcomes for Faculty

Transfer of Ideas and Techniques

The constant influx of new PhDs to Lawrence ought to lead to significant transfer of new ideas and techniques (e.g., laboratory procedures, performance innovations) to Lawrence faculty and students. The benefits of this transfer will remain long after individual Fellows have left Lawrence.

Assessing Transfer of Ideas and Techniques:

Fellows will be asked in their annual reports to document instances of the transfer of ideas or techniques, such as specific laboratory procedures or performance innovations that they have taught to students or shared with Lawrence faculty, in informal opportunities, the classroom, workshops, and collaborations. Mentors, department Chairs, faculty more generally, and students will also be asked (in annual reports and surveys) about examples of such transfers and their evaluation of their impact. Additionally, Lawrence faculty could be asked in their annual reports about any examples of how they have benefited by transfer of ideas and techniques from Fellows.

Faculty-Fellow Research and Teaching Collaborations

Because it is likely that there will be greater overlap of professional interests between Fellows and Lawrence faculty than commonly occurs among the faculty. This ought to encourage a greater likelihood of professional collaborations that not only add to the intellectual atmosphere of the campus, but can reinvigorate Lawrence faculty, keeping them more connected to their own fields and expanding their professional activities and contributions. Similarly, with overlap between Fellows' and faculty members' interests, co-teaching is a likely outcome (and is also encouraged to help develop Fellows' teaching skills by working with an experienced member of the faculty).

Assessing Faculty-Fellow Research and Teaching Collaborations:

Research and teaching collaborations between Fellows and faculty will be documented through the annual reports of Fellows, department Chairs, and mentors, which will specifically ask about these collaborations. Joint publications and performances by faculty and Fellows will provide objective indicators of the amount of professional collaboration (which could be compared to the frequency of such collaborations among tenure-track faculty). Given that we will encourage a greater degree of overlap of interests between Fellows and faculty than we do among the tenure-track faculty, there ought to be a significantly greater number of Fellow-faculty collaborative efforts than is normally the case among the faculty. The presence of Fellows as collaborators may also increase the production of scholarship among Lawrence faculty - which can be assessed by comparing current productivity to average productivity from previous years (from information on annual faculty reports).

Subjective assessments of the impact and quality of these teaching and research collaborations will also be solicited from the Fellows, faculty members, and students involved in them (as well as from faculty more generally -- Brief Faculty Survey. Narrative comments concerning these collaborations from the individuals involved in them will provide insight into the effects of these collaborations.

Faculty Members' Opportunities to Teach Advanced Courses and to Provide Individualized Instruction

The presence of the Fellows ought to create more flexibility in terms of what Lawrence faculty are able to teach (by providing an extra pool of teaching faculty). This, in turn, may enable Lawrence faculty to teach advanced courses they otherwise would not be able to make room for in their schedules or to devote more time to individualized instruction.

Assessing Faculty Members' Opportunities to Teach Advanced Courses and to Provide Individualized Instruction:

Documentation of changes in what departmental faculty have been able to teach (including ISs and tutorials) due to the presence of Fellows will come from department Chairs' reports (because Chairs understand what courses faculty members would have had to teach if a Fellow had not been present). This question can also be asked in a brief faculty survey. As noted above, the beginning of the Fellows program coincides with a large increase in overall enrollment, mainly due to a much larger incoming Freshman class. Without the Fellows, it is likely that many advanced classes would not be taught as faculty are forced by enrollment pressures to switch to lower-level courses.

End of Year Evaluation and Retreat

At the end of each academic year, Fellows, mentors, and department Chairs will be asked, as part of their annual reports to assess the success of the Fellows program. The strengths and weaknesses of the program will be discussed at a Bjorklunden retreat weekend toward the end of the academic year, attended by Fellows, mentors, and Fellows Committee members. Possible improvements to the program that can be implemented during the next academic year will be discussed.

Summary of Evaluation Plan Goals and Measures

Benefits to Fellows and Wider Academic Community

Developing Excellent Teachers

Developing Excellent Scholars/Artists

Fostering Long-Term Career Success and Contributions

Benefits to Students

Enrichment of Curriculum

Increased Research and Performance Opportunities

Advising for and Connections to Graduate Programs

Benefits to Lawrence University

Academic Reputation and Recognition

Intellectual Excitement on campus

Benefits to Faculty

Transfer of Ideas and Techniques

Research and Teaching Collaborations

Faculty Opportunity to Teach Advanced and Individualized Courses