Systematic Program Review - Report to Overview Committee

SPR Home

Systematic Program Review
Report to Overview Committee
May 2001

Introduction
Since its launch in May 1999, Systematic Program Review (SPR) has become an increasingly visible component of the University's academic planning activities. In the first year of implementation, detailed policies and procedures were established; guidelines and templates were developed; a website was created; and reviews of almost thirty program areas were undertaken. This 'year-end' report focusses on activities in 2000/01, the second year of implementation.

Outcomes of 1999/2000 Reviews
SPR reached a milestone in April 2001, when the Planning Committee submitted its report to Council on the outcomes of the first set of reviews. Procedures for this 'outcomes' phase were developed during the Fall of 2000, enabling the final portions of the approved SPR Policy and Procedures to be put into practice. Briefly, the procedures are as follows (some of these activities may take place concurrently):

The Vice-President Academic (VPA) and Dean of Graduate Studies and Research (Dean, CGSR) consider self-study documents, reviewers' reports, department/Deans' responses, and their own discussions with reviewers, and prepare recommendations on the assessment category (A, B, C or D) to be assigned to each program.

VPA and Dean, CGSR forward recommendations to Planning Committee along with all review documents. (Self-study documents are available for consultation in the University Secretary's Office.) Recommendations are copied to relevant Deans and to PhD Committee of CGSR¹.

VPA and Dean, CGSR meet with relevant Deans to discuss possible outcomes and action plans.

Planning Committee endorses or rejects recommendations of VPA and Dean, CGSR, and informs relevant program Deans, VPA, and Dean, CGSR of its decisions.

Planning Committee reports results of reviews to Council, including reasons for any disagreements with VPA and Dean, CGSR.

Program Dean and program head(s) develop an 'action plan', subject to approval by VPA and Dean, CGSR.

Action plans, reviewers' reports, responses, and other information from reviews are available to the committees of Council and CGSR for use when decisions on programs and resource allocation are required (and particularly when program proposals arising from SPR are assessed).

Because the 1999/2000 reviews were the first of a new process, there was some delay in reaching the recommendations stage. However, once decisions had been made on matters such as the form and content of the recommendations, communications protocol, and procedures for action plans, the Vice-President Academic and Dean of Graduate Studies and Research were able to prepare recommendations on the outcome category for each program reviewed in 1999/2000. These were submitted to the Planning Committee on February 5, 2001.

Throughout February and March, the Planning Committee assessed the reviewers' reports, departmental and Deans' responses, and outcome category recommendations. The Committee's decisions on category assessments were reported to the April 2001 meeting of Council (see Attachment I). Four programs were rated A, eleven B, and fourteen C. There were no Ds.²

In every case, the Vice-President Academic and Dean of Graduate Studies and Research concurred with the review team's assessment, and the Planning Committee endorsed each recommendation. Such agreement is a tribute to the exceptional quality of the reviewers, who were recruited from top universities throughout North America. A list of reviewers is attached (Att. II).

While the reviewers' recommendations ranged from refocussing and updating curricula to strengthening basic research, the Planning Committee's report also noted that 21 of the 29 reviews called for increased faculty resources, and that most identified additional resource shortfalls. It is hoped that governments will take notice of the very real impact their systematic underfunding is having on the quality of post-secondary education.

In the presentation to Council and subsequent media coverage of these outcomes, the overwhelming majority of discussion was positive. The fact that the University now has a review process has in itself generated enthusiasm, as it demonstrates our commitment to improving the quality of the programs we deliver to students. The review standards, developed and approved by our own faculty, are intentionally demanding, and the programs rated 'A' under this system were ranked among the best in the world. Even the 'C' programs, with only a few exceptions, were praised by the reviewers for their dedicated, high-quality faculty. We have received excellent guidance for improving all programs, and plans are already underway to address the issues raised by the 'C' reviews. The hard work of the units under review was also acknowledged, and it is due to their efforts as much as to the reviewers that SPR is on the way to achieving its primary goals of program improvement, accountability, and provision of information for program and resource allocation decisions.

Significant consequences are attached to the designation of an A, B, C or D, particularly in the case of 'D' programs, which will be considered for termination, and 'C' programs, which must have an action plan approved within a specified timeframe or be considered for termination. One of the unique features of Systematic Program Review is its combination of summative review (grade with consequences) and formative review (suggestions for improvement), and the actions arising from this first round of reviews will prove the wisdom of that approach: programs rated 'C' cannot simply let their reports sit on a shelf!

While the outcome categories have general consequences, it is the 'action plans', currently under development, which will specify detailed strategies, timelines and responsibilities, and from which proposals for program revisions will flow. A key feature of these action plans will be their relationship with college strategic plans and planning processes.

SPR policy indicates that the results of program reviews are to be used for planning purposes at the University level as well. It is expected that Council and its committees will work to make SPR an integral part of their planning and approval processes - and some sub-committees have already done so. Since actions arising from SPR are subject to the University's existing processes of negotiation and approval, links between review outcomes and academic or resource planning may not always be explicit; but they will be there nevertheless, reflected in plans for program revision, resource allocation, identification of priorities, and new initiatives, at all levels.

Reviews 2000/2001
The following programs were reviewed in the second year of SPR:

Education: graduate programs in Educational Administration, Educational Foundations (include the M. Cont. Ed.), and Curriculum Studies;

Law: undergraduate LLB and graduate LLM programs;

Nursing: undergraduate NEPS and BSN (post-registration) and graduate MN programs.

Arts & Science: two interdisciplinary undergraduate programs, Regional and Urban Development (RUD) and Land Use and Environmental Studies (LUEST).

A list of reviewers is attached (Att. III). Site visits for all of these programs have taken place, and reports have been received for the earlier reviews. The Nursing review was a particular challenge to organize due to the involvement of SIAST (both Kelsey and Wascana), and we would like to acknowledge the extra efforts that Nursing faculty and staff made to ensure an effective review. All of the 2000/01 site visits were completed successfully, and we await the results with interest.

The Master's program in Agricultural Extension was also originally scheduled for review in 2000/01. However, on near-completion of their self-study, the program coordinators and relevant Deans agreed to propose program termination voluntarily rather than incurring the expense of an external review that would serve no useful purpose - a true 'self study'.

Reviews 2001/2002
The following programs are scheduled for review in the academic year 2001/02:

Engineering: all undergraduate and graduate programs;

Natural Sciences: most undergraduate and graduate programs delivered by departments within the Science Division of the College of Arts and Science, and some interdisciplinary BSc programs;

Education: undergraduate (BEd) programs;

Music: undergraduate programs, including the combined BEd/BMus(MusEd).

Review scheduling is rarely straightforward, and this list represents the outcome of much discussion and consultation. 2001/02 will be a heavy year, with at least 5-7 site visits per term and two additional site visits (BEd and Music) to be carried over to September 2002.

Preparation for these reviews began in January 2001, when organizational meetings were held with Deans and Department Heads from each College and Division. We are now in the process of requesting nominations for reviewers, providing statistics and other information, and guiding units through the self-study process.

Additional Activities and Comments, 2000/01
Communication: The SPR Coordinator and Executive continued efforts to inform and assist others within and outside the University, including the following:

CIRPA Conference: Caroline Davis and Pauline Melis presented a paper, "The Joy of Reviewing: Implementation of a New Program Review Process", at the October 2000 conference of the Canadian Institutional Research and Planning Association (CIRPA). The paper and powerpoint presentation are both available on the CIRPA website at http://www.cirpa-acpri.ca/prevConferences/saskatoon2000/proceedings.html.

Since the CIRPA conference, Dr. Davis has frequently provided advice to the new program review coordinator of Memorial University, who is using our process as a model.

The Coordinator developed a 'Frequently Asked Questions' document, Advice to Units Preparing for Systematic Program Reviews, which was distributed to all heads of programs undergoing review in 2001/02 and will eventually be published on the SPR website. This document responds to FAQs about the process (e.g. what counts as a program, how review teams are selected) and provides advice on matters such as the self-study, alumni surveys, and the site visit.

An interim report for 2000/01 was presented to the SPR Overview Committee in November and transmitted to Council in December 2000. In connection with that report, the SPR Coordinator also met with the Budget Committee of Council in November.

A special SPR session of the Department Head Leadership Program was arranged for associate and assistant Deans in March, 2001, where those who had been through the process shared advice with others.

Presentation to the Department of Post-Secondary Education and Skills Training: On May 3, 2001, Michael Atkinson, Gary Kachanoski, Caroline Davis and Pauline Melis made a presentation on SPR and its outcomes to the Department in Regina, followed by an informal and very positive 1 1/2-hour discussion.

The 1999-2000 outcomes were widely publicized, with a press release, article in On Campus News, articles in the StarPhoenix, coverage by local radio and television, and a panel/phone-in show on Shaw Cable 10.

SPR Resources: SPR financial and human resources were covered in the November 2000 SPR report (forwarded to Council in December 2000). In brief, the direct costs of reviews are approximately $6000 - 8000, depending on the number of externals (includes honoraria, expenses, printing and postage); and direct operating costs are $20-30K per year (includes secretarial support). In-kind support and costs borne by the units under review are significant, but difficult to calculate. Human resource requirements have not changed, but there have been some changes in specific personnel and duties within the College of Graduate Studies and Research (CGSR), where resources are assigned to SPR-related activities such as the preparation of Graduate Program Reports, updating of databases, and handling of actions resulting from reviews. The Coordinator (1.0 FTE) is still provided by the University Studies Group. A special note of thanks is due to Sandra Calver, formerly of CGSR and now in the Office of the Vice-President Academic, who served as Acting SPR Coordinator throughout February 2001.

Response to Feedback: While the vast majority of comments about SPR have been positive, improvements are continually being made to the process in response to suggestions from reviewers, units under review, and others. This year, for instance, we learned a great deal about the kind of support needed to steer undergraduate interdisciplinary programs through the review process; and the new Advice to Units document is based largely on participants' questions and experiences.

The November 2000 report provided a summary of feedback and observations on the process, which will not be reiterated here. It should be noted, however, that the units under review this year have echoed last year's favourable comments about the self-study process. It revealed issues of which they had been unaware, brought issues to the fore that had been recognized but not discussed (sometimes for years), and helped to clarify objectives and future possibilities. The SPR student outcomes survey templates have also continued to prove useful, and many units are eager to continue surveying their graduates on a regular basis.

Plans for the future
Besides undertaking the large number of reviews scheduled for the next few years and managing outcomes of the completed reviews, SPR administrators will continue to improve and assess the process itself. Long-term tasks include the development of standardized SPR data packages and an interactive web-based data system, and the revision of policy and procedure documents as necessary. Current documents, including the six-year review schedule, are posted on the SPR website at http://www.usask.ca/vpacademic/spr. We will also continue to communicate the process and activities to the University community and beyond, including participation in a workshop planned for Council and its committees in September.

While SPR has already produced some tangible benefits (outlined in the Nov. 2000 report), the submission of the Planning Committee's report on outcomes of the 1999/2000 reviews marked an important new phase of the review process. Now that outcome categories have been determined and plans for the implementation of recommendations are under development, it is time to translate the reviews into action!

This report is submitted with thanks to SPR participants from the units involved, the review teams and SPR staff, whose efforts made it possible to reach this stage of a complex, arduous, but ultimately rewarding process.

Caroline Davis
Coordinator of Systematic Program Review

¹ The Policy and Procedures state that the recommendation will be submitted to the PhD Committee of the College of Graduate Studies and Research as well as to the Planning Committee of Council; however, with Graduate Program Review and SPR now merged into one process, it has been agreed that outcomes recommendations are to be made to the Planning Committee alone, with a copy to the PhD Committee.

² See SPR Policy and Procedures, Section XIII for an explanation of the four outcome categories.

Attachments:

I. Outcomes of 1999/2000 Reviews [Download here]

II. SPR Review Teams 1999/2000 [Download here]

III. SPR Review Teams 2000/2001 [Download here]

V. "In-kind" Costs of SPR [Not Available for Download]

VI. Expenditure Forecast [Not Available for Download]

SPR Home

I.	Outcomes of 1999/2000 Reviews	[Download here]
II.	SPR Review Teams 1999/2000	[Download here]
III.	SPR Review Teams 2000/2001	[Download here]
V.	"In-kind" Costs of SPR	[Not Available for Download]
VI.	Expenditure Forecast	[Not Available for Download]