Systematic Program Review - Annual Report

SPR Home

Systematic Program Review
Annual Report
May 2002

Introduction
The University of Saskatchewan's program review process, Systematic Program Review (SPR), has almost reached the half-way point in its first six-year cycle. By the end of this academic year, 62 graduate and undergraduate programs will have undertaken in-depth self-studies, been reviewed by highly qualified external reviewers (89 in all), and received recommendations for program improvement. Programs reviewed in the first two years have begun the crucial task of implementing changes, many of them major. The policies and procedures established during the first year of SPR have proved effective, while continuing to be refined through practice. As we complete the third year of SPR, our 2001/02 report can look back in pride on the accomplishments of SPR staff, reviewers, and the programs reviewed, while recognizing issues that remain to be addressed.

Section III (page 3) of this report notes a minor change to the text of the SPR Policy and Procedures, which was approved by the SPR Overview Committee at its April 12, 2002 meeting in order to clarify the meaning of the SPR assessment categories (see Attachment I).

Update on the Reviews of 1999/2000
In April 2001, the Planning Committee reported to Council on its assessment of the programs reviewed in 2000 - the first set of outcomes from the new process. These outcomes were covered in the May 2001 SPR Report: very briefly, four programs were rated A, eleven B, fourteen C, and none D.¹ Reaction to the Council presentation and subsequent media coverage was positive, revealing broad support for program review and for the commitment such a process demonstrates to the improvement of academic programs. The review standards, developed and approved by our own faculty, were noted to be intentionally demanding; external experts ranked the 'A' programs among the best in the world. Even the 'C' programs, with only a few exceptions, were praised by reviewers for their dedicated, high-quality faculty.

One year on from these outcomes assessments, major changes are underway. Actions resulting from SPR (some of which are still at the proposal stage) include: major curriculum revisions; clarification of program objectives; program mergers and departmental restructuring; conversion of weak majors to minors; improved policies and procedures relating to admissions, administration, advising, and course evaluation; initiatives to enhance the student experience; strategic hiring of new faculty; introduction of mechanisms to encourage faculty research (e.g. teaching release, grant writing workshops, mentoring); and the creation of new positions crucial to program delivery. Other types of changes may be in the offing: for example, the College of Graduate Studies and Research (CGSR) is discussing the possibility of devolving some administrative authority to 'A'-rated graduate programs, if they wish.

As expected, some SPR procedures were modified and new procedures established in the course of this first outcomes process. The SPR 'action plans' envisaged by the original procedures will likely be superseded by the University's new Integrated Planning Process, in which responses to SPR (including undergraduate program responses) are incorporated into college plans. At the graduate level, the CGSR has established additional procedures: an SPR Outcomes Subcommittee² has been established as a joint committee of the Ph.D. and Academic Committees, together with additional members of the Graduate Faculty, to evaluate and assist in the development of the 'graduate action responses' required from all graduate programs rated 'C'. The 'C' programs from 1999/2000 were asked to submit action responses by January 15, 2002. If action responses are not approved by the committees and Dean of Graduate Studies, admissions to the program in question are suspended, effective September 1, 2002, until a satisfactory response is received and approved.

The responses to the first round of SPR reviews have proven the wisdom of combining formative review (recommendations for improvement) with summative review (assessment categories, with attendant consequences): programs rated 'C' cannot let their reports sit on a shelf!

Update on the Reviews of 2000/2001
The following programs were reviewed in the second year of SPR:

Education: graduate programs in Educational Administration, Educational Foundations (including the M.Cont.Ed.), and Curriculum Studies;


Law: undergraduate LLB and graduate LLM programs;


Nursing: undergraduate BSN programs (Nursing Ed. Prog. of Sask. (NEPS) and post-registration program) and graduate MN program;


Arts & Science: two interdisciplinary undergraduate programs, Regional and Urban Development (RUD) and Land Use and Environmental Studies (LUEST).


These 11 programs were assessed in 7 site visits, by 25 reviewers. As in the previous year, external reviewers were drawn from respected institutions throughout Canada and beyond, including the University of British Columbia, Queen's, OISE/Toronto, Texas A&M, Sydney (Australia), and others (see Attachment III).

In early January 2002, once all of the reports and responses had been received, the Vice-President Academic and Acting Dean of Graduate Studies and Research forwarded recommendations to the Planning Committee of Council. In all cases, they concurred with the assessment of the review team, although they did note that in one case (Educational Foundations), a minority report had been submitted.

The Planning Committee of Council, which is responsible for determining outcomes assessments, reviewed all of the documentation and reported its assessments to the April 18th meeting of Council. The results, listed in Attachment IV, comprised eight Bs and three Cs. Apart from one comment related to self-study workload, the report generated no discussion at Council. The results were announced in a news release on the same day.

As in the previous year, no 'D' ratings were assigned. However, on near-completion of the self-study for the Master's program in Agricultural Extension (scheduled for review in 2000/01), the program coordinators and relevant Deans agreed to propose program termination voluntarily.

Changes are already occurring as a result of the 2000/01 reviews. For instance, the College of Arts and Science has established a new Office of Interdisciplinary Studies to provide administrative and clerical support to interdisciplinary programs; the College of Nursing has been authorized to recruit eight new faculty members to support the NEPS program; communication between graduate programs and the College of Graduate Studies and Research has improved; major curriculum revisions are underway, including some expansion of practical components; some units are providing additional support for faculty upgrading; and the role of the project-based Master's degree is being clarified.

In addition to its assessment, the Planning Committee identified several concerns arising from these reviews: the broad range covered by the 'B' category; the comparability of outcome categories across different colleges; the need to support interdisciplinarity; and the practice of the College of Graduate Studies and Research not to allow program changes prior to SPR review. One of these concerns, the definition of the 'B' category, was brought to the SPR Overview Committee at its April 12th meeting. The Overview Committee agreed that, for clarification and consistency, the following revisions should be made to the category definitions found in the SPR Policy and Procedures, Section XIII:

Category A: delete the word 'fundamental', i.e. "Few, if any, fundamental changes are required" is revised to read "Few, if any, changes are required."


Category B: delete the word 'fundamental', i.e. "Some fundamental changes should be made" is revised to read "Some changes should be made."


Categories C and D are unchanged (C continues to include "several fundamental changes must be made" and D to include "many fundamental changes are required"). These revisions will come into effect for programs reviewed in the academic year 2002-03 (including reviews 'carried over' from 2001-02). Changes are indicated in Attachment I.

Update on the Reviews of 2001/2002
The following programs were under review in the academic year 2001/02:

Engineering: all undergraduate and graduate programs;


Natural Sciences: undergraduate and graduate programs delivered by the departments of Chemistry, Computer Science, and Physics & Engineering Physics, together with some interdisciplinary programs.


In addition, the following reviews will be 'carried over' from 2001/02 to the first term of 2002/03:

Education: undergraduate (BEd) programs;


Music: undergraduate programs in Music and Music Education;


Natural Sciences: undergraduate and graduate programs delivered by the Department of Geological Sciences, together with some interdisciplinary BSc programs. (Programs in Mathematics and Statistics will be reviewed at a later date.)


By the end of May 2002, site visits will have been completed for all of the Engineering programs and for the first three Natural Sciences areas, and most of the reviewers' reports and program responses will have been received. (Reviewers are listed in Attachment V.) The programs which were carried over to 2002/03 are in the midst of their self-studies, and the Vice-President Academic and Dean of Graduate Studies and Research are currently selecting review teams.

Update on Reviews of 2002/03
In addition to the programs 'carried over' from 2001/02, the following programs are scheduled for review in the academic year 2002/03:

Commerce: all undergraduate and graduate programs except for the MBA, which was reviewed in 1999/2000;


Social Sciences: most undergraduate and graduate programs delivered by departments within the Social Sciences Division of the College of Arts and Science, together with some undergraduate interdisciplinary programs. Programs in Psychology will be reviewed early in 2003/04 in order to tie in better with the timing of their accreditation review, and programs in Anthropology will be reviewed with the Humanities, in line with the upcoming departmental restructuring.


With fifteen site visits to schedule, 2002/03 is likely to be a heavy year for SPR staff. As usual, every effort was made to accommodate reasonable program requests, including those mentioned above, other delays and rescheduling, and special groupings of programs for review.

Preparation for these reviews began early in 2002, when organizational meetings were held with Deans and department heads from each College and Division. We are now in the process of requesting nominations for reviewers, providing statistics and other information, and guiding units through the self-study process.

Additional Activities and Comments, 2001/02
Changes to Organizational Structure: During the first two and half years of SPR, procedures were established and reviews coordinated by a full-time Coordinator, Caroline Davis, with the support of a full-time secretary. In September 2001, on Dr. Davis' return to the University Studies Group, the position of coordinator was split into three part-time secondments: Murray Fulton (Acting Associate Dean of CGSR) as Academic Director of SPR, Tonya Wirchenko (Director of Graduate Programs) as SPR Administrator, and Brenda Wright (Assistant Graduate Program Administrator) as SPR Administrative Assistant. Attachment VI outlines the basic duties of these positions. In January 2002, when Dr. Fulton took administrative leave, Dr. Davis was reassigned to SPR for a six-month term as Academic Director. Recruitment is now underway for the half-time position of Associate Dean (Systematic Program Review), CGSR.

Communication: The Academic Director/Coordinator of SPR and the SPR Executive continued efforts to inform and assist others within and outside the University, including the following:

The SPR report for 2000/01 was approved by the Overview Committee and transmitted to Council in May 2001.


In May 2001, the SPR Executive made a presentation on SPR and its outcomes to the Department of Post-Secondary Education and Skills Training. This was followed by an informal discussion, with a very positive response received from the Department.


SPR continues to be featured in speeches of the President and senior administration, in college newsletters (e.g. Agriculture, Law), and in the University's Annual Report (headline in 2000/01 report: "Guaranteeing quality education").


Dr. Davis revised the 'Frequently Asked Questions' document, Advice to Units Preparing for Systematic Program Reviews, which has been widely used and welcomed by colleges and departments. With the assistance of the SPR Administrator, she is currently preparing a 'Frequently Asked Questions' document for reviewers.


Dr. Fulton published his reflections on SPR in the January 25, 2002 edition of On Campus News (headline: "SPR a positive force for programs, understanding & openness").


Dr. Davis provided input into an HRD Canada study of the evaluation factors used to assess the quality of higher education systems in Canada.


A news release was prepared to coincide with the announcement of the outcomes of the 2000-01 reviews to Council on April 18, 2002.


Plans are underway for communications this Fall with the University community (including students) and with the new Department of Learning.


Subsequent Review Cycles: Once all programs have been reviewed by SPR, subsequent review cycles will likely be based on a different approach. At its April 12th meeting, the Overview Committee discussed the possibility that only programs rated 'C' in the first cycle would undergo in-depth review in the next cycle.

Feedback and Observations on the Process: Most of the feedback received in 2000-01 echoed comments summarized in previous reports. While the vast majority of feedback was positive, challenges identified by the programs under review continued to be the workload required for the self-study process and some confusion around the outcomes process, including the length of time taken to determine outcomes after the review.

With respect to the outcomes process, it should be noted that in the first few years of SPR, most of the emphasis was placed on establishing the 'front end' of the process, such as scheduling, self-studies, reviewer selection, site visits, and reporting. As indicated above (Section II), the College of Graduate Studies and Research has now developed a process for 'action responses' to graduate reviews, and it is expected that college-level SPR action plans (including undergraduate actions) will be incorporated into college plans within the University's new Integrated Planning Process.

With respect to the self-study workload, SPR has established several options for assistance, explored additional mechanisms, and clarified expectations for report length and detail. Better communication is needed, since there continues to be a tendency for units to produce more material than is actually required. Reviewers themselves have commented on the excessive documentation (although they continued to be impressed with the process, with their reception on campus, and with our faculty, staff and students). It should be noted in this connection that according to some heads of accredited programs, SPR requires less documentation than many accreditation processes.

Although units expressed concern about workload, they invariably qualified these comments with praise for SPR. The self-study process itself was considered extremely helpful in revealing issues, clarifying objectives and suggesting future directions, and participants appreciated the opportunity to obtain feedback from their current and former students. In some cases, an unanticipated benefit was an improvement in relations between various providers of a program. Reviewers were also commended: with a few exceptions, faculty felt they showed insight into their programs, provided valuable advice, and provided an external validation of both their concerns and their strengths.

Each review provides new learning opportunities for SPR staff: different disciplines and types of programs (e.g. interdisciplinary) raise new challenges, and participants are always able to come up with a request we have not encountered before! The Advice to Units and forthcoming Advice to Reviewers are based largely on participants' questions and experiences, and SPR policies, procedures and guidelines are continually being refined in response to feedback.

Murray Fulton's On Campus News article identified additional messages that have emerged from the SPR process. He noted that, without exception, reviewers have focused on "academic quality … on the many things that are part of a well-functioning and vibrant academic environment". While the issues raised by reviewers were very diverse - reflecting the tremendous diversity in our program array - three issues appeared frequently: (i) with extra work and innovative ideas, our faculty are managing to provide high-quality programs with the resources available; (ii) in general, we are putting more effort into undergraduate teaching than into graduate programs and research (and the ratings suggest strong graduate programs enhance the associated undergraduate programs, while the converse is not true); and (iii) the University's Aboriginal initiatives, so vital to the province, are "generally on the right track", although they need development. Dr. Fulton also commented on the efforts units put into their reviews, the benefits of the process, the high quality of the program proposals arising from SPR, and the ways in which SPR has contributed to openness across campus and a better understanding of the diverse programs we offer.

This report is submitted with gratitude to all participants from the units under review, to the review teams, to SPR staff, and particularly to Murray Fulton for his four-month term as Academic Director. Thanks to their hard work and enthusiasm, SPR is well on the way to achieving its fundamental goals: demonstrating accountability for the programs we offer, providing information to assist in decisions about program revisions and resource allocation, and, above all, improving the academic programs we deliver to our students.

Caroline Davis
Coordinator of Systematic Program Review

¹ See Attachment I for an overview of SPR outcome categories, and Attachment II for a complete list of programs under review.

² This sub-committee, the graduate outcomes procedures, and the action responses are described in the CGSR document, Systematic Program Review Outcomes Sub-Committee: Terms of Reference and Outline of the Process, October 2001.

Attachments:

I. SPR Assessment Categories (2-page table) [Download here]

II. Six-year Review Schedule (updated May 2002) [Download here]

III. SPR Review Teams 2000/2001 [Download here]

IV. Outcomes of 2000/2001 reviews [Download here]

V. SPR Review Teams 2001/2002 [Download here]

VI. Table of SPR administrative structure [Download here]

SPR Home

I.	SPR Assessment Categories (2-page table)	[Download here]
II.	Six-year Review Schedule (updated May 2002)	[Download here]
III.	SPR Review Teams 2000/2001	[Download here]
IV.	Outcomes of 2000/2001 reviews	[Download here]
V.	SPR Review Teams 2001/2002	[Download here]
VI.	Table of SPR administrative structure	[Download here]