SPR Home

Systematic Program Review Report to Overview Committee November 2000

---

An Interim Report was delivered to the Systematic Program Review Overview Committee in January 2000, which focused on SPR startup activities. The current report highlights the reviews undertaken in 1999/2000, notes some benefits of the process so far, and comments on the development of SPR through its first year of implementation.

1. Background and Overview of the Process

   In February 1999, University Council approved a new process for program review at the University of Saskatchewan: Systematic Program Review (SPR). SPR is intended to encourage program improvement and innovation, demonstrate accountability for programs, and provide information for decisions concerning program revisions, deletions, and resource allocation.

   As established in the approved Policy and Procedures, each review consists of a self-study; a site visit, report and recommendations by the review team; department/college response; recommendation by the Vice-President (Academic) and Dean of Graduate Studies and Research; final assessment by the Planning Committee; and outcomes such as an 'action plan' to implement recommendations. All actions resulting from reviews (e.g. major revisions, program termination) must go through the normal approval procedures of Council and its committees.

   Review teams consist of at least one internal reviewer from a cognate discipline at the U of S and at least two external reviewers from other universities in North America. Programs are evaluated on the basis of the standard set of criteria approved in A Framework for the Evaluation of Academic Programmes, 1996, including quality (as indicated by curriculum, faculty, learning environment, infrastructure and outcomes), demand, and efficiency in use of resources. For some programs, uniqueness and relevance to the province are also important factors.

   As a University-level initiative, SPR is the responsibility of the Vice-President (Academic); this responsibility is shared by the Dean of Graduate Studies and Research when graduate programs are reviewed. A full-time Coordinator of Systematic Program Review reports to the Vice-President (Academic) to implement the new policies and procedures and coordinate all reviews. The process is overseen by the SPR Executive Committee, which holds key responsibilities such as development of the review schedule, selection of reviewers, and resolution of policy issues. The SPR Policy and Procedures also provide for an SPR Overview Committee (a subcommittee of the Planning Committee) to maintain liaison with the major committees of Council, advise on the process, and receive progress reports. The attached organizational chart shows the membership of each of these committees, as of January 2000 (Att. I).

2. Startup Activities

   Upon approval of the SPR process, an Implementation Committee was struck to guide its implementation and day-to-day management (see Att. I). This committee consists of the SPR Coordinator and representation from the SPR Executive, University Studies Group, and (from Nov. 1999) student unions. The Implementation Committee and its working groups dealt with numerous issues related to roles and responsibilities, procedures and timelines, information requirements, and site visit logistics, to name just a few.

   Besides the development of procedures for the reviews themselves, the startup phase required several major activities:

   • SPR was officially 'launched' in May 1999, with media coverage and distribution of informational binders to the University community and government.

   • The SPR Executive developed an initial six-year review schedule in consultation with Council committees and Deans' Council.

   • The College of Graduate Studies and Research developed standardised statistical reports on each program under review and created a new interactive web-based database, the Graduate Student Information System (GSIS).

   • When it was learned that all student outcomes surveys require ethics approval, the SPR Coordinator developed templates for SPR student outcomes surveys and guidelines for their use, which were granted ethics approval in November 1999. These templates assist units by providing a pre-approved survey mechanism and pre-tested questions appropriate to SPR.

   • A website was developed to provide SPR policies and procedures, the most recent schedule, contact information, background documents, and the student outcomes survey template. Eventually, review documents and an interactive database will also be available on-line. The site can be accessed at http://www.usask.ca/vpacademic/spr.

   • Numerous templates and guidelines were developed over the course of the first year to clarify procedures, answer 'frequently asked questions', and simplify the process of review coordination in future years.

3. Reviews 1999/2000

   For the purposes of SPR, a 'program' is a generally defined set of courses and other requirements as described in the University Calendar, which a student must successfully complete to obtain a specific degree in a Council-approved area of specialization.

   Table 1 lists the programs which underwent review in the first year. In total, 29 program areas were reviewed in the first year, including 15 undergraduate and 14 graduate areas of specialization. To accommodate all of these reviews, ten site visits were scheduled in term two, and 34 reviewers were selected.

   Table 1: Programs Reviewed in 1999/2000

   AGRICULTURAL-RELATED PROGRAMS

   Agricultural Economics BSA Agricultural Economics GRAD Agricultural Economics

   Agronomy BSA (Agronomy): combination of Ag.Econ., Soil Science and Plant Science teams and site visits

   Animal Science BSA Animal Science GRAD Animal Science

   Applied Microbiology/Food Science BSA Applied Microbiology BSA Food Science BSc Food Science GRAD Applied Microbiology GRAD Food Science

   Plant Science Group BSA Agricultural Biology BSA Crop Science BSA Horticulture Science BSA Plant Ecology/Rangeland Resources GRAD Crop Science GRAD Horticulture Science GRAD Plant Ecology

   Soil/Environmental Group BSA Agricultural Chemistry BSA Environmental Science BSA Soil Science GRAD Soil Science

   COMMERCE

   MBA Masters of Business Administration

   EDUCATION-RELATED PROGRAMS

   Educational Psychology/Special Education GRAD Educational Psychology GRAD Education of Exceptional Children

   PHARMACY AND NUTRITION PROGRAMS

   Pharmacy BSP Pharmacy GRAD Pharmacy

   Nutrition and Dietetics BSc (Nutr) Nutrition GRAD Nutrition

   TOXICOLOGY PROGRAM (interdisciplinary)

   GRAD Toxicology

   Thanks to the efforts of the reviewers, the units under review, the SPR committees, SPR Coordinator, and other University staff, all reviews were successfully completed. In one case, this required heroic efforts when a reviewer had to cancel because of major surgery three days before the original site visit! Even in the normal course of events, the smooth operation of these reviews required the completion of many complex and time-critical tasks, as well as the startup activities described in Section II, above. For instance:

   • The units sponsoring the programs under review were responsible for preparing self-study documents. This was universally noted to be a time-consuming and labour-intensive task, but one that yielded significant benefits (see Section VI below).

   • The review schedule was revised to accommodate requests for early reviews.

   • Units were provided with CGSR Graduate Program Reports, reports on Library holdings for each program area, and additional statistics to assist with self-study preparation.

   • Meetings were held with Deans and Department Heads to explain the process and determine the timing and structure of each review. Liaison with Deans, Department Heads and other program staff continued on a close and frequent basis thereafter for each review.

   • Ten site visits were scheduled between January and May 2000 - no small task, given the scheduling constraints of reviewers, key college and program personnel, and the Vice-President (Academic) and Dean of Graduate Studies and Research. A sample site visit itinerary is appended, for Soil Science (Att. II). (This visit was also highlighted in On Campus News, March 24, 2000.)

   • Ten review teams were selected (an eleventh 'virtual team', for Agronomy, consisted of members from the three related review teams). Selection of appropriate reviewers was noted to be one of the most critical factors in ensuring a high-quality review, and careful attention was paid in the first year to qualifications and conflict of interest. The 11 internal and 23 external reviewers were of outstanding calibre, internationally recognized scholars from institutions highly regarded in the discipline under review, who without exception worked well together and provided thorough, informed and useful reports. Most review teams included American as well as Canadian reviewers in order to provide the national and international perspective recommended by SPR. A list of reviewers is attached (Att. III).

   The final outcomes of each review will be determined this Fall. The Vice-President (Academic) and Dean of Graduate Studies will prepare preliminary recommendations, and will discuss outcomes and actions with the responsible Deans. All review materials will be sent to the Planning Committee for final determination of the outcome categories, which will be reported to Council. The assessment categories (A, B, C, D) are described in the SPR Policy and Procedures, Section XIII. If required, an action plan will be developed by the relevant Dean and program head, to be approved by the Vice-President (Academic) and Dean of Graduate Studies. Actions arising from SPR will be implemented through the normal U of S approval processes.

4. Reviews 2000/01

   The following programs are scheduled for review in the second year of SPR:

   • Education: graduate programs in Educational Administration, Educational Foundations, and Curriculum Studies;
   • Law: undergraduate LLB and graduate LLM programs;
   • Nursing: undergraduate NEPS and BSN (post-registration) and graduate MN programs.
   • Arts & Science: two interdisciplinary undergraduate programs, Regional and Urban Development (RUD) and Land Use and Environmental Studies (LUEST).>

   These programs are currently in their self-study phase. Reviewers have been selected for most programs, and site visits have been scheduled for the Fall reviews.

   The Master's program in Agricultural Extension was originally scheduled for review this term, but after near-completion of the self-study, the program coordinators and relevant Deans agreed that it would make more sense to propose program termination at this stage, rather than proceeding with an expensive external review. They are to be commended for carrying out a true self-study - the benefits are apparent.

   Preparation for the 2001/02 reviews will begin in January 2001 or earlier, starting with the organizational meetings with Deans and Department Heads. The complete SPR schedule is attached, updated as of October 2000 (Att. IV).

5. Resources Required for SPR

   The majority of support for SPR is currently provided in-kind. By re-allocating their own resources to SPR, the University's central administrators demonstrated their commitment to this process at a time of fiscal restraint. Staff were re-assigned from the University Studies Group (USG) and College of Graduate Studies and Research (CGSR) as follows:

   • the Coordinator (1.0 FTE) was seconded from USG and approx. 0.25 FTE USG data analysis effort was re-directed to SPR;
   • CGSR re-assigned approx. 0.33 FTE senior clerical and approx. 0.25 FTE senior ASPA to SPR, re-directed a portion of its former Associate Dean's budget to pay 1.25 FTE staff for SPR support and GSIS data entry, and donated space and facilities for SPR operations.

   A clear need for additional secretarial support led the SPR Executive to authorize payment of 0.5 FTE clerk-steno II out of SPR funds starting in February 2000 (to be increased to 1.0 FTE in 2001 when SPR assumes responsibility for the portion currently funded by CGSR). However, this is the only salary payment from the SPR budget. An analysis of all 'in-kind' costs of SPR is attached (Att. V).

   As noted in the Planning Committee's February 1999 report to Council (Systematic Program Review at the University of Saskatchewan), USG and CGSR may not be able to sustain this level of support without additional resources, and the question of long-term resources for SPR must be addressed.

   To cover the direct costs of reviews (externals' expenses, printing, etc.), $65,000 was allocated to SPR from the Operating Budget in 1998/99 and $60,600 in 1999/00. The first year's review costs were much as estimated at approximately $80,000, including honoraria of $1000 for each external reviewer and an average of $1500 per external in expenses. It is estimated that review costs will range from $70,000 to $140,000 per year, depending on the number of reviews scheduled. An expenditure forecast is appended (Att. VI).

   Costs to the participating units must also be borne in mind. SPR has been able to provide some assistance to units in the form of Graduate Research Fellows (for graduate program reviews only), but the brunt of the self-study workload generally falls to department heads, self-study committees, and/or other faculty and staff designated to prepare the self-study. Early discussions with Deans indicated a willingness to provide some support for SPR from college sources, and this proved very helpful in the first year.

6. Feedback and Observations on the Process

   The vast majority of feedback has been positive. Some examples:

   • Reviewers commented that they were impressed with the process, enjoyed the visit, and found it very worthwhile. Internal reviewers noted that the review gave them a new perspective on their own programs and will be helpful for their own reviews, while externals appreciated learning about programs and review processes at another university, and brought back a favourable impression of the U of S.

   • Units commented that the self-study process provided them with a wealth of useful information about their unit and programs and motivated them to reflect on their program array and objectives. One unit noted that the SPR process was much more a true 'self-study' than their accreditation report was. Programs that were about to propose or were in the midst of major revisions reported that the self-study and the external reviewers' comments helped them to improve their plans for change and implementation. The process also encouraged units to survey students and provided a strong stimulus to make changes which might have been 'in the pipeline' for years with little action.

   • As a result of SPR, Calendar entries are being clarified, certain administrative functions have improved, central databases are more up-to-date, and units are more aware of the importance of submitting and verifying statistics.

   • The SPR student outcomes survey templates have proved useful to several colleges and departments as the basis for surveys of both graduates and current students. Some units have used the templates to design non-SPR surveys as well.

   • Units in Agriculture noted that scheduling the SPR review shortly after the accreditation site visit did have the intended effect of reducing departmental workload.

   Although SPR has proved to be extremely well designed, small improvements are continually being made as the process is implemented. For instance:

   • In response to 'frequently asked questions' and the refinement and clarification of the SPR policies and procedures, units are now provided with more detailed guidelines on matters such as criteria for reviewer selection, preparation of the self-study, and development of the site visit itinerary.

   • Numerous comments on the excessive length of self-study reports in the first year led to guidelines for document length, the suggestion that certain documents should be provided on-site rather than in the self-study binder, and permission to use common appendices within program groups.

   • The SPR process now allots more time for preparation of the self-study document, i.e. 4-6 months rather than the 3 months originally envisaged.

   • Delays in provision of statistics were not as frequent in the second year and should be even less likely once standardized data packages and interactive databases have been developed.

   While it is too early to comment on review outcomes and implementation of recommendations, additional benefits anticipated from SPR include fulfilment of the three objectives mentioned above (program improvement, demonstration of accountability, and provision of information for decisions on programs and resource allocation) and other long-term benefits such as:

   • Improvement of the extent to which programs contribute to the University's broader goals;

   • Establishment of mechanisms for collecting feedback on student satisfaction and outcomes;

   • Improvement of central databases in terms of both accuracy and completeness;

   • identification of programs as areas of potential institutional priority.

   SPR is vital to the University's renewal agenda. As described in A Framework for Planning, SPR will assist the University in putting into practice its core principles of autonomy, quality and accountability. It is a key strategy for achieving the University's highest priority goals, particularly the goals of improving the quality of instructional programs and fostering the teacher-scholar model.

7. Presentation to CIRPA Conference

   Saskatchewan hosted the year 2000 conference of the Canadian Institutional Research and Planning Association, attended by over a hundred representatives of universities, colleges, government and the private sector from across Canada. Since several universities had expressed interest in our new program review process, the decision was made to present a paper on SPR at this conference. Entitled "The Joy of Reviewing: Implementation of a New Program Review Process", the presentation was made by Caroline Davis and Pauline Melis to an appreciative audience. Additional advice has been sent since then to other universities (e.g. Memorial) who are using our process as a model.

8. Plans for the Future

   Besides undertaking the large number of reviews scheduled for the next few years and managing the outcomes of the reviews just completed, SPR administrators will continue to improve and assess the process itself. Long-term tasks include the development of standardized data packages for SPR reviews and an interactive web-based data system, and the revision of policy and procedure documents as necessary. We will continue to communicate the process and activities to the University community and beyond, including a planned presentation to the Department of Post-Secondary Education and Skills Training next year.

   A crucial task, now that we have reached the 'outcomes' stage, will be to establish procedures for follow-up, and to monitor the use of review results and implementation of action plans. A review process cannot be considered successful if its reports simply sit on a shelf; we must ensure that SPR achieves the goals for which it was created.

   This report is submitted with thanks to the faculty, staff and students of the units under review, members of the SPR committees, and all others on campus whose efforts are contributing to the improvement of the University's degree programs through SPR. A special, heartfelt note of thanks is due to the late Sylvia Wallace, champion and inspiration of Systematic Program Review, who has been much missed this past year. We hope that we have done justice to her vision in our implementation of this new program review process.

Caroline Davis
Coordinator of Systematic Program Review
February 11, 2000

Attachments:

I.	SPR Organizational Chart	[Not Available for Download] [Available from SPR Office]
II.	Example of a site visit itinerary (Soil Science)	[Not Available for Download] [Available from SPR Office]
III.	SPR Review Teams 1999/2000	[Download here]
IV.	Six-year Review Schedule	[Download here]
V.	"In-kind" Costs of SPR	[Not Available for Download]
VI.	Expenditure Forecast	[Not Available for Download]

SPR Home