SPR Home

Systematic Program Review Annual Report May 2003

---

1. Introduction

   Systematic Program Review at the University of Saskatchewan is approaching the end of its fourth year of what was originally to be a six-year cycle but, because of reviews delays for several programs, is now expected to take seven years. By the end of this academic year, 97 undergraduate and graduate programs will have been reviewed, along with self-studies, by review teams of highly qualified external reviewers and internal reviewers. Recommendations for program improvement, along with a grade assessment of A, B, C or D, are made by each review team in a report submitted within a month of the site visit.

   When reports are submitted, they are first checked by the SPR Executive to ensure that the reports fulfill the requirements for the review, including grades for designated programs. Then, in turn, the department responds to the report and its recommendations, followed by a response by the college. The SPR Executive reads the responses and makes a grade recommendation to the Planning Committee. University Council is the final voice on SPR program reports and grades.

   Beginning July 1, 2003, a new Academic Director of SPR was appointed for a term of five years. Trevor Gambell, Professor of Curriculum Studies, was appointed Associate Dean, College of Graduate Studies and Research, and Academic Director, Systematic Program Review, in a 0.5 FTE position.

2. Follow-up on Reviews Undertaken in 2001/2002

   The SPR Office receives review team reports, responses from units and from the College.

   The SPR Office summarizes reports and responses and sends all documentation and the summary package to the Planning Committee. This has been done for the following programs:

PROGRAM	RANKING
Chemistry
BSc	C
GRAD MSc, PhD	B
Computer Science
BSc	A
BComm (Development of Information Systems)	C
GRAD MSc PhD	B
Engineering:
Agricultural and Bioresource
BE	B
GRAD PGD, MEng, MSc, PhD	B
Chemical
BE	B
GRAD MEng, MSc, PhD	B
Civil and Geological
BE (Civil)	B
BE (Geological)	B
GRAD MEng, MSc, PhD	B
Electrical
BE	A
GRAD MEng, MSc, PhD	A
Environmental
GRAD PGD, MEng, MSc, PhD	C
Mechanical
BE	A
GRAD PGD, MEng, MSc, PhD	B

 

3. Update on Reviews Undertaken in 2002/2003

   The following programs were reviewed in 2002-2003.

   Term One:

   Commerce:	Accounting (B.Comm., M.Sc. M.PAcc)
   	General Business (B.Comm.)
   	Human Resource Management (B.Comm.)
   	Marketing (B.Comm., M.Sc.)
   (Note that the MBA program was reviewed in 1999/2000.)
   Education:	Undergraduate (B.Ed.)
   Geological Sciences:	Geological Sciences (B.Sc., Graduate*)
   	Geophysics (B.Sc)
   	Environmental Earth Sciences (B.Sc.)
   	Paleobiology (B.Sc)
   *Note: Graduate covers Master's and Ph.D. programs

   Term Two:

   Commerce:	Finance (B.Comm.)
   	Business Economics (B.Comm.)
   	Production/Operations Management (B.Comm.)
   Social Sciences:	Archaeology (B.A.)
   	Archaeology (B.Sc.)
   	Archaeology (Graduate)
   	Classical and Near Eastern Archaeology (B.A.)
   	International Studies (B.A.)
   	Economics (B.A., Graduate )
   	Business Economics (B.A.) (*Note: Bus Econ (B.Comm.) was reviewed with Commerce Programs)
   	Geography (B.A., B.Sc., Graduate)
   	Political Studies (B.A., Graduate)
   	Public Administration (B.A.)
   	Sociology (B.A., Graduate)
   Note: Social Sciences programs in Native Studies and Psychology will be reviewed in the first term of 2003/04, while Anthropology will be included in the Religious Studies review in 2003/04, in line with departmental restructuring.
   Music:	B.A., B.Mus., B.Ed/B.Mus.(Music Ed)

   This year was a demanding one for the SPR office because of the very busy scheduling in both terms. This situation occurs when programs are delayed and reviews must then be squeezed in to already-committed schedules in later years. We hope that next year, with the catch-up in delayed reviews for Native Studies, Psychology, and Mathematics and Statistics, that we will be back on schedule and less frenetic.

   See Appendix B for a list of review teams for the above programs.

4. Planning for 2003-2004 Reviews

   This year sees programs in the Humanities and Fine Arts as well as Kinesiology being reviewed. This will include the following:

   Fine Arts:	Art and Art History (Term 1)
   Drama (Term 2)
   Humanities:	English (Term 1)
   	Languages and Linguistics (Term 2)
   	Religious Studies and Anthropology (Term 1)
   	History (Term TBA)
   	Philosophy (Term 2)
   Kinesiology (Term 2)
   Delayed Reviews:	Psychology (Term 1)
   	Native Studies (Term 1)
   	Mathematics and Statistics (Term 2)

   The arrays of some of these programs requires some especially careful planning, with challenges in selection of external reviewers who can comment on several areas within a program. This is especially the case in Languages and Linguistics with French, Spanish, German, Slavic, Russian, and Linguistics areas. In the case of Philosophy, we will need an external examiner who can focus on the STM Philosophy program, an unique situation among shared program units.

   The review of the Women's and Gender Studies program is delayed to 2004/05 because the current head is on leave and an acting head in place. This makes it very difficult, if not impossible, to mount a self-study document in time, and to identify nominees for reviewers.

5. Policy and Procedural Changes in 2003-2004

   Several changes took place during the year to address either redundancies in procedures, or to overcome difficulties in the SPR process. At times these changes necessitated coordination with other units on campus to expedite procedures. However, the SPR office resolved several important issues that, we believe, lead to a smoother and more efficient operation of the process.

   4.1  Identification of Internal Reviewers: Associate Professors and Recent Professor Emeriti

   With SPR now in its fourth year, units are finding it increasing difficult to identify possible internal reviewers. In one instance the Academic Director had to approach eight internal nominees before an appointment could be made. This problem occurs because so many units are undergoing review, and thus prominent faculty have no time to commit to another review in the same term; the pool of potential reviewers is being depleted (we do not ask reviewers to act more than once in a single review cycle); sabbatical and administrative leaves restrict the pool; and experienced faculty retire and are replaced by junior faculty. Thus, SPR took to the Planning Committee a request to extend the internal nominee pool to include Associate Professors with experience and expertise, and recent Professors Emeriti, recent being defined as no longer than five years retired. This request was approved and is now in effect, though to date we have not had to apply it.

   4.2  Templates for Review Team Reports: Undergraduate and Graduate Programs

   The SPR office is commonly asked by review team for exemplary reports they can use as models. This is a difficult request to meet, because different programs and disciplines tend to use unique structures and discourse styles. To address this need, the SPR office developed a report template based on the review team guides sent reviewers before they arrive for the site visit. Additional features are clear recommendation fields, and a comments and reflection section on the site visit and review process itself. We piloted this template with the first review teams in the Spring of 2003 with great success. This feature is now part of the review process. An electronic copy of the template is given to the review team on the first day on site, along with an explanatory memo. As a result, we have found consistency in reports, and clear identification of recommendations we can track, and to which departments and colleges can readily respond. (See Appendix C.)

   4.3  Incremental Grades for Programs

   We have found a tendency of some review teams to use incremental modifiers when assigning grades. For example, + and - signs are used, as in A- or C+, or adjectival modifiers are employed, as in "a strong B" or "a weak A". At the initial meeting with the review team on site, the Academic Director now makes clear the need to be unambiguous with grades, using A, B, C or D only.

   4.4  Frequently Asked Questions for Review Teams

   In June of 2002, the SPR office, under the then Academic Director, Dr. Caroline Davis, develop a Frequently-Asked-Questions document for review teams, to address many of the questions that external reviewers had, and which were taking up considerable office time in addressing, often in repetitive manner. This document, now sent to review teams ahead of the site visit, has been useful for review teams. This document is available on the SPR website.

   4.4  Statistics Reports for Self-study Documents

   Units were reporting difficulties obtaining statistical reports of program data from the office of Institutional Analysis (formerly University Studies Group), which are needed for interpretation and incorporation into self-study documents being worked on at least six months ahead of scheduled site visits. Institutional Analysis was up to two years behind in generating reports, because of restructuring and loss of personnel. Thanks to cooperation between the IA office and SPR, especially the efforts of Rob Schultz, the new director of IA, the IA office has made a priority to have available reports of units undergoing SPR in the following year. Also, Rob Schultz will meet with the units about to enter SPR at the organizational meeting held by the SPR office. This will expedite access to data required, and address any issues of types of data required and their intended and most effective use.

6. Feedback on SPR

   Finally, we are pleased to provide some comments on the SPR process and the operation of the SPR office and personnel.

   Positive Comments from Review Teams: As a matter of routine, we ask reviewers to comment on the process and procedures, including arrangements and information provided reviewers before the site visit and on site. These comments enable us to make changes and add features, as we did with the report template.

   The following comments have come from review team reports and from within the University community during the current academic year.

   "The externals were VERY impressed by how well-organized the SPR office was. They have never seen anything quite like it at another institution. The offer to send them follow-up on what will happen as a result of their visit particularly impressed them. Everyone was very helpful, prompt in answering requests for information, and forthright in offering opinions."
   ~ Brett Fairbairn, Internal Review team Member for International Studies Review

   "I thought that the process was handled at a very high, and professional level. There is a great deal of uncertainty for department heads and faculty members in entering the SPR process - there is no question that this process is stressful. There is no way around this. This being stated, I felt that the SPR office did everything possible to support the process. The information provided was timely and of a high level. Communications with the SPR office allowed us to greatly improve our self study document, and to structure the site visit to ensure that the externals gained a balanced impression.

   I believe that SPR was both affirming and positive for our department and accomplished the following. First I believe that our department gained a better sense of itself through the SPR process. In an academic environment, it is rare that we reflect collectively on what we are, and what we aspire to be. From our department's perspective, this was perhaps the most valuable outcome of the process. Second, the process allowed us to objectively evaluate where our programs stand relative to other programs across Canada - this allowed us to identify and understand both our strengths and weaknesses. Third, the process of putting together the Self Study Document was, in itself, a worthwhile exercise. It allowed for personal examination - each faculty member better understands their performance relative to others across various dimensions. It allows us to identify trends which reflected both excellence and weakness."
   ~ Glenn Feltham, Department Head, Accounting, Letter to Ken Coates and Tom Wishart, Feb 2003

   "The Systematic Program Review of Earth Sciences was carried out by three highly acclaimed external reviewers with extensive experience in earth science program development and delivery. Their report is thorough, and provides a long list of recommendations that will guide us in program renewal."
   ~ Jim Basinger, Head's Response, Feb 28, 2003

   Areas for Improvement: The SPR Office tracks comments from the general organizational meetings (held with each program head about one year prior to the review) as well as general comments from the Overview Committee. A document prepared by the Planning Committee was referred to the Overview Committee this spring highlighting a number of changes to procedural aspects of the process, which the SPR Office, in collaboration with Council, will address in the next round of reviews.

   Below is a summary of procedural suggestions from various review teams and from the Overview and Planning Committees:

   • change the amount of time for site visits (a number of review teams indicated that the site visit was too short)
   • during the site visit: build in more for review teams to meet together just as a review team at the start and end of the site visit
   • allow for the review team to meet with faculty members in smaller groups or individually
   • the role of and the instructions to the Internal Reviewer need to be clarified and spelled out more explicitly to ensure that the External Reviewers understand what we are asking them to do, and what we are asking them not to do.
   • reviewers need to understand that only letter grades without pluses or minuses are to be assigned to programs.
   • the matter of resources and support for departments and colleges, Institutional Analysis, and the College of Graduate Studies in conducting SPRs should be reviewed. The role and importance of input from students (present, alumni) needs to be addressed.
   • the length of time involved in getting the SPR reports from Reviewers to the Planning Committee should be reviewed to see if the process can be shortened.
   • there is no provision in the SPR for the amalgamation of programs as a possible outcome. This seems to fall through the cracks with the program level focus of SPR.*
   • College-wide initiatives that address needs of several programs/departments are typically not addressed in SPRs. This seems to fall through the cracks with the program level focus of SPR.*
   • the matter of comparisons of SPRs (reports and grades) across departments and within colleges needs to be addressed as part of preventing invidious comparisons and fostering departmental and college morale.
   • the summaries of SPR reports, of the responses of Department Heads, Deans, and the University provided to members of the Planning Committee were well done and proved helpful to the members of the Planning Committee in carrying out their presentations and discussions. These should be continued. Those responsible for preparing such summaries should be thanked for their efforts.

   *  The Planning Committee recognizes that these items can be dealt with appropriately at the Departmental/College level, and hence simply wishes to note their having been mentioned in Planning Committee discussions.

---

Respectfully submitted,

Trevor Gambell, Academic Director, Systematic Program Review
Ken Coates, Acting VP and Provost
Tom Wishart, Dean of Graduate Studies and Research

May 2003

---

Attachments:

I.	SPR Review Teams 2001-2002	[Download here]
II.	SPR Review Teams 2002-2003	[Download here]
III.	Template for SPR Review Teams to Prepare Undergraduate and Graduate Program Review Reports	[Download here]

SPR Home