SPR Home

A Background Paper and Proposal for A Systematic Programme Review Process at the University of Saskatchewan

---

Prepared by the Academic Planning and Priorities Committee, April 29, 1996

 

A. Introduction: Universities under Scrutiny

In 1985, the Carnegie Foundation for the Advancement of Teaching issued a statement expressing disquietude about the de-valuation of higher education at a time of increased competition for scarce resources. "After decades of strong bipartisan support, the current debate about budget priorities has been focused almost exclusively on numbers and on the negative aspect of higher education. The larger perspective has been lost. We hear how much education costs, not how much it is worth. We are told that students are exploiters, rather than tomorrow's leaders. And we are reminded of the abuses, not the benefits of higher education"¹. Over the past two decades, universities have experienced increasing and competing demands for their resources while at the same time obtaining decreasing funds to sustain the array of programmes offered. The "crisis" in higher education that has resulted has been described as "...not merely one of public confidence vis-a-vis the performance of higher education; it is also, and perhaps more fundamentally, an internal crisis of purpose, that is, one which touches on the very nature of individual institutions, their roles and functions and their place in the total higher education system"². Today, the public's traditional interest in accountability, i.e. integrity in the use of public resources, has extended "...to a requirement for assurance that universities are actually achieving what they claim to be pursuing in teaching, research, and public service and, furthermore, that they are doing so as efficiently as possible within their unique circumstances"³. While it may be encouraging to note that there is broad agreement that universities can best fulfill their appointed role if they continue to enjoy substantial autonomy in decision making, "there is also increasing agreement -certainly outside universities and increasingly indeed among the more realistic in the groves of academe- that continued enjoyment of the self-regulatory model of university funding and governance must be earned by the external demonstration of accountability. Autonomy is to accountability as rights are to responsibilities"⁴.

The University of Saskatchewan is not immune to national and international trends which are challenging the higher education sector. If it is to remain a vital partner in the provincial and national educational systems, it must take an active part in interpreting its role within the educational system. Through a greater commitment to institutional accountability, universities, including the University of Saskatchewan, should maintain their freedom from political interference and thereby preserve one of the greatest values of the academy -- autonomy. Systematic review of programme offerings is one way for the University to demonstrate its willingness to analyze, to question and to improve its programmes and thereby to respond to the high value placed on publicly-funded institutions within society.

¹ E. Boyer, "Message to the Nation: Do not adjust this vision", Times Higher Education Supplement, 5 only 1985, cited in Universities Under Scrutiny, Organisation for Economic Co-operation and Development, Paris, 1987, 92.

² Ibid., 7.

³ J. Cutt and R. Dobell, eds., Public Purse, Public Purpose: Autonomy and Accountability in the Groves of Academe. The Institute for Research on Public Policy, 1992, ix.

⁴ Ibid., ix.

 

B. Working Group Mandate

In his White Paper on Academic Planning, Vice-President Browne described the formulation of a process for the reviews in the following way:

"External reviews of units (both academic and non-academic) are widely used and are regarded as useful mechanisms for gathering independent unbiased suggestions for unit improvement. The committee must study such processes and recommend on the possibility of establishing such a system for our University. Should a review process be recommended, the fundamental aim must be that of unit improvement so that department heads and unit managers obtain genuinely useful and helpful feedback."⁵

From this framework, the Academic Planning and Priorities Committee (APPC) established a Working Group in April 1993 "to assess and develop a process for the systematic review of instructional programmes offered by the University of Saskatchewan"⁶. The members of the Systematic Programme Review Working Group included Sylvia Wallace (College of Pharmacy) chair, David Atkinson (College of Arts and Science), Patrick Browne (Vice-President Academic), Graham Simpson (College of Agriculture), Ron Sutherland (College of Arts and Science), Barrie Dubray (University Studies Group) and Pauline Melis (Office of the Vice-President Academic).

To date, the Working Group has examined review processes at universities in Canada, compiled information on past and current academic planning activities at the University of Saskatchewan, and conducted a literature review. From their discussions, the Working group developed a Background paper on systematic programme reviews and prepared a preliminary proposal for approval by the Academic Affairs Committee and Council.

⁵ P. J. Browne, White Paper on Academic Planning, January 1992, 5.

⁶ Minutes of the Academic Planning and Priorities Committee, May 3, 1993.

 

C. Review Processes at the U of S

The first activity of the Working Group was to determine the past and current review processes at the University of Saskatchewan. The Working Group met with Professor Blaine Holmlund who provided information on review processes at the University. The Working Group also solicited information from Deans on current planning and review processes within Colleges. The following section summarizes the information obtained.

1. A History of Internal Review Processes:

In 1975, at a seminar meeting at Emma Lake, the Board of Governors discussed the issue of academic programmes at the University including the Board's responsibility and effectiveness in this area. As recorded in the minutes of the Board meeting:

"The discussion made special reference to duplication of courses; size of classes including offering of some senior classes in alternate years; over specialization or too narrow specialization in some disciplines; financial support and competitive position of departments and colleges of other universities; inter-provincial (western) rationalization of programmes; long range academic planning; re-assessment of current programmes and possible retrenchment or reduction; necessity of research for good teaching."⁷

The minutes of the Board meeting also record adoption of a policy on unit reviews.

"It was recommended that the Board should establish a policy for regular assessment of colleges and departments (and other units) prior to the appointment of a new Dean or Head. The objectives should be established for a 5-10 year period and reviewed in relation to the objectives of the University. The assessment should be done by using inside resources such as the Universities Study Group and outside resources such as inviting specialists to the Universities to review the unit and submit recommendations. The Board agreed to this policy of assessment ...."⁸

The general purpose of these reviews was described as follows:

"...to provide the Board of Governors and senior university officials with a better understanding of the operation of the College; of its relationships to other units on campus and other provincial organizations; of the adequacy of resources provided; of the accomplishments of the College; and of any problems or opportunities that exist. In initiating this policy, it was hoped that the report generated would serve as useful background information and as a stimulus for the Dean and the Faculty of the College to undertake a critical self review of their programmes and to lay out plans for the future"⁹.

The primary objective of the review was to provide "the opportunity for the college to undertake an intensive self-review and [to] provide the new Dean or the incumbent with a base of information which will enable him or her to more effectively plan and implement desirable changes over the next five years or so"¹⁰.

Resulting from this recommendation, a series of reviews of "units" were conducted by the University Studies Group under the direction of Blaine Holmlund (Director), Newman Haslam (Vice-President Academic), and Leo Kristjanson (Vice-President Planning). This "review period" covered February 1976 to December 1981, starting with a review of the College of Education and ending with a review of the School of Agriculture. A total of 37 "units" were reviewed -- 7 Colleges, 26 Departments, 2 Schools, the Student Counselling Service, and the Student Health Centre (Appendix A, List of Reviews). The general procedure used included the establishment of a steering committee in the College or unit under review, the preparation of a preliminary report with the assistance of the University Studies Group, an on-site visit and preparation of a report by external reviewers, and finally, distribution of the external reviewers' report to faculty and to the Board. The report was not widely circulated outside the unit in order to maintain an atmosphere of frank and open discussion about deficiencies and shortcomings within the unit (i.e. the outcome of the review was supportive, not punitive). None of the Colleges and/or units with accreditation processes were included in the series of reviews.

Throughout the 1980s, the process utilized by the University of Saskatchewan was emulated by several other Western Canadian universities.

After completion of a cycle of unit-based reviews, the University of Saskatchewan turned its attention to issues facing the institution as a whole by embarking on the Issues and Options project under the direction of Blaine Holmlund (Vice-President Planning). This project examined issues such as accessibility to the University, extension activities, teaching effectiveness, and university governance. The project ended in November 1990 when George Ivany became President.

⁷ Summary of Discussions and Recommendations of the Board of Governors (Seminar Meeting, Emma Lake, June 13-15, 1975):

⁸ Ibid.

⁹ Blaine Holmlund, "Introduction to the Review Process", from the Review of the College of Arts and Science, c1980.

¹⁰ Blaine Holmlund, "Report on College Reviews", a report prepared for the Board of Governors, c1980.

 

2. Current Internal Review Processes:

a. Accreditation Reviews: Accreditation is a common method of conducting reviews of programmes within professional colleges. Accreditation reviews generally measure the programme against standards established by the accreditation body for its members. At the University of Saskatchewan, eight Colleges have accredited programmes (Table I) and most of these programmes have been accredited for the maximum period of time.

Significant features of accreditation processes include a self-study document, an on-site evaluation, a report to an accrediting board, and a final report to the home institution. The accreditation process usually judges whether programmes "meet a minimum standard". Normally, programmes wishing to be accredited apply to an accrediting/professional organization. This organization collects fees, selects external reviewers, and provides other resources for the on-site evaluation/report. The home institution usually absorbs the cost of the self-study component of the process, a cost which may not be insignificant in terms of staff and other resources.

Programmes may be approved unconditionally for a maximum or minimum number of years, or conditionally for a specified period of years to allow the programme time to meet the requirements for accreditation. Only rarely are programmes denied accreditation. The consequences of losing accreditation status may not be specified by the accrediting organization. In fact, levying consequences of non-accreditation may not be within the purview of the accrediting body, especially if the accrediting body is at arm's length or is a separate organization from the professional licensing authority. In addition, students graduating from non-accredited programmes may have alternative routes to qualify for practice (e.g.. writing professional examinations).

As an institution, the University has pointed to the number and level of accredited programmes as a way to demonstrate the high quality of its curricula. However, accreditation processes have their critics. In its recent report, the Ontario Task Force on University Accountability noted the following:

"Considerable reliance continues to be placed on this form of accountability. But the results are sometimes seriously questioned. When accreditation is undertaken by an agency representing a consortium of institutions, criticism is sometimes registered on one or other of two opposing grounds. It is argued that the minimum standards are set too low in order to produce a positive report and therefore the result is meaningless. Conversely, in other cases it is argued that the standards are set artificially high to protect the interests of the existing members of the consortium with a vested interest in restricting the number of degree-granting institutions accredited in the region. This same criticism is sometimes levelled against professional associations which are considered to have a vested interest in the status quo and to be unsympathetic to institutions offering new approaches to the academic discipline"¹¹.  

Table I: ACCREDITATION within Colleges and Departments

at the University of Saskatchewan

 

College	Accreditation of Program	Last Date of Accreditation	Length of Term	Departments Included in Accreditation	Next Date for Review	Report Availability	Comments
Agriculture	No
Arts & Science	No						Two programs within the College are accredited (Psychology to 1998and RUD to 1994).
Commerce	No						300 American Schools are accredited; Alberta, UBC & Calgary were accredited, but UBCchose not to re-new
Dentistry	Yes	1990	7 year	All depts included	1997	Through College
Education	No						Graduates are licensed to teach in Saskatchewan.
Engineering	Yes	1989	5 year	All depts included	1994	Through College
Extension	No						Some certificate programs may be accredited.
Graduate Studies & Research	No
Law	No						Graduates must pass provincial bar examination and meet the national standards although no accreditation process exists
Medicine (Undergraduate)	Yes	1988	7 year	All depts included	1995	Through College	1988 recommendations are being followed
Medicine (Post-Graduate)	Yes	1991	6 year		1997	Through College	Several reports available
Nursing	No					Through College	College basic four-year program has "approved status" for a 5 year term (i.e. similar to accreditation)
Pharmacy	Yes		7 year		1994		First accreditation to be conducted in 1994
Pharmacy (Nutrition)	Yes	1993	5-7 years		1998	Available through Division	All four phases were accredited but with different terms (A/B 7 yr; C/D 2 yr)
Physical Education	No
Veterinary Medicine	Yes	1991	7 year		1998	Yes

 

One of the major benefits of accreditation appears to be that the process is a catalyst for change within the programme under review even if that change is a method by which to ensure uniformity of curricula across the country. Another benefit is that the accreditation process provides contact between the University and the profession -- the external community which is the major "market" for graduates.

The major disadvantages of the accreditation process are reflected in the notion that the accredited programme "meets a minimum standard" -- an event which tells the university little about the quality of the programme offered and its relationship to other similar programmes across the country. As well, no credit appears to be given if an institution exceeds the minimum standards specified. Secondly, the institution appears to have little control over the accreditation review itself and little opportunity to influence the direction of the accreditation review. Further, accreditation reviews usually are directed at the undergraduate level and exclude graduate programmes.

Before conclusions can be drawn about whether accreditation reviews provide a sufficient information base for the purposes outlined in this paper, the various accreditation processes must be examined more closely. One could start by asking whether the data and analysis required by a university cyclical review process would meet the needs of the accreditation self study, or conversely, whether the data and analysis provided to the accreditation body would meet the needs of the University study and what if anything should be added to provide the type of information consistent with a university review process. In the future, a closer examination of the accreditation processes by universities and accrediting agencies may be required if the needs of both groups are to be realized and duplication of effort avoided.

¹¹ Report of the Task Force on University Accountability, University Accountability: A Strengthened Framework. Ontario, 1993, 21.  

 

b. College of Arts and Science: Of the Colleges without accredited programmes, only the College of Arts and Science has established an internal policy for academic reviews. Implemented since 1992-93, the major features of this policy include the development of a self-study document by the department under review, the solicitation of internal and external opinions on the department and its programme offerings, an on-site external review, a departmental response to the review report, and, a progress report on the outcome of the review process. A schedule of reviews has been prepared and all departments will be reviewed over a five year period (Appendix B, Schedule of Reviews).

Although Type C Arts and Science programmes include majors in the Basic Health Sciences (Anatomy, Biochemistry, Microbiology, Physiology) and Computational Science, these Departments are not included in the reviews. Also, since the reviews are of academic units, interdisciplinary programmes are not specifically mentioned in the College policy document, but presumably would be reviewed with the Department in which the programme is based (if one can be identified).

c. Other Internal Review Processes: While no specific internal review processes exist in other Colleges, programmes, such as those in Agriculture, are reviewed on an irregular basis when the faculty determine that the curriculum should be updated. Also, programmes offered in professional colleges, such as Education, Law, and Commerce, while not accredited, must be of sufficient quality to ensure that students graduating from these programmes will be adequately prepared for provincial or national examinations administered by the professions.

Current review processes already consume a significant amount of energy, resources and time. Moreover, a "cycle of reviews" exits even if it is not officially recognized by the University. Given the existing, albeit unco-ordinated, review cycle already underway at the University, the relationship of the current accreditation/departmental reviews to whatever new process is devised must be considered. Ideally, the review process must have a framework that is compatible with all of the types of review processes underway. One other important aspect to be determined is the involvement of the collegium and the central administration in the review process itself and in the monitoring of these procedures.

 

 D. Feasibility of Cyclical Reviews

Based on the White Paper, the major activity of the Working Group was to examine the feasibility of establishing a systematic review process at the University of Saskatchewan. The Working Group agreed that five key elements should be considered before recommending whether university-wide systematic programme reviews should be implemented. These five key elements are: 1) the planning environment in general and the planning practices at other Canadian universities; 2) an analysis of the current review processes; 3) the purpose of the review process and the types of review processes that might be implemented; 4) the implications of a university-wide review process (including cost, the overlap with existing review processes, the organization of planning, and the degree of integration with the budgetary processes); and, 5) the relationship of systematic programme reviews to other university-wide planning activities. A discussion of each of these elements should provide guidance as to whether a systematic review of programmes should now be undertaken by the University of Saskatchewan.

1) The Planning Environment:

In considering the planning environment, two primary issues were identified --accountability and the review processes implemented at other Canadian universities. Both of these issues relate to the need for the University of Saskatchewan to implement a review of its curricula if it wishes to demonstrate its accountability to the Province and does not wish to be excluded from the practices of the mainstream universities.

a. Accountability: As alluded to earlier in this report, there is a growing national trend to ensure greater accountability within universities and greater responsiveness to the needs of the people that universities are intended to serve. The recent report of the Ontario Task Force on University Accountability (May 1993) is a case in point. Established in response to the concern over the accountability of Ontario universities for the large amounts of public funding they receive, the Task Force argued that universities, if they are to maintain their important role in the economic, social, and cultural development of the country (the ultimate reasons that they are funded from the public purse), must provide evidence of sound management and expenditure of these monies. It argued that "[a]ccountability is not only about the management of physical assets and human resources but also about how social policy is fulfilled....Increased competition for limited public funds has led to the recipient of these funds being asked to be more accountable for them. Universities have not been exempt. They too are being asked to demonstrate how they are meeting the basic policy objectives of government, and to show that they are managing their revenue in an economic and efficient manner"¹².

At a September 1993 meeting of the Council of Ministers of Education in Canada (CMEC), a consensus was reached about the importance of addressing national education issues in a concerted way. The Ministers agreed to investigate how provinces and territories can share expertise in a period of scarce resources to enhance the quality and accountability of the education systems across Canada. They further agreed to establish an action plan that builds on the existing features of Canada's education systems. Priority areas for development include 1) informing the public on the quality of learning in Canada; 2) commencing immediate discussions with Statistics Canada on a programme to develop better measurements of the performance of the education systems, such as completion rates, follow-up of graduates and public satisfaction; and 3) establishing a national work group to examine curriculum comparability and joint initiatives in curriculum development¹³. At the promulgation of this agreement, Pat Atkinson, Saskatchewan Minister of Education Training and Development, stated that "the declaration opens the door to more co-operation in a number of areas including: distance education, credit transferability, assessment of prior learning, inter-provincial collaboration on curricula, and information on accountability in education"¹⁴.

At the provincial level, the issue of the two universities and their accountability was raised in the final report of the University Programme Review Panel (UPRP) chaired by Dr. A.W. Johnson (March 1993). This report commented on the need for greater accountability within the university sector and suggested several ways to achieve this, including an integrated way to inform the public about how the individual universities and the whole university system are functioning, the publication of data on trends, the creation of an Advisory Council on University Education, and public forums¹⁵.

The UPRP report also suggested that the University move to develop a systematic review process and work with other universities to explore ways by which programme reviews can become more consistent. The report recommended that the university follow-up in two areas, namely:

"the two universities [should] implement regular and systematic evaluations of the range and the quality of their programmes. The two universities should develop parallel systems to achieve this, along the lines of the University of Regina unit evaluation model, and...completed team evaluations [should] be published.¹⁶" and

"the two universities [should] establish an inter-university working group on effectiveness evaluation and approach the Association of Universities and Colleges of Canada (AUCC) and other universities to explore developing generalized, university-wide accreditation or programme reviews.¹⁷"

While the increasing interest in accountability of public institutions raises concern about the ultimate use of the information provided to Governments and other stakeholders, the following excerpt illustrates that universitie' greater participation in programme review can have a positive connotation:

"Simply ignoring the increasing financial pressures and growing concern for accountability would exacerbate the situation and place the survival of universities in Canada as relatively autonomous institutions in serious jeopardy....An active co-operative response to external controls by universities...will enable universities to shape the information agenda to their advantage - at the very least, having it reflect reasonably accurately their circumstances and performance - and to control its evolution. Universities will therefore enjoy a more stable environment. Further, active co-operation would be likely to reduce the severity and frequency of both prospective conditions on funding and retrospective reporting and auditing requirements, and might even - by demonstrating that what is spent is well spent - contribute to more generous funding levels for universities."¹⁸

In addition to the potential for the attainment of a positive result from the review process for the university as a whole; i.e., a higher degree of funding for the programmes offered, regular review of programmes provides one strategy to ensure that the University of Saskatchewan achieves the first goal in its mission statement: "to offer a rich array of challenging academic programmes"¹⁹. If the University adopted a systematic programme review process, it could provide timely and consistent information to the public about the type and quality of its programmes and help to ensure its programmes are consistent with its mission.

¹² Ibid., 11-12.

¹³ Communiqué of the Council of Ministers of Education, Canada, Victoria, September 28, 1993.

¹⁴ Government of Saskatchewan News Release, September 29, 1993.

¹⁵ Looking at Saskatchewan Universities: Programs, Governance, and Goals. Report of the University Program Review Panel, March 1993, 57-58.

¹⁶ Ibid., page 53.

¹⁷ Recommendations of the UPRP, March 1993, page 53.

¹⁸ James Cutt, Universities and Government: A Framework for Accountability, The Institute for Research on Public Policy, 1990, page vii.

¹⁹ University of Saskatchewan Mission Statement, approved May 1993, 3.

 

b. Review Processes at Other Canadian Universities: Information on review processes was obtained from several universities in Canada: UBC, University of Victoria, University of Alberta, University of Regina, University of Manitoba, and Dalhousie University (Table II). The summary of review procedures at other Canadian universities by Holdaway et al. [CSSHE Professional File, Fall 1991] also provided useful information and analysis of the process.

From the information available, several general observations can be made. Review processes at other universities tend to follow the unit model, although a programme approach to reviews usually occurs for reviews of graduate programmes. Purposes of the review processes also vary from one university to another although it is fair to say that reviews, when conducted, are generally done to provide helpful information to faculty and administrators on ways to improve the curriculum offerings of the institution. Some universities, such as UBC have defined a process for departmental reviews but have not conducted them in cyclical manner. Documents from several sources commented on the importance of "follow-up" to the reviews, but highlighted a concern that follow-up was often lacking. In cases where there is a "Senate" [equivalent to the University of Saskatchewan Council] committee involved in the review process, there is a requirement to report to Senate on the progress of reviews. Although accreditation reviews are acknowledged as having a somewhat different focus than other types of unit reviews, most guidelines indicate some attempt is made to co-ordinate accreditation and unit reviews. Some provinces have established mechanisms to conduct reviews of university programmes on a provincial scale; e.g., the Ontario Council on Graduate Studies and the recently formed Commission for the Audit of Programme Review Processes in Quebec.

 

 

TABLE II: PROGRAMME/UNIT REVIEWS AT OTHER CANADIAN UNIVERSITIES

 

University	Focus/Scope	Co-ordination (Authority)	Policy (Date)	Cycle (Status)	External Review
UBC	departmental reviews (UG/GS programmes) interdisciplinary graduate programmes	Faculty/Dean   Faculty of GS	Senate Policy 1977	irregular on request
UVic	academic units (UG/GS programmes)	VP Academic	Procedures/guidelines Senate Planning Committee 1985	7-10 y (4-6 reviews/y)	as appropriate
U of A	academic, admin and support units	PACCR follow-up by VPs	UPC recommendation 1978 PACCR appointed 1980 inactive since 1990	one cycle completed (126 units over 10 y)	yes
Regina	academic, admin and support units	VP Academic (functioning as PACCR)	Draft Policy 1990	volunteer units -- in "pilot" phase	yes
Manitoba	departments, faculties, schools (UG/GS programmes)	Academic Review Committee of Senate	Senate proposal 1982 ARCS Terms of Reference 1985	7 y one cycle completed	yes
Dalhousie	units Faculties   GS programmes	Faculties Senate Academic Planning Committee Faculty of GS	Guidelines, Senate 1985 Terms of Reference, Senate 1992 Terms of Reference, Graduate Studies 1986	5-10 y 5-7 y   8 y new within 4 y	yes yes     yes for PhD

 

 

We note that the first two initiatives recommended by the Principal's Advisory Task Force on Resource Issues at Queen's University were:

"Along with a regular report on university-wide indicators of performance in relation to stated goals, a more comprehensive, standardized, and rigorous system of evaluation of departments, schools and faculties should be developed in consultation with them."²⁰

"The system of evaluations must be closely integrated into the decision-making processes of the university."²¹

While other universities were helpful in providing information on the general purposes of the review processes, three areas of special interest were not included in their responses. First, very little information on the budget allocated to support the review process was available. We do know that the University of Victoria allocates $50,000 for 4-6 reviews/y and the University of Alberta allocated $225,000/y for PACCR (for a ten year period). Secondly, the impact of the review processes on resource allocation to programmes that were reviewed was not provided. Thirdly, there was little information on whether the review process can be supported in the long term and how it has been continued past the first cycle of reviews. Only the Universities of Alberta and Manitoba commented on these aspects and both with different results: the University of Alberta abandoned the review process after one cycle while the University of Manitoba was utilizing the information collected to develop a process for the review of graduate programmes. A more complete analysis of the processes at specific universities is contained in Appendix C.

²⁰ Meeting the Challenges: A Discussion Paper Based on the Work of the Principal's Advisory Task Force on Resource Issues, September 1992, page, 9.

²¹ Ibid., page 10.

 

2) Analysis of the Current Review Processes:

After examining the current review processes used by the University or provided by an external agency for the University programmes, several deficiencies of the current processes were apparent. One of the most important deficiencies is that not all programmes are being reviewed. For example, one very visible gap is the lack of a review process for graduate programmes. The other gap relates to programmes which have no established internal or external review process, and may only be reviewed when the faculty/department/college wish to change the curriculum. Maintaining the status quo for programme/unit reviews may place such programmes without formal accreditation or review processes in an unfavorable "bargaining" position. They are left without externally-validated arguments for programme changes and resource requirements and thus are vulnerable to elimination or re-organization based on budgetary expediency.

Secondly, the origin and purpose of current review processes are different and therefore different information and analysis are required. At present, the information collected and the criteria applied are variable and do not lend themselves to comparison between programmes. For accredited programmes, information and analysis are directed to an external body; for the College of Arts and Science, information and analysis are internal to College. There is a burden on the University Studies Group because each information request is unique and may require significant administrative adaptations to assist with each individual request (e.g. computer programming time for specific statistics).

Thirdly, there is at present no central repository of the information and/or analysis that is provided through the current unofficial schedule of reviews. While the President and the Vice-President (Academic) are aware that reviews are being conducted and are usually informed of the results of the reviews, there is no mechanism to incorporate the results of the reviews in the University's academic planning process. Information from reviews could assist in establishing priorities and determining resource allocations by providing an indication of the institution's programme strengths.

Fourthly, there is at present no co-ordination of review activities although resources have been committed directly or indirectly (through Colleges and the central administration) to support these processes. The Working Group is concerned about this lack of co-ordination because "[m]any of the strategic choices that are implied in such notions as reallocation, setting priorities, competitive advantage, and the determination of institutional strengths and weaknesses require a synthesis of data. Isolated criteria applied to singular programme reviews do not provide the coordinated, institution wide perspective that is necessary to generate change"²².

Finally, a follow-up, an essential component to an effective review process, is lacking. Recommendations by the review committee and/or responses by faculty must be considered within a reasonable time period following the review itself. The follow-up can be conducted in three ways: 1) by the department itself; 2) by the college and/or central administration; and 3) by the University Council (through Academic Affairs). All of the information available to us points strongly to the need for a follow-up mechanism within central administration to ensure that the recommended actions are carried out and adjustments made to the programme being delivered. Otherwise, there appears to be little point to the expenditure of resources on these activities.

Are there any advantages of the current system? A possible advantage might be that the unit itself makes internal changes based on the results of the review; however, the central administration and thus the wider community is unaware of these changes except through the Academic Affairs Committee process (and this usually amounts to what looks like "tinkering" with the programme rather than change in an organized fashion). Another might be that the current system protects the status quo and therefore the current programmes are under less threat based on their being "buried" within the curriculum of the unit/College rather than scrutinized by the external community (either the central university or others). Thus, units are left with considerable leeway to make as little or as much change as they can support until they are faced with a budgetary reduction which forces them to lobby to support their maintenance at the current level .

²² Seymour, 40 - 41.

 

3) Purpose of the Review Process and Types of Review Processes:

An important consideration prior to embarking on such a major undertaking as a systematic programme review process is the ultimate purpose of the review process and the type of review process that will be recommended. This section addresses each of these issues.

a. Purpose of the Review Process: The Working Group believes that the purpose of the review process must be clearly articulated and understood. In order to do this, it is important to review the historical rationale for programme reviews. These indicate that programme reviews are generally undertaken "...to inform decision-makers about the strengths and weaknesses of programmes; to determine the status of programmes in relation either to specified standards or to the performance of other programmes; to provide information for planning; to help institutions make decisions about programmes - their installation, continuation, modification, expansion, or termination; to help an institution make decisions about reductions in expenditures and increases in efficiency, and to demonstrate accountability"²³. Generally too, the primary emphasis is placed on programme improvement²⁴. Indeed, this aspect of programme review was specifically mentioned in Vice-President Browne's initial discussion paper.

However, it is clear that programmes cannot continue to be reviewed with the expectation of an infinite allocation of resources upon completion of the review. We must reassess the purposes of the review process itself. In a recent publication by the Association for the Study of Higher Education entitled Developing Academic Programmes: The Climate for Innovation, Daniel Seymour provides a fresh perspective on the rationale for the review of programmes which links review of existing programmes to development of new programmes.

"Historically, research and writing in higher education have concentrated on the use of means to assess the strengths and weaknesses of ongoing programmes. The purpose of such assessments, usually in the form of programme reviews, has been the improvement of the programme's quality. More recently, the emphasis has shifted toward the use of programme review as a means to make decisions about the long-run viability of programmes and allocation of resources. Programme discontinuance, the detailed analysis of an existing programme for the purpose of deciding its future in the institution has also become more heavily studied as scarce resources have forced colleges and universities to retrench.

...In a growth environment, new programmes are a natural by-product of growth and are not perceived to require complex planning. Ideas emerge from the faculty, are supported by the administration or not, go through a standardized approval process, and then become a permanent addition to a college or university's brochure. While programme planning has not been seen as very important in a growth environment, it has not been seen as very practical in a no-growth environment. Institutions with high fixed costs, like colleges and universities, find it difficult to generate flexible resources. As highly decentralized institutions, they also find it politically and psychologically difficult to retrench and to innovate simultaneously.

Thus, while the corporate world sees new product development as requisite for its future existence (and, consequently, is intrigued with questions about the source of ideas for new products, the causes of failure, and planning new products) the academic world see the development of new programmes as an innate occurrence"²⁵.

Seymour goes on to define programme evaluation as "[t]he process of specifying, defining, collecting, analyzing, and interpreting information about designated aspects of a given programme and using that information to arrive at value judgments among decision alternatives regarding the installation, continuation, modification, or termination of a programme" ²⁶. He notes that several aspects of this definition are important : it makes judgments about programmes based on information; it sees judgements as decision alternatives not just descriptive data; and, it sees academic programmes in a temporal perspective (they come and go)²⁷. Further, the definition "...refers to three separate stages: the development of new programmes (installation), programme review (continuation, modification), and programme discontinuance (termination)"²⁸. From this definition, Seymour concludes that academic programme planning and programme evaluation are identical. He describes the key elements of academic programme planning as:

* a process which involves an ongoing series of actions.

* a data based process which requires both experiential and future-oriented information.

* a process that is related to the instructional programme component - graduate, undergraduate, degree, and nondegree.

* a mechanism that requires choices and setting priorities²⁹

Seymour concludes by stating that "[t]his broad view of academic programme planning, from programme development to discontinuance, is important because it is the context within which programme innovation occurs"³⁰. Programme innovation should surely be one of the major purposes attached to programme review and academic planning. Indeed, it can be argued that maintaining and developing quality programmes when resources are diminishing is an important activity in and of itself, i.e., transformation of the curriculum must occur, even in times of fiscal restraint. "If resources were not constrained, there would be insufficient motivation to endure the anxieties of unsparing self-assessment and the conflicts arising from the decision to replace old programmes with new ventures"³¹.

²³ Holdaway and Harris, "Approach to Reviews of University Programs, in CSSHE Professional File, Number 2, Fall 1987, 2.

²⁴ Ibid., 2.

²⁵ Seymour, Developing Academic Programs: A Climate for Innovation. ASHE-ERIC Higher Education Report # 3, Washington: Association for the Study of Higher Education, 1988, iv-v.

²⁶ Ibid., 30 [emphasis in original]. Seymour also defines programme "review" and "evaluation" and differentiates between them. For the purposes of this report, the Working Group has adopted the definition of programme evaluation in the broader context, i.e. encompassing the development of new programmes and/or the discontinuance of present programmes.

²⁷ See Seymour, 30.

²⁸ Seymour, 30.

²⁹ Ibid., 30-31.

³⁰ Seymour, 31.

³¹ Seymour, 40.

 

b. Types of Review Processes: The most common form of review process is the "unit" review. An alternative approach could be a true "programme" review. Both of these types of review processes are explored below.

"Unit" reviews: The university community is most familiar with the "unit" review concept. "Unit" reviews provide a large amount of information including 1) meeting specific needs; 2) a comparison of units of similar type in a provincial/national context; 3) a description of the expectations of the College/University on the unit under review; 4) a review of the resources allocated to the unit ; 5) a description of administrative functioning; and, 6) workload. Although sometimes labeled a "programme" review, few institutions actually conduct a "programme" review (as defined below).

When viewed as part of a strategic planning exercise, which by definition takes into account external forces and internal strengths and builds on these, the disadvantages of "unit" style reviews become more apparent. If academic planning is to take place in a resource-constrained environment, it must take into account the forces influencing the university 's curriculum and nurture innovation. Usually this means that criteria which force choices at the institutional level are selected to ensure that the programme offered remains relevant³². A major disadvantage to the "unit" review approach is that by definition the process is studying the current composition of programme offerings and therefore preparing a justification for the continued existence of the unit under review with perhaps a few modifications of some aspects of the curriculum. There is little incentive to units to change in a dramatic fashion because the unit itself has a vested interest in the maintenance of the status quo.

Further, unit reviews build on the current fragmentation of departments and disciplines rather than identifying mechanisms which might free up resources for innovation within the curriculum. This has led to a lack of institutional flexibility and a decrease in the options of an institution in the global sense. Unit reviews, while examining the scholarly activities of a department/College, may de-emphasize the programme offered by the unit under review and its interconnectedness to other programmes offered by the institution. Given the competition for resources within universities and the highly decentralized nature of these institutions, it will become increasingly more important that other ways of reviewing curriculum offerings are found to ensure innovation and continued relevance.

"Programme" reviews: The University of Saskatchewan describes a programme as "...a generally defined set of courses and other requirements described in the calendar, which the student must successfully complete to obtain a specific degree, certificate or diploma or other recognized qualification, such as meeting the admission requirements to another programme"³³. A possible method by which to review the university 's curriculum would be at the programme level³⁴. A student's perspective of the University is often focused on the successful completion of a programme of study rather than identifying with a "unit". Some reviews, namely accreditation reviews, are currently conducted as programme reviews.

Attempting to review curriculum offerings on a programme by programme basis would be inefficient and would not necessarily address the restructuring of instructional programmes within a unit. For example, noting that within the University's description of a programme, a major and honours degree are results of different programmes, illustrates how inappropriate it would be to conduct a review, programme by programme. On the other hand, unit reviews may not adequately examine those aspects of a programme which are delivered by other than the "home" unit and may miss major components of the programme that are being delivered to the student by other units. Further, while the major historical purpose of assessment has been the improvement of programme quality, there are no guarantees that this has occurred through the unit review approach.

It may be possible to develop a compromise between the "unit" and "programme" types of reviews; i.e., one which provides a unit-based programme-focussed approach to curriculum review. If such an approach were adopted, it would be important to recognize that the "unit" review concept is by far the most popular of review types currently in use and therefore it will be important to ensure that duplication of effort at the unit level is minimized. Also, If a new approach that combined the two types of reviews were adopted, the University may be placed in a position of greater responsiveness to its internal communities [faculty and students] and external environment [including Government and the general public].

"Given the opportunity, every programme can be justified as being essential to the mission of the institution. Accreditation agencies, programme alumni, impassioned students, and tenured faculty can make for a strong advocacy group. Consequently, the desire and the means to make comparative judgments about proposed programmes must...be available. Such judgments are part of a process that matches institutional strengths with a specifically defined mission - the result being a selective set of priorities"³⁵.

³² Seymour lists demand, quality, programme duplication, need, mission compatibility, and costs (in that order) as the crucial criteria. See 41.

³³ 1979 Report of the Calendar and Student Information Subcommittee on Recommended Nomenclature, A 8. This definition is virtually identical to Seymour 's definition.

³⁴ Seymour's concept of a programme portfolio provides a model for such a review process. "Such evaluation activities as new programme development, programme review, and programme discontinuance have been traditionally viewed as separate activities. A more appropriate perspective is to view the academic programmes that an institution offers as its programme portfolio. And as the institution's mission or environmental conditions change, that portfolio will necessarily evolve, with some new programmes added, others reconfigured or strengthened, and still others discontinued" . See Seymour, ix.

³⁵ Seymour, ix.

 

4) Implications of a University-wide Process:

Several implications exist for the University if a university-wide cyclical review process is undertaken. These implications include the cost of the review process, the overlap with existing review processes, the organization of planning, and, the degree of integration with the budgetary processes. Each of these implications is explored below.

Cost: Only partial cost figures from other universities are available to us. Given the current budgetary situation facing the university, the need for a systematic programme review must be clearly demonstrated and the allocation of resources to this activity justified. Nonetheless, resources, both college-specific and university-wide, are already being expended on review processes without having established any central repository for the results of these activities or mechanism for follow-up.

From the information available to us, there are a variety of review processes currently underway across the country and the extent of these processes is directly related to the resources available. Several models, ranging from those requiring minimum levels of funding to those requiring a high degree of funding, can be proposed. Whatever model is adopted, the University must be in a position to review its programme offerings at least once every seven to ten years or there would be little purpose in establishing such a process. Further, the university community must agree that the development of a university-wide perspective regarding its programmes is a valid activity.

Scope of a system-wide review of programmes: To gauge the number of years which would be required to complete a review of all programmes obviously depends on how we count the number of programmes and how many reviews per year can be supported. Assuming we will proceed with a unit-based programme-focussed approach to reviews, such that all undergraduate and graduate programmes in a department would be reviewed concurrently, the total number of reviews would be 70-90 (Appendix C). Over a 10-year cycle, this would mean 7-9 reviews per year.

Out of the approximately 70-90 programmes, 16 undergraduate programmes have an accreditation review process and the College of Arts and Science is committed to reviewing 4-5 departments per year. One major question yet to be resolved is the review process for colleges which have an accreditation process for their undergraduate programme(s), but also have department-based graduate programmes which are not currently reviewed; e.g., the College of Medicine's M.D. degree programme undergoes accreditation review by an external agency, however 10 departments in the College offer graduate (M.Sc./Ph.D.) and/or undergraduate (B.Sc.) programmes which are not reviewed. A similar problem exists for Veterinary Medicine.

Overlap with existing review processes: Whatever process is adopted or developed, it must work within the current time-table of reviews, that is, it must be co-ordinated with the existing review cycles for both accredited programmes and for review of units in the College of Arts and Science. Also, whatever proposal is developed must recognize the additional demands placed on the unit and/or programme under review while meeting the needs of the institution as a whole, namely, in the demonstration of accountability in its current programme offerings and in its responsiveness and/or flexibility in prioritizing current and new initiatives. If the assessment of programme strengths and weaknesses is to be considered in resource allocations, the university must have a role in determining the criteria for the review of programmes and in the implementation of review processes.

Organization of planning: The University must ensure that academic planning occurring at college and university level is integrated. Organization of academic planning at the university level must be clearly associated with a management portfolio so that planning activities can be directed to best serve the institution's needs. Our current structure must be reviewed to ascertain the best locus for academic planning activities. Further, academic planning must be linked to the budgetary process if it is to assist in the development of priorities for the institution as a whole and to guide colleges in their development of programmes under their jurisdiction. A clear and visible link with the budget process must be outlined if the academic planning activities are not to be undervalued by the institution.

5) Relationship to other University-wide Planning Activities:

If the University is to implement a systematic programme review process, the relationship of this process to other university-wide planning activities must be considered. A considerable amount of work/energy is required by all Deans and Department heads in responding to a myriad of requests for information; such requests should be coordinated whenever and wherever possible.

APPC's Methodology Working Group has been developing a system to analyze and present quantitative information which can be extracted from the University's computer databases. These "programme profiles" will contain information on enrollment, class size, source and cost of instruction and form the numerical basis for any self-study component of a systematic review process. Having developed a template and accompanying computer programming for programme profiles, the workload for the University Studies Group to support current and future review processes should be reduced.

 

F. A Proposal for Systematic Programme Reviews at the U of S

After consideration of these issues, the APPC has concluded that the University of Saskatchewan must develop a systematic review process as one of the strategies to demonstrate the excellence of our curricula and our commitment to continuous evaluation and change. We believe that the best approach to conducting such reviews is through the adoption of a unit-based programme-focussed approach to curriculum review.

A unit-based programme-focussed approach is essential to the University for many reasons. Such an approach to curriculum review offers the University the opportunity to be innovative even in times of fiscal restraint. Through a review of programmes and their interrelationships, the possibilities available to re-structure and adapt to the external environment, and therefore obtain greater programme flexibility, can be raised without threatening the existence of the sponsoring unit. An approach which considers the interconnectedness of programmes would provide a better picture of the programmes offered by the university as a whole and thereby provide a "...basis for more strategic decisions involving setting priorities, allocating resources, and eventually developing or discontinuing programmes."³⁶. This approach would emphasize "...the importance of strategic thinking and the interrelationship of the three stages [programme development, programme review, and programme discontinuance]"³⁷.

A unit-based programme-focussed approach would provide an administrative base for the review (a College or department), but would be structured to ensure programmes delivered to students remained the primary focus of the review. It would place greater emphasis on programme offerings and give a higher profile to the various programmes that can be taken within the University's programme portfolio. It would ensure that the university was better positioned to react to opportunities within the external environment, to capitalize on new initiatives, and to respond in a more innovative fashion in the design and maintenance of its current programme offerings. A programme-focussed approach would link existing programmes to the development of new programmes thereby creating a continuum wherein programmes could be created, reviewed (for continuation or modification), and terminated. It would therefore ensure that the programme offerings of the university are transformed through academic planning occurring at both the unit and university level.

Further, the creation of a unit-based systematic programme-focussed review process would ensure that all programmes offered at the University of Saskatchewan were reviewed on a periodic basis to confirm their continued vitality and relevance to the university's central mission. The systematic nature of the process would ensure that information gathered is consistent for all programmes and that reviews are conducted for the same purposes. As well, the systematic process would ensure that the University fulfills the expectation that it is an accountable institution to its stakeholder groups in a public manner. Adoption of such a process would demonstrate responsiveness to recent influences within the planning environment including review processes sponsored by Government. It would ensure that the University of Saskatchewan is synchronized with the practices of other major Canadian universities. It would fulfill one of the goals of the recently approved Mission Statement. In short, a unit-based systematic programme-focussed review process is a proposal whose time has come.

Thus, APPC recommends that the University of Saskatchewan initiate a systematic review of programmes on a regular cycle.

Prior to acceptance in principle of the above recommendation, the APPC recognizes that the campus community will require some information about the general nature of the process that is being proposed. At this time, the APPC is providing only a general outline of the proposed process. The process of programme reviews outlined below is the initial approach suggested by the APPC to undertake these reviews. The APPC will monitor and report to Council in a regular way on issues that arise from the proposed review process.

In approving the preliminary proposal, the intent is that APPC initially serve as a Steering Committee to refine and develop policy for reviews, clarify issues related to costs, and "supervise" an initial set of reviews. Before beginning any of the reviews, even on a trial basis, the proposed guide for self-study guide and external reviewers will be sent to the Academic Affairs Committee for their approval (by December 1994). After completing the trial period, a complete report including specific recommendations on the feasibility, conduct, and structure of reviews will be prepared for approval by Council.

 

Proposal for the Structure of the Review Process:

The systematic unit-based programme-focussed approach recommended by the Committee is based on the following concepts:

" The focus of all reviews will be the academic programmes.

" Every programme will be reviewed at least once in a seven to ten year cycle.

" Each programme will be reviewed in a systematic way, i.e., all programmes will be reviewed (including interdisciplinary programmes). A similar methodology will be used for every review but sufficient flexibility will be allowed to accommodate differences in programmes and units.

" The review process will be co-ordinated to eliminate duplication of effort.

" The review process will identify a central repository of information for all reviews.

" All reviews will take place in the light of resource availability. Consequences of reviews may be to expand, modify, combine, delete, and/or downsize programmes.

The review process will describe on the following: Purpose, Organization of Planning, Phases/Stages of Review Process, Information for reviews, Issues to be addressed in review by the programme itself and by the external reviewers, Resources, and Consequences/Follow-up³⁸. Each of these elements is described below.

1. The primary purpose of the systematic programme review process will be to evaluate the quality of the academic programmes offered at the University of Saskatchewan. Other purposes for the review process include:

" informing decision-makers about the strengths and weaknesses of programmes with a view to programme improvement;

" determining the status of programmes in relation either to specified standards or to the performance of other programmes;

" providing information for planning;

" assisting in the decision-making process about programmes - their installation, continuation, modification, expansion, or termination;

" assisting in the determination of reductions in expenditures and increases in efficiency, and demonstrating accountability.

2. The review should examine among other characteristics: objectives of the programme and the extent to which they are met, quality of the programme, demand, programme delivery, cost (efficiency, links to other programmes/units), uniqueness (provincial, national), and, the programme's relationship to College and University Mission Statement.

3. The review should address key issues which might change over time but which would be identified in guidelines for the self-study phases as well as in directions to external reviewers.

Such issues might include the following questions: What is the quality of the programme? If the quality of the programme is poor, how can it be improved? If the demand for the programme is low, is the demand a reflection of the programme quality? Can the demand be changed? If the demand for the programme is too high, is it realistic to accommodate this demand? Are there more effective and efficient ways to deliver the programme?

4. The components of the systematic programme review process should include a self-study, an external review, the unit's response to the external review, and a final report evaluating the external review and the unit's response.

As indicated earlier, APPC, in consultation with departments and colleges, would be responsible for developing the guides for self-study and external review and submitting them to the Academic Affairs Committee for approval. These interim guides and procedures would be used for reviews during the trial period, and modified as required.

5. The information collected for the systematic programme review process should, wherever possible, build on the current information required of units which are accredited or which have approved academic review processes. In cases where no approved review process exists, the systematic programme review process should be adopted.

Programme reviews should take place in association with other review processes wherever possible and avoid unnecessary duplication of effort in Colleges/Departments which already participate in a review process. How we might conduct and co-ordinate reviews of those programmes which already undergo an accreditation or other type of review will be determined in consultation with the Deans of accredited programmes and the Academic Review Committee of the College of Arts and Science.

6. The array of programmes offered within a unit or a non-departmentalized college should be reviewed at the same time wherever practical.

This would avoid duplication of effort and allow an examination of the interdependence of programmes within the unit.

7. All programmes must undergo a review process. The sequence of reviews will be determined through a combination of the following methods: 1) the existing "informal schedule", 2) voluntary participation by programme heads; or 3) identification by the Academic Affairs Committee, the Budget Advisory Committee, the President, and/or the Vice-President (Academic).

An "informal" sequence of reviews, based on schedules for accreditation reviews and the College of Arts and Science review of academic units already exists. The sequence for the systematic programme review process should take that existing sequence into consideration. In addition, the results of any review should be reported centrally.

8. The University should establish a support structure and follow-up mechanism for the systematic programme review process.

At a minimum, establishment of a support structure would include:

" identifying a coordinator of review processes

" exploring the possibility of establishing an academic planning and review committee.

At a minimum, establishment of a follow-up mechanism would include:

" monitoring of results of reviews by the Academic Affairs Committee including recommendations to Council for action where appropriate.

" reporting results of reviews to the President, the Vice-President (Academic) [who will have the ultimate responsibility for ensuring that the review is conducted and that the recommendations are carried out], the Dean of the appropriate college, the programme head, and the Academic Affairs Committee of Council.

" reporting the status and progress of reviews to Council in a general way.

" ensuring that plans for programmes/units arising from the review process become part of the academic plan for units (including budget submissions and proposals for major changes, courses, and programmes) [thereby linking academic planning and resource allocation in a visible manner].

9. Programme reviews should be co-ordinated and funded from central sources.

Resources will be required at the unit, College, and university level, for the external reviewers, and for the University Studies Group.  

³⁶ Seymour, 48.

³⁷ Ibid., 48.

³⁸ R. J. Barak and B. E. Breier, in Successful Program Review. (San Francisco: Jossey-Bass Publishers, 1990 offer the following as important components of the programme review process: "a statement of purpose, expected results, and expected uses; criteria for judging results of the review process and for judging the design of specific reviews; specification of administrative responsibilities and policies for conducting reviews and distributing results, monitoring implementation, and providing information; design policies and guidelines; procedures for identifying new information needs, updating review plans, and evaluating the review process itself; resource requirements (staff and dollars); and a summary schedule of programme reviews." See page 39.

 

SPR Home