Systematic Program Review - A Framework for the Evaluation of Academic Programmes

A Framework for the Evaluation of Academic Programmes

Academic Planning and Priorities Committee, April 29, 1996

A. Introduction

The University of Saskatchewan's Mission Statement, adopted in May 1993, states that "As an academic community, our mission is to achieve excellence in scholarly activities of teaching, discovering, preserving and applying knowledge" and lists as one of the University's goals "to offer a rich array of challenging academic programs."¹ Although every programme currently offered contributes to the Mission and Goals of the University of Saskatchewan, the array of programmes is not static but changes with time, influenced by student and public demand for new areas of study, by the interests of faculty, by emergence of new fields of specialization 'within' and 'between' disciplines, and by funds available to operate programmes. The University Objectives, developed from this Mission Statement, noted that:

"In continuing its commitment to a rich array of challenging liberal, scientific, professional and artistic educational programs, the University of Saskatchewan wants to ensure that its undergraduate, graduate and certificate programs are of high quality. To maintain quality, the University, as well as its Colleges and Departments, must make choices to change, combine, or delete existing programs and specializations, and to introduce new initiatives."²

Further, Objectives A12 and C3 emphasized the need for processes to identify areas within the current array that should be preserved, strengthened, modified, consolidated, or deleted; to identify and initiate innovative programmes to add to the existing array; and to integrate financial, academic and administrative planning. In any such process of evaluating or making decisions related to programmes, it is important that all programmes be reviewed using the same basic set of criteria or framework -- thus, the framework developed and described herein.

The first version of A Framework for the Evaluation of Programmeswas developed by the Academic Planning and Priorities Committee (APPC) in April 1994 and distributed as a "working document" to department heads, deans, the Academic Affairs and Budget Committees during the 1994-95 academic year. An abbreviated version of the document was also distributed to the campus community in the Special Report on Academic Planning(June 1994). Discussions were held with various groups and comments were invited from all who received the document. This version, dated April 29, 1996, incorporates changes resulting from those discussions and APPC's experience using the framework during the course of the Programme Audit Project.

¹University of Saskatchewan Mission Statement,approved by Council (April 1993), Senate (April 1993) and Board of Governors (May 1993).

² University of Saskatchewan Objectives,approved by University Council, April 8, 1994.

B. The Framework

While recognizing the University's many roles include teaching, research, extension and public service, this framework was constructed for the purpose of evaluating the University's academic or instructional programmes; i.e., courses of study leading to a degree, diploma or certificate. The framework is based on identification of those characteristics of primary importance for academic programmes and provides a list of general criteria,applicable to most, if not all, programmes.

A Framework for the Evaluation of Academic Programmes

Of primary importance to the University of Saskatchewan is that academic programmes:

be of high quality
be in demand by students and the public
use resources efficiently

In addition to the three primary characteristics related to quality, demand and resources, for some programmes it is also important to consider:

the unique features of a programme, and

the relevance of the programme to Saskatchewan

We must also keep in mind other of the University of Saskatchewan Objectivesincluding our commitment to fair and equitable accessto our programmes, to equity, to environmental responsibility, and to an international perspective in our endeavours.

The University is committed to developing programmes which exhibit the three primary characteristics above, recognizing that our present programme strengths lie with the programmes which exhibit those characteristics. Programmes should be assessed so as to direct change both in terms of programming and resource allocation; i.e., to determine how/why they do not meet criteria and whether changes to remedy the situation are feasible.

The unique features of a programme and its relevance to the province should be viewed as characteristics of secondary importance, having first established the quality of, demand for, and costs associated with a programme. Certain core disciplines/programmes represented within any university are not expected to be unique. However, it is still possible that instructional methods or particular sub-specializations might be described as unique within the province or region. On the other hand, being 'unique' assumes greater importance where the cost of delivering a programme is high or demand for the programme is low. The importance or relevance of a programme to the province may relate to building on economic or other strengths which already exist. On the other hand, a programme may serve as a nucleus contributing expertise and services which would otherwise be unavailable to the community.

The Table which follows identifies essential components for each of the primary characteristics and general criteria associated with them. The components are not necessarily independent and therefore similar criteria may appear for different components; e.g., both curriculum and learning environment have a criterion related to education equity. The relative importance of criteria for different types of programmes varies; e.g., success of faculty in obtaining external research support may be more important for graduate than undergraduate programmes and more important for the natural sciences than the humanities.

A Framework for Evaluation of Academic Programmes
Characteristics	Components	Criteria
High quality	Curriculum	curriculum is designed to meet the objectives for the programme (e.g. array and sequence of courses, modes of instruction and evaluation, development of skills, acquisition of knowledge, synthesis of information). programme provides students with the elements of a liberal education by encouraging the development of broadly informed, reflective and literate minds capable of independent and critical thinking. programme includes opportunities for synthesis, application and integration of knowledge within and between disciplines. programme is current, both in content and modes of instructional delivery, and reflects a responsiveness to changes in the discipline. curriculum reflects the goals of education equity. curriculum provides sufficient flexibility to individual students to choose courses according to their own interests within and outside their major discipline (e.g. electives). programme meets or exceeds accreditation and/or national standards (if they exist).
	Faculty	faculty responsible for/involved in programme are well- qualified; i.e., have the appropriate academic and/or professional qualifications to support and develop the programme. In the case of graduate programmes, this includes active involvement in scholarly work. faculty maintain and update the skills and knowledge appropriate to their discipline through involvement with academic, professional and/or scientific organizations. faculty are nationally/internationally recognized for their scholarly and/or professional work; e.g., have obtained awards and/or local/national/international invitations to present their work to colleagues in their discipline. scholarly work of faculty has made a significant contribution to the discipline. faculty are committed to developing their teaching skills. faculty are successful in obtaining external research support.
	Learning Environment	students are involved in evaluating instruction and perceive instruction is effective. programme incorporates a variety of modes of instruction, accommodates different learning styles and, where possible, allows flexibility in scheduling. teaching within the programme demonstrates responsiveness to new developments in the field, including incorporating practical experiences, where appropriate. programme integrates teaching and scholarship. approaches to instruction and students reflect a commitment to the goals of education equity. instructional methods and philosophies are consistent with programme objectives. scholarly work of faculty enhances the learning environment.
	Infrastructure	adequate numbers of appropriately trained staff are available to support the programme. necessary facilities and equipment are provided. appropriate library resources are available. the organization and administration of the programme and the academic unit(s) delivering the programme are effective and supportive of the programme.
	Outcome	programme achieves its educational objectives. students are satisfied that the programme has helped them achieve their personal and/or professional goals. students completing graduate programmes are 'successful' in that they find employment or pursue endeavours which utilize the advanced training in their field of study. the academic load does not impose undue barriers to completion such that students can complete the programme in the regular allotted time. qualifications/education of students graduating from specific 'professional' programmes are acceptable to licensing bodies and/or employers. students are successful in national/international examinations or competitions. employers or subsequent graduate supervisors are satisfied with the performance and academic preparation of students.
In demand	Student demand, Market demand and/or societal need	interest by students of the Province is sufficient to establish or to maintain a programme and to allocate resources to it. market demand (provincial, national) for graduates justifies the size of the programme which is offered by the University. the programme attracts outstanding students from within and outside the Province, while still providing general access to other applicants. high demand for junior 'service' courses is sufficient to maintain *some* programmes within an academic area.
Uses resources efficiently		programme is delivered in a cost-effective manner, relative to other similar programmes. where student demand for a programme is low, high demand for 'service' courses justifies maintenance of the area of study and the incremental cost of offering the programme is low. major areas of research, scholarly or artistic work are associated with opportunities for graduate education.

Unique Relevant to the province		programme is unique in content (e.g. specialization) and/or approach - nationally, regionally (Western Canada), provincially [in descending order of priority]. programme builds on and contributes to the cultural and economic strengths of Saskatchewan. faculty and other personnel associated with the programme provide services and expertise otherwise unavailable.

C. Applications of the Framework

The framework was developed to apply to both existing programmes and new programmes and to guide not only academic decisions, but also decisions on allocation of resources.³ To date the framework has been or is being used by APPC:

to conduct the Programme Audit Project, a survey of all existing instructional programmes.

to design the self-study and external reviewers' guides for Systematic Programme Reviews.⁴

to develop Academic Affairs Committee guidelines for submitting and evaluating proposals for programme revisions.

to develop Academic Affairs Committee guidelines for submitting and evaluating proposals for new programmes.

While the University has always, and should always, focus on the academic merit of programme proposals, our financial outlook requires more attention to be paid to the resource and budgetary implications of any proposal for a new or revised programme. Regardless of the origin of initiatives for new programmes, without new external funding, support for new programmes can only be realized by re-allocation of funds, or from restructuring and combination of existing programmes.

When faced with decisions which require a comparison between an existing and a proposed programme or among several proposed programmes, there must be a rigorous process of selection and discrimination in arriving at decisions about which programme proposals to support. Even when a new programme does not require significant re-allocation of funds, it is important that the process for reviewing new programmes be consistent with criteria used to review existing programmes.

Changes to existing programmes and specializations may also require re-allocation, as would decisions that the quality of a set of programmes can be maintained only with additional resources and elimination of some programmes.

Resources should be allocated taking into consideration:

a) the strength of the programme [as measured by criteria described in the framework.]

b) the number and type of specializations in a discipline which can be supported within the available resources without compromising the integrity of the core of the programme.

c) the minimum size of a programme

Below a particular enrolment level, the viability and cost per student of a programme become major questions. With a low level of demand for a programme, especially when the majority of courses are specific to and only used by students in the programme, it may not be cost-effective to offer the programme. However, in some cases, because of interaction between departments or Colleges in delivering programmes, maintaining certain courses in a discipline may be essential. This is not to imply that high demand for junior 'service' courses is, in and of itself, reason to maintain all programmes within an academic area. In such cases, it will be important for the University to develop a method to determine the incremental costs involved in maintaining the major.

d) the maximum size of a programme the University should support.

When student demand exceeds our capacity to deliver a programme, admission and/or enrolment quotas may be established to limit the size of a programme, or additional resources may be allocated to meet the demand. Enrolment should be limited when meeting student demand would compromise quality of the programme or result in insufficient resources being available for other high quality programmes. For those programmes which are more specific and less flexible (e.g. those whose primary objective is to educate and train professionals), decisions on enrolment quotas should consider market demand for graduates as well as student demand for entry.

³ Framework for the Evaluation of Programmes. A Working Document. April 29, 1994.

⁴ A Background Paper and Proposal for a Systematic Programme Review Process at the University of Saskatchewan. Approved by Council, April 1994.

D. Conclusions

Growth and change are critical to maintaining a vital academic institution. Change may take the form of minor or major revisions to existing programmes, or deletion and addition of programmes. At the same time, the University is committed to reviewing all of its programmes on a regular cycle, regardless of whether curricular revisions have been proposed. While the need for changes to existing programmes and opportunities for introducing new programmes may be identified by Departments, Colleges or the University as a whole, and occasionally by groups outside the University, it will be the faculty within the discipline and cognate disciplines who must ultimately develop ideas for proposed changes or new programmes. Departments, Colleges and the University must direct the change and ultimately decide on whether proposals should be approved or rejected.

The framework is intended to be used as a basis of a co-ordinated academic planning process on all levels within the University. It provides a basic set of criteria which is intended to serve as a template for the evaluation of programmes. Discipline-specific criteria and criteria differentiating expectations between levels of programmes (e.g. honours versus major, graduate versus undergraduate) will be developed for 'in-depth' reviews of programmes (i.e., for the Systematic Programme Review process). For those programmes which currently undergo accreditation reviews, such specific criteria likely already exist. In addition to any specific criteria which might be developed, the general criteria within the framework should be viewed as open to review and revision from time to time.

SPR Home