Select
a different paper
Download a copy of the entire paper
Introduction
This paper examines assessment methods which can be used to evaluate peer interaction in Web-based education using computer-mediated communication. Traditional assessment methods do not measure or evaluate the development of learning demonstrated in peer interaction. In order to examine assessment of interaction in learning, this study first reviews the pedagogical foundations of interaction and learning. There is an extensive body of research which concludes that interaction using the medium of computer-mediated communication can facilitate learning. This study will review the critical factors for the successful use of this medium. Alternative assessment methods which are used to evaluate learning as well as accomplishment are discussed.
The purpose of exploring this topic is to discover how instructors can harmonize the principles of learner-centred education with student assessment in an electronic environment. Case studies of university courses were selected to illustrate assessment strategies and peer interaction using computer-mediated communication.
Definition of Terms.peer interaction, collaboration, group learning, cooperative learning, network learning are all terms describing the type of interaction this paper addresses (Harasim, 1996). This paper does not explore interaction between instructor and learner except in the case of assessment feedback.
Computer-mediated communication (CMC) includes text formats of communication such as: email, bulletin boards and synchronous and asynchronous communication. This paper will focus on group communication events: 'one-to-many,' 'many-to-many,' or 'computer conferencing' (CC). These formats are open documents available to all members of a course.
Assessmentdescribes "the measurement of progress toward a learning goal" (Simonson, Smaldino, Albright & Zvacek (2000), p. 207).
Evaluation describes the critical examination of instruction for effectiveness in achieving learning outcomes. Much of the literature uses these two terms interchangeably when describing student assessment (Simonson et al. (2000).
Interaction and LearningInteraction appears to be an important factor in learning. Rowntree asked in 1995 what elements of learning would occur without interaction:
But if we want to help them [learners] formulate questions of their own and make their own meanings, then discussion of some kind (though not necessarily with us) seems essential. What I suspect colleagues are likely to be differing about here is not whether it is possible to learn without interaction but about whether non-interacting learning can be worthwhile (educational?) learning. (as cited in Berge, 1999 p. 5)
In Berge's descriptions of Web-based instruction (1998) he discusses pedagogical, technical and social principles which guide the design of online instruction. All three have implications for peer interaction. The purpose of including peer interaction within an online course should be to assist in meeting the learning outcomes. The supporting technical framework should be transparent to the learner and easy to use. The learner should perceive a friendly social environment which encourages the development of trust and cooperation among participants.
Social interaction among peers is important to learning (Bonk & Cummings, 1998; McIsaac & Blocher & Mahes & Vrasidas, 1999; Moller, 1998; Morgan & O'Reilly, 1999). First, social interaction allows learners to establish a personal connection to other students and to the instructor. Without this foundation, authors agree interaction at an educational level is less effective.
What then are the expected benefits of peer interaction? Cazden's summary of the cognitive benefits of peer interaction includes four major points:
1) students are forced to confront each other's ideas; 2) students can enact complementary roles, provide mutual guidance and support, and can serve as scaffolding to help each other accomplish learning tasks that might otherwise be too difficult; 3) students can find a direct relationship with a real audience from which they can get meaningful feedback; and 4) students can experiment and construct new understandings and ideas in peer discourse setting. (as cited in Ruberg & Moore & Taylor, 1996, p. 245)
Peer interaction occurs on two dimensions: social and cognitive. Interaction among learners will not spontaneously occur just because it is technically possible (Gilbert & Moore, 1998). All interaction must be planned events with targeted outcomes (Flynn, 1992). Without instructional planning, interaction might not move beyond a comfortable social experience into the cognitive level (Gilbert & Moore, 1998).
CMC and LearningWith the advent of online education in the early 1980's, innovative educational technologists have combined the delivery of content with methods of communication and interaction (Burge 1998, Eastmond 1996, Harasim 1999). CMC used for collaboration has been widely acknowledged as most appropriate in higher education (Dehler & Porras-Hernandez, 1998; Eastmond, 1996; McIsaac & Blocher, 1998) where one finds learners who are "self-directed, gregarious, technically proficient, and motivated … The upper-level 'seminar,' with its discussion expectations, manageable class size, student familiarity with content, and closer teacher-student relationships is often seen as the ideal environment" (Eastmond p. 382). Surveys of online education reveal a wide variety of disciplines using CMC: education, humanities, languages, social sciences, computer science, and some sciences (Harasim, 1999; Mason & Bacsich, 1998).
Eastmond, in the quotation above, identifies some of the personal attributes of learners who benefit most from CMC interaction: self-directed; motivated; and knowledgeable in the subject area. He also identifies group size as an important factor in promoting lively discussion. Eastmond and others (Dehler & Porras-Hernandez, 1998; Harasim, 1987) identify two technical aspects of CMC which facilitate group learning: the recording of group communication which can be accessed at learner's convenience, and the ability to provide real-time (synchronous) or delayed time (asynchronous) encounters.
CMC is like an online library with a meeting room. The resources are the collected opinions and discussions of the learners and the instructor. The 'collection' and the meeting room are open 24 hours a day. As the learners interact with each other in the meeting room and review the documents in the collection, they have the opportunity to reflect upon their own learning.
The Open University (OU) in the United Kingdom has used CMC since 1986 to support teaching and learning and it is now "embedded into the culture (although by no means into all of its courses)" (Mason & Bacsich, 1998, p. 249). With over 20,000 students, as of 1998, using CMC from one institution, OU provides a valuable model to researchers (Harasim, 1996; McIsaac & Blocher, 1998).
Mason and Bacsich's recommendations on the use of CMC. Mason and Bacsich surveyed instructors and students from OU. They found the following nine elements common to successful use of CMC. They are:
Tolmie and Boyle's recommendations on the use of CMC
Tolmie and Boyle (2000) have developed a similar list of seven critical factors necessary for successful use of CMC.
They include:
Analysis illustrates the complexity and dynamics of control issues in instruction, interaction and CMC. Mason and Bacsich's recommendations focus more on instructor perspectives. Tolmie and Boyle, while echoing many of the same points, speak more the student's perspective. With this in mind, it is all the more surprising that Tolmie and Boyle have no recommendations about assessment. Both agree that the instructor using CMC must structure opportunities for interaction and group development.
Alternative AssessmentA priority for students in post-secondary courses is an understanding of the evaluation criteria used to determine their success or failure (Morgan & O'Reilly, 1999).
There is inherent tension in the assessment and evaluation of collaborative learning (Morgan & O'Reilly, 1999). Between the poles of teacher-centred instruction and learner-centred education, group learning falls within the domain of learner control. Yet ultimately, assessment of learning is the responsibility of the instructor. Alternative assessment reduces the tension between instructor and learner control.
Alternative assessment is a method which incorporates constructivist principles into assessment. Three approaches are described by Simonson et al. (2000). Authentic assessment uses real-world simulations to determine the degree of transfer of classroom knowledge to typical real life situations; performance-based assessment involves performing a skill and may involve the student's description of knowledge about the skill; constructivist assessment, which may involve collaborative learning, emphasizes thinking strategies used by the student. All approaches involve the learner in selecting the assessment method and establishing the assessment criteria.
The instructor may base the assessment of student accomplishment on portfolios and exhibitions compiled by the learner. The learner selects material which demonstrates a level of knowledge of content plus a demonstrated ability to apply knowledge. Portfolios and exhibitions may be difficult to evaluate (Simonson et al, 2000). Their use, however, provides motivation for engagement in collaborative exercises (Moller, 1998), especially where a learning goal is the development of team building skills.
It is important to note that early advocates of the use of group learning using CMC like Beckwith (1987) did not recommend activities such as problem-solving necessarily be assessed. "Applied as a formative evaluation tool, the instrument may be used to monitor, control and improve group progress" (p. 104). Beckwith, in his article, was exploring the untapped potential of CMC to facilitate group learning. While great changes have occurred in the use of CMC for collaboration in the last 15 years, assessing group learning is not widely applied.
Bonk and Cummings (1998) and Morgan and O'Reilly (1999) suggest that assessment tools need to be re-evaluated for Web-based education.
To take advantage of the medium, we need to harness new opportunities for interaction, dialogue and debate among open and distance learners, and to create assessments that promote the value of these learning processes. (Morgan & O'Reilly, 1999, p. 41)
Morgan and O'Reilly recommend seven assessment tools: "peer and self-assessment; teamwork and collaborative assessment tasks; online dialogue and debate; simulations and role plays; problem solving; online testing; digital scrapbooks and portfolios" (p. 35). These tools are selected with learner involvement. Morgan and O'Reilly recommend the instructor provide frequent feedback, or formative evaluation, to learners as group communications as well as private communications to individual learners.
Several authors encourage the use of alternative assessment. Portfolios, exhibitions and learning contracts are described by Nevo (1996), Mason and Bacsich (1998), and O'Donnell and Caffarella (1998). Some institutions, like the Virtual University (VU) developed at Simon Fraser University, have applied alternative and group assessment widely. In a report which appeared in 1999 (based presumably on the previous academic year), Harasim reports that as part of final assessment 21% of courses grade group assessment; 25% of courses grade online participation, 13% of courses grade "incorporation of process (that is, cooperation or how well students developed issues" (p. 48). Researchers at VU are currently developing models for measuring online learning. It will be interesting to read of their progress.
Selected Case Studies : Interaction, Learning and CMC
Three case studies have been selected which illustrate research methods evaluating interaction, CMC, and learning in university level courses. This number does not represent the total research in this area by any means. The studies were selected using the following criteria: they were short term studies of specific courses; the authors used measurement standards to assess their research questions; the courses illustrated unique applications of course design and CMC; the courses incorporated group activities some of which were assessed by instructors; and, the studies are relatively recent representing a mature application of CMC.
There is a wide variety in strategies for analysing electronic education which these three cases illustrate. Each research team has selected a different measurement instrument for evaluating learning. They also use different theories for analysing messaging in an electronic medium. The summaries provided below do not include a detailed description of each study except as they address the three issues of evaluating learning, evaluating communication, and describing assessment tools.
1. Bonk and Cummings (1998). "A dozen recommendations for placing the student at the centre of Web-based learning." Bonk and Cummings designed and implemented a course for pre-service teachers at Indiana University using the American Psychological Association Learner-Centered Psychological Principles (http://www.apa.org/ed/lcp.html#Standards) to evaluate learning. They developed a 12-point checklist for student-centred Web-based learning. Their recommendations include assessment of student and group learning, and Web-based conferencing. Group activities embedded in instruction are discussions and debates. In addition, students could post "reflection logs." The instructors used alternative assessment to evaluate individual student performance using a student portfolio. It is unclear which method was used for instructor assessment of group projects. Peer evaluation of co-operative projects was based on mastery learning checklists. The Bonk and Cummings case study provides a valuable guide for incorporating student activities in the Web environment and the evaluation of learning.
2. Howell-Richardson and Mellar (1996). "A methodology for the analysis of patterns of participation within computer mediated [sic] communication courses." Howell-Richardson and Mellar studied two separate computer conferences within a training course for educational trainers. The purpose of the study was to examine the role of the moderator in promoting learner-independence through collaborative and peer-learning. One group, which was a subset of the entire class, had a student moderator. The other group included all students and had a 'guest speaker' moderator who was the author of one of the course readings. The student moderator's role was to a large degree predetermined by the group and the course tutor. The study compared messages in both groups using the Speech Act theory. The researchers conducted a detailed breakdown the following: characters per message, messages initiated by students and moderators, distribution of messages during the duration of the conferences (each conference took place over several days), and links connecting messages with responders. In addition, the messages were analysed for content under the categories of group focus or task focus. Group messages were classified as: organizational, rechannel, socio-affective, debilitative, metacomment. Task messages were classified as: initiate, reject/disagree, confirm, refer, summarise, request.
Howell-Richardson's and Mellar's conclusions were not surprising. Although both student groups were satisfied with their experiences, there was a significant difference in the types of messages in the small group moderated by a student compared with the whole class moderated by a guest speaker. The plenary group spent more time discussing tasks and less time discussing group issues. The plenary group had 70% task-oriented messages and 13% group-oriented messages. The subgroup had 54% task-oriented and 43% group-oriented messages.
Howell-Richardson and Mellar did not put forward any recommendations on course design for collaborative learning and CMC based on their conclusions. Their study is, however, significant to the topic of this paper. It provides an example of how messaging can be analysed in a quantitative manner in order to support or refute an hypothesis on CMC. They claim the Speech Act theory can be applied successfully to analysing CMC:
Speech Act theory provides a means of probing the structure of discourse both in terms of surface relations of forms, and underlying relations of communicative functions and relates these meanings to their context. (p. 52)
Their study is also significant in that it describes a real-life example of the influence of an expert speaker in CMC discussions, a strategy which can be used to vary the type of dialogue in groups.
3. Ruberg, Moore, and Taylor (1996). "Student participation, interaction and regulation in a computer-mediated communication environment: a qualitative study." Ruberg, Moore, and Taylor published a study documenting the use of CMC to supplement classroom work in a plant science lab course. The goals of project were to "provide an interactive forum where students were encouraged to explain their ideas and thereby extend and refine their newly acquired knowledge through meaningful dialogue with their peers" (p. 244). The authors selected a taxonomy developed by Marzano which links cooperative learning tasks with levels of thinking. The authors selected exercises which addressed two levels of effective learning: "thinking dimensions three and four: extending and refining knowledge and using knowledge in meaningful ways" (p. 246). Message interactions were tabulated using quantitative analysis: sender, words per message, and ratios of participation. The message content was analysed using Levin's "message act analysis" system which is based on Initiation-Response-Evaluation (IRE) sequences (initiation by teacher, reply by student, evaluation by teacher) initially described by Mehan. In addition to "message act analysis," messages were assessed for rhetorical content using Butler's coding system. Ruberg revised Butler's system of 12 categories into 10: "1) question, 2) reply, 3) consensus building, 4) evaluation, 5) topic initiation, 6) assertion, 7) acknowledgment, 8) off-task, 9) qualification/definition, and 10) clarification/elaboration" (p. 256). Their conclusions were based on "observations, interviews, analysis of electronic transcripts and self-reports from students" (p. 244).
An important element of this study was the comparison of interaction and learning in the face-to-face sessions with the CMC sessions. In spite of some problems using the technology of CMC, the authors observed more even distribution of participation in CMC. The students in the courses also commented on this point, identifying it as a positive aspect of the CMC discussions. Students supported the use of CMC in this course, stating they had learned from discussions. It appears the only positive outcome of this application of CMC was increased participation which improved overall group development.
The authors do not clearly relate their evaluation of interaction and learning compared to Marzano's model of tasks and effective learning. The authors point to the need for critical selection of tasks and more linking with previous learning in order for this medium of interaction to enhance learning. They describe significant problems with lack of preparation for CMC dialogues and a failure to "make conceptual connections" (p. 265) which would strengthen learning. These points may illustrate a gap in the use of Marzano's model to link tasks to higher level thinking skills.
Future Directions
Assessment of peer interaction using computer-mediated communication will continue to benefit from research conducted in three areas: interaction in Web-based education; learning in Web-based education; and assessment strategies for group learning. One of the major challenges in all areas is discovering elements of success which transcend individual applications (Burge, 1994; Mason & Bacsich, 1998). Research at institutions like the Open University, Virtual University, and other centres of excellence in the application of educational technology and online learning will have the resources to examine applications over many disciplines and with different levels of learners.
Should case-based research in these areas evaluate according to quantitative or qualitative principles (McIsaac et. al., 1998)? The Howell-Richardson and Mellar study may be criticized for its complex and elaborate proof of a generally accepted assumption; the moderator's expertise and assigned role can affect discussion. McIsaac et al. (1999) and Burge (1994) produce compelling arguments for the use of naturalistic evaluation. They place strong emphasis on instructor and student participants' perceptions to guide the interpretation of data. As long as a comfortable social environment is the foundation for cognitive development in the electronic environment (Bonk & Cummings, 1998; McIsaac et al., 1999; Moller, 1998; Morgan & O'Reilly, 1999), it seems unlikely that statistical data alone can describe the elements of its success or failure.
ConclusionIn examining this topic I am reminded of the metaphor of fabric used by Burge in her article published in1994. The issue of engagement of peer interaction in computer-mediated communications prompted an exploration into the topic of assessment of interaction and learning. In this case, interaction, engagement, and assessment loop over and under each other to form the fabric of learning. Learning increases with interaction; a community of learners interacts more if they can relate to each other on a personal, social level; interaction improves learning if it has an identifiable task and outcome; learners become more engaged in tasks if they are assessed on their performance; assessing group activities motivates students to interact on a cognitive level, which increases learning.
Computer-mediated communication is now a mature medium widely used at all levels of education. Its use is often not a choice but a necessity particularly for today's online and distance learners. Technical advances are already changing it from the predominately text-based format we use today into a multimedia event. As educators we should remember it is a tool not a diversion, and its application should be directed at creating opportunities for measurable learning.
Beckwith, D. (1987). Group problem-solving via computer conferencing: the realizable potential. Canadian Journal of Educational Communication, 16(2), 89-106.
Berge, Z.L. (1998). Guiding principles in Web-based instructional design. Educational Media International, 35(2), 72-76.
Berge, Z.L. (1999). Interaction in post-secondary Web-based learning. Educational Technology, 39(1), 5-11.
Bonk, C.J. & Cummings, J.A. (1998). A dozen recommendations for placing the student at the centre of Web-based learning. Educational Media International, 35(2), 82-89.
Boschmann, E. (Ed.). (1995). The electronic classroom: A handbook for education in the electronic environment. Medford, NJ: Learned Information, Inc.
Brescia, W. & Schaumburg, H. & Duffy, T.M. Collaboration over the Web: Strategies and goals. [online]. Available: http://www.ihets.org/learntech/distance_ed/. (January 10, 2001).
Brookfield, S.D. (1998). Discussion. In: M.W. Galbraith (Ed.), Adult learning methods: A guide for effective instruction (pp. 169-186). Malabar, FL: Krieger Pub. Co.
Burge, E.J. & Roberts, J.M. (1993). Classrooms with a difference: A practical guide to the use of conferencing technologies. Toronto: Ontario Institute for Studies in Education.
Burge, E. J. (1994). Learning in computer conferenced contexts: the learners' perspective. Journal of Distance Education, 9(1), 19-43.
Burge, E.J. & Roberts, J.M. (1998). Classrooms with a difference: Facilitating learning on the information highway. (2nd ed.). Montreal: Cheneliere/McGraw-Hill.
Campos, M & Harasim, L. (1999). Virtual-U: Results and challenges of unique field trials. Virtual University. (July/August). Available: http://horizon.unc.edu/TD/vu/1999-07.asp. (February 1, 2001).
Dehler, C. & Porras-Hernandez, L.H. (1998). Using computer mediated communication (CMC) to promote experiential learning in graduate studies. Educational Technology, 38(3), 52-55.
Eastmond, D.V. (1996). Computer-mediated communications. In T. Plomp & D. P. Ely (Eds.). International encyclopedia of educational technology. (2nd ed.). (pp. 382-388). Oxford: Pergamon.
Flynn, J.L. (1992). Cooperative learning and Gagne's events of instruction: A syncretic view. Educational Technology, 32(Oct.), 53-60.
Gilbert, L. & Moore, D.R. (1998). Building interactivity into Web courses: Tools for social and instructional interaction. Educational Technology, 38(3), 29-35.
Harasim, L. (1987). Teaching and learning on-line: issues in computer-mediated graduate courses. Canadian Journal of Educational Communication, 16(2), 117-135.
Harasim, L. (1996). Computer networking for education. In T. Plomp & D. P. Ely (Eds.). International encyclopedia of educational technology. (2nd ed.). (pp. 484-489). Oxford: Pergamon.
Harasim, L. (1999). A framework for online learning: the Virtual-U. [online]. Available: http://www.telelearn.ca/g_access/news/r9044.pdf. (February 1, 2001).
Howell-Richardson, C. & Mellar, H. (1996). A methodology for the analysis of patterns of participation within computer mediated communication courses. Instructional Science, 24, 47-69.
Mason, R. & Bacsich, P. (1998). Embedding computer conferencing into university teaching. Computers and Education, 30(3/4), 249-58.
McIsaac, M.S. & Blocher, J.M. (1998). How research in distance education can affect practice. Educational Media International, 35(1), 43-47.
McIsaac, M.S. & Blocher, J.M. & Mahes, V. & Vrasidas, C. (1999). Student and teacher perceptions of interaction in online computer-mediated communication. Educational Media International, 36(2), 121-131.
Moller, L. (1998). Designing communities of learners for asynchronous distance education. Educational Technology Research and Development, 46(4), 115-122.
Morgan, C. & O'Reilly, M. (1999). Assessing open and distance learners. London: Kogan Page.
Nevo, D. (1996). Student evaluation. In T. Plomp & D. P. Ely (Eds.). International encyclopedia of educational technology. (2nd ed.). (pp. 246-251). Oxford: Pergamon.
O'Donnell, J.M. & Caffarella, R.S. (1998). Learning contracts. In: M.W. Galbraith (Ed.), Adult learning methods: A guide for effective instruction (pp. 275-301). Malabar, FL: Krieger Pub. Co.
Rada, R. (1998). Efficiency and effectiveness in computer-supported peer-peer learning. Computers and Education, 30(3/4), 137-146.
Ruberg, L.F. & Moore, D.M. & Taylor, C.D. (1996). Student participation, interaction, and regulation in a computer-mediated communication environment: A qualitative study. Journal of Educational Computing Research, 14(3), 243-268.
Simonson, M. & Smaldino, S. & Albright, M. & Zvacek, S. (2000). Teaching and learning at a distance : foundations of distance education. Upper Saddle River, N.J. : Prentice-Hall Inc.
Tolmie, A. & Boyle, J. (2000). Factors influencing the success of computer mediated communication (CMC) environments in university teaching: a review and case study. Computers & Education, 34(2), 119-140.
Trentin, G. (1998). Computer conferencing systems as seen by a designer of online courses. Educational technology, 38(3), 36-43.
Veerman, A.L. & Andriessen, J.E. & Kanselaar, G. (2000). Learning through synchronous electronic discussion. Computers & Education, 34(3/4), 269-290.