In this article I describe and reflect on the redesign and implementation of a peer review process for a relatively large class, focusing primarily on student writing. I will describe the process of implementation in what I would like to conceptualize as a de-centered classroom, challenging the assumption that a large group impedes instructors from enabling highly participatory learning, and proposing, instead, that a careful use of widely available information-technology tools can play a positive role in the creation of such an environment.

The ideal setting of a liberal arts college, with small classes in a seminar-like environment, is far from the reality of large groups in big classrooms that most instructors face in large higher education institutions. In my case I was expected to teach in an auditorium, a place designed for me to speak and for the students to listen, thus creating a passive learning environment that is less than desirable pedagogically. This physical venue and the relatively large number of students created a major logistical challenge, making unpractical even the idea of breaking the larger group into smaller groups for structured and collaborative work.¹ As most instructors teaching to large crowds have experienced, the only viable alternative is to structure the course around regularly scheduled and engaging lectures and presentations, leaning on instructional aids to provide the more intimate opportunity for interaction that sections offer, but still leaving the student on the bench as a passive player. In terms of assessing and evaluating the work of a large group, the temptation of a passive model is even greater: what is wrong with multiple-choice exams “automatically” graded by a computer, and the assignment of a couple of essays to be summarily graded with the help of class readers, as a means to render a grade for the roster at the end of the academic term?

Drawing on what I have learned from colleagues about teaching, learning and assessment, and enthusiastic about the enfranchising potential of the right use of the right technological tools, I decided to challenge myself to imagine a different classroom environment for a large group. The fundamental pedagogical drive of the changes that I intended to enact relates to the work of Paolo Freire. His work centers education in dialogue, as the strategy that is “explicitly critical and aimed at action,” where students re-define their role in the classroom to become subjects of their own destiny.² The challenge is to generate a pedagogical praxis in tune with a vision of a classroom centered on learning, as opposed to teaching exclusively by lecturing, and where the key question is not so much what the instructor can teach but rather what the students need to learn and can learn by participating in the process.

The idea of peer evaluation to improve student involvement is hardly an innovation in the field.³ However, to organize an efficient participatory system and to gather information that could lead to a reliable evaluation of the potential benefits, I needed to start with a few critical questions: What difference does it really make to focus on process? Is there a measurable and positive impact of peer review in the preparation of reviewers and reviewees in their professional lives (as writers)? How would I go about creating such a tool, making the load reasonable and manageable?

I explore the first question not only in terms of a perceived or measured improvement in student performance, but also in terms of a shift in the students’ understanding of their role in the classroom. I will also describe how, relying on technology information tools (a database and a Web site), I organized this exercise so that a measurable impact was produced and critically reviewed. I also provide a discussion about the tools that I used to implement this exercise and the tools that I am currently using in my practice. This is the story in a nutshell.

Focus on Process: The Importance of Peer Review
A few years ago I was asked by my department to teach the Major Proseminar, a course designed to introduce upper-level students in the Social and Behavioral Sciences Program to central issues of theory and method. The class, varying each semester from sixty to eighty students, was a large one by the standards of our small campus. As this was my first experience teaching in front of a large group for my program, I was concerned with being able to engage students in a process of learning theory and method without missing the level of individualized attention that an instructor can provide in much smaller classes of twenty to twenty-five students.

Of all the different aspects of teaching a course, the one that is often of most concern to instructors is the writing component. In this particular case, students were expected to write two essays, one on the concepts of culture and equity, and one on theory and method in the social and behavioral sciences. While the delivery of a solid midterm or final essay is often perceived by the students as the key to faring well in a given course, I decided to explore shifting the emphasis to the process by assigning a seemingly disproportionate weight of points to the participation of students in the peer review exercise. The Peer Review, in itself a relatively minor assignment compared with the crafting of the essays, was worth a full one-third of the grade of each essay. By giving specific weight to the process I was trying to challenge the linear approach I had used in the past, where the instructor provides a prompt for the essay and then the student somehow produces an outcome that reflects their level of expertise in the subject. In a dialogical process, such as the one I set out to implement in this course, the instructor and the students work in a collaborative environment to understand and carry out the process. Consequently, the final essay becomes simply a byproduct of this process. The participation in the process and not the product is, therefore, re-centered as the key pedagogical event, and as the means by which the student achieves competency.

To carefully prepare the process of peer review I first thought about time, ensuring there was enough of it for students to prepare their essays, exchange materials, review materials and submit the reviews to their peers. I also factored in time to rework the essays into a final version and allowed space to reflect on the entire process as a group. I also carefully worked out the materials and sessions necessary to explain the criteria that students would use to evaluate the papers. Making the criteria visible and accessible through examples and practice became a critical requirement for the success of the peer review. The instructions for the assignment were delivered and discussed in small groups during special sessions. I wanted to have at least two reviewers for each paper to add to my own review, thus simulating what happens in a professional review process.

This process of review seems fairly doable with the maximum recommended of twenty-two students per group, but how would I do this with sixty to eighty students? Here is where technology can play a liberating role. Using readily available tools I organized the submission of the papers and their distribution to reviewers, as well as the collection and redistribution of reviews back to the authors. For the collection of papers I utilized a message board, which allows students to upload their essays as attachments in a confined virtual space linked to the course and not open to the public at large.⁴ (I was not aiming at a blind review so having the names in the essays was not an issue.) I am confident that this can be replicated in most university settings today using similar tools.

In this particular course, students were required to write two papers. The first of the two essays focused on the application of theory (e.g. Interactionist Theory, Personality Theory, Conflict Theory) to the analysis of any given social, psychological, cultural, political and/or economic phenomena. I made it very clear to the students that the dead-line for this first draft was as important as the deadline for the final version. In a more linear version of the process the instructor usually provides the prompt for the essay. The students engage in the process of writing very much on their own, based on what they have been able to grasp during class time. The role of the instructor is then to judge the extent to which the student was able to elaborate a good essay, following instructions and matching the expected standards for the class. In an alternative dialogical model, there is an intense interaction between the instructor, peer reviewers and the student in the definition and crafting of the assignment. In this environment the quality of the participation in the process rather than the final product is the focal point. A solid essay has the potential to show that a student has mastered the art of writing a paper. This is what I remember doing as a college student. Why bother with all this emphasis on process? It seems to me that there are two critical advantages: the first one is that more solid essays will be produced as a result of such emphasis, as the learning will become enduring for the student. Secondly, the students will be able to transfer the skills learned to other contexts, thus expanding dramatically the relevance of the experience well beyond the specific course. How to write a great essay becomes visible, achievable and valuable as a process for many more than just a handful of students.

Going back to the process of review, the students accessed electronic copies of the essays for review. This was a most important advantage for me as an instructor. I did not have to collect, photocopy and redistribute the essays. I just monitored the process of up-loading materials to the message board and then the process of submission of reviews to the same message board. Each paper was reviewed by two peers, and by the instructor, so that authors received a total of three evaluations of their work.

I asked the students as reviewers to determine whether or not an essay met the stipulated criteria for structure, style and content. The definitions of style, structure and content were discussed in advance and made available to reviewers. If a reviewer decided to mark a paper as not having met certain criteria, he or she was required to explain the decision and to provide feedback to the author. This led the reviewer to carefully consider the criteria. For example, if a reviewer marked “content” as not met, the reviewer would be expected to say, for example, “the conclusion is missing” or “the conclusion includes an element that was not substantiated in the paper. . . .” The idea is for the reviewer to identify trouble spots and provide guidance to the author.

The insert below is an example of the comments from a student review of another student’s work. The reviewer identifies eight different aspects in the paper that have had not been sufficiently developed by the author. After indicating with a “No” that a given aspect failed to meet the standard, the student provides comments:

Comments

Flow: You don't need to define theory in general so specifically, just talk about your own theory. Complies with assignment: All the elements are not present; you need to go into more detail for the 3rd part of the deliverable, because I can't tell which of the two options you were going for. Personality: There are very few of your own views; you can apply these to part three or part four. Sources: You should cite the definitions for the different types of theory. Also, you don't have notes in the text for all the items in the bibliography. Critical Understanding: No alternative views are presented, this and more depth on the central view is needed. Evidence: You need more evidence or statistics to back up the main assertions. Perspective: Talk more about child trauma as a whole in order to provide perspective. Conclusion: Summarize the main points and provide your own analysis. Obviously, you're not done yet. Good luck finishing it up.

Because I was interested in documenting the process, I created an online tool (see image 1) where each reviewer posted the evaluation and submitted comments. This is a step that other instructors do not have to organize for their peer reviews but one that I wanted to create so that I could quantify and monitor the process. Reviewers can simply submit their reviews in a Word Processor template back to the same message board. In my case I had the students send this information to the database; it was then available for review to each individual student.

The authors accessed the reviews and considered the comments in preparation for the final version to be turned in three to four weeks after the initial draft. In the time elapsed between the review and the final deadline students were encouraged to talk to the instructor or the teaching assistant for additional guidance, and to talk among themselves.

Upon review of the first draft of the essay, I assigned a temporary grade to the paper, making it clear to the student that such would be the grade I would have assigned to the paper if presented to me as a final version. The comments from students about the experience of an interim grade confirmed my belief that it helped as some sort of a quantitative benchmark, evidencing how much room for improvement was left for that paper. At the end of the semester, my job evaluating the final drafts of the paper was much easier. I had to compare the two versions (draft and final) and make a judgment regarding the extent to which the author considered the comments and tackled the issues that were identified as problematic in the process of peer review. Once again I resorted to existing technology to make this task easy and even enjoyable. Most word processors such as OpenOffice or MS Word, for example, have a feature that enables the user to compare different versions of the same document. This makes very it easy to isolate the changes and indirectly observe the decisions the authors have made in terms of revising their drafts.

Although the peer review can take place with simple paper and pen in the classroom, I believe there is a critical advantage in the use of electronic means that made the design and implementation of the system a worthwhile investment. The database stores a numeric and textual record of the interaction of peer reviewers with the text and the process. This information can be quantitatively analyzed and compared with other information of a qualitative nature gathered during the semester.

Comparative Analysis: What difference does it make?
To address the first and critical question, what difference does it make? I kept a careful record of interactions related to the process of peer review. I believe that the gathering of information through peer review renders two crucial opportunities; one for the students to understand in a practical way how to craft a high quality narrative in their fields, and secondly, an opportunity for the instructor to identify strengths and weaknesses in the process and its instruments (assignments, rubrics, and categories for evaluation). While a general understanding of issues and progress for the group and individual students is readily apparent, I was able to explore the data in more detail. The data allowed me to compare differences in performance of individual students and the group in the two exercises during the semester and to compare how the students evaluated their peers with my own evaluation of the same work.

I noticed, for example, that the cumulative mean for structure was slightly higher for deliverable 2 (theory and method) than for deliverable 1, which could be construed as the result of the students applying the experience acquired during the crafting of the first deliverable to the second. What I found to be reassuring was that in all three areas of structure, content and style (Tables 1, 2, and 3), the means of the scores of the second deliverable are consistently higher. This, again, can be construed as evidence that there was transference of skills from one assignment to the next.

Table 1
Mean for Structure: Breakdown by Deliverable

Structure (10)

Deliverable	Mean	N	Std. Deviation
1 2 Total	6.72 7.33 7.02	57 55 2.435	2.698 2.435 2.578

Table 2
Mean for Content: Breakdown by Deliverable

Content (10)

Deliverable	Mean	N	Std. Deviation
1 2 Total	5.63 6.15 5.88	57 55 112	2.769 2.059 2.449

Table 3
Mean for Style: Breakdown by Deliverable

Style (5)

Deliverable	Mean	N	Std. Deviation
1 2 Total	2.96 3.31 3.13	57 55 112	1.414 1.069 1.263

A second source of valuable information came from comparing the individual cumulative scores by specific criteria. The criteria utilized to evaluate structure included guiding structure, flow, thesis, paragraph by topic, transitions, sentence structure, grammar and spelling, format, if they have been proofread and finally, timeliness of submission.  For example. Table 4 shows that thesis was an aspect of the structure of the papers that reviewers tended to find deficient. As a result, I reviewed how instructions and support were provided in the class with regard to this specific aspect of the assignment.

Table 4. Structure

Specific Criteria	Score
Guiding structure:	.75
Flow:	.71
Thesis:	.54
paragraph by topic:	.79
transitions:	.65
sentence structure:	.84
grammar/spelling:	.73
format:	.54
proofread:	.51
timeliness:	.95

The specific criteria for content included: compliance with the assignment, presence of a central idea, purpose, position, critical understanding, idea development, evidence, comparative nature, involvement of author and the conclusion. In the majority of the draft essays, students scored lower in this area than in the other two areas of structure and style.

For the first essay, recommendations for improvement in style accounted for 43% of the entries. Specifically, the reviewers recommended more attention to sources and definitions (18 and 19 entries respectively). Under the rubric of content, development was the most mentioned subtopic at 22 entries. Reviewers also emphasized the need for attention to the main thesis in the essays (14 times). A solid 82% of the qualitative entries for the essay on Culture and Equity are one or more of the four above-listed issues.

Current Practice
As semesters go by, I find myself using a less complicated version of the tool that I describe here, but I remain fully invested in the concept and practice of early assessment with the support of information technologies. Regardless of the topic that I am teaching, I make a systematic effort throughout the semester to discuss with the students the design and activities of the class as a learning experience. In most cases I find that the initial reaction from students is surprise: why would I spend so much time explaining how I intend to teach the class? As the reflective sessions continue, I often find the students understanding the active role they will be playing and in the end shifting to talk about learning and not teaching. Peer review is now a constant in all of my different courses, framed by a very explicit and thorough discussion on the expected outcomes, criteria and standards for the course. I find most students engage in peer review exercises with a sense of responsibility; furthermore, it is common for them to express that, by reviewing their peers’ work, they have an opportunity to better understand the strengths and weaknesses of their own work. An important change in the process is that I have resorted to less detailed schemes for evaluation. For example, I have changed the sets of criteria to make them more narrowly focused on fewer items, and this seems to work better. The aid of technology continues to be a critical component of this practice. I still use digital files as the only format exchanged in class and as new software packages are offered this seems to become easier. In most of the courses that I teach now, instead of using a database as described in these pages, I use a simple version of a blog or message board that allows for the recording of the responses and the exchange of files with more ease. For example, I use now Moodle, an open-source software that has a tremendous potential for this type of exercise. 

Conclusions
The exercise and use of technology to enhance learning as a student-centered pedagogical practice is driven by four fundamental aspirations: 1. to focus on learning, as opposed to teaching, and to generate the best strategy for transmitting, integrating, interpreting and extending knowledge; 2. to diversify strategies, as the core of an instructional praxis that is based on assets and responsive to multiple positions of students; 3. to create opportunities for critical thought and inquiry as an aspiration to provide an environment that is both rich and challenging, and to facilitate the process by which the students can take advantage of opportunities for learning; and 4. on a more practical stance, the aspiration for a meaningful and efficient use of technology, applying it in a way that makes a measurable difference in the student achievement of skills and competencies.  What difference does student-centered focus in practice really make? It enables students to own the process of learning as it liberates the instructor from the traditional authoritative role of gatekeeper. Shifting attention from the final essay to the process of crafting and reviewing peer work also created an observable difference in the quality of the participation of the students.

Adding a layer of peer review to the process of evaluation of the course has the potential to create a tremendous amount of work, an issue particularly worth considering when teaching a relatively large class. To handle this potentially cumbersome and time-consuming process I resorted to electronic tools and solutions that I already had at hand as part of the information technology available on campus, and I did some development work of my own creating a database-driven application to collect and redistribute documents and reviews. In retrospect I believe that while the system did work seamlessly as a single Web page space embedded in my course Web site, I could have used other means such as a message board in Blackboard, Moodle or any other instructional platform with little or no development needed on my part. This is to say that instructors in most universities today already have the resources and tools to implement a process similar to the one I describe here.

In shifting to this student-centered focus, my role as instructor became that of enabling and legitimizing a process that was driven by the students, rather than that of the sole judge of the quality and integrity of the work. The peer review process also brings an additional challenge to both writers and reviewers as multiple reviewers, three per paper, provide feedback to the same author. Handling diverging advice would remind writers that at the end of the day they have to make the decision to take or not recommendations from the different reviewers. 

As a corollary I want to say that I am not sure that I have offered the reader a story of a wonderful and edifying success. What I know I wanted to share is the experience of a process that displaced myself from the role of a traditional instructor into one of a facilitator. I also offered a candid reflection on the implications, risks and advantages of such decentering of my own praxis based on the central pedagogical assumption that re-centering the classroom from teaching to learning is not only possible with relatively large groups, but critical for substantial and enduring achievement of skills to take place.

Notes
I would like to show my sincere appreciation to Stephanie Watson, an outstanding CSUMB student I have had the opportunity to work with. Stephanie has helped me plan and carry out the teaching and assessment analyzed in this project. I owe gratitude to my VKP colleagues at CSUMB Rina Benmayor, Renee Curry, Christine Sleeter, Gilbert Neri, Cecilia O’Leary, and Renee Perry for their extraordinary support. In addition to my gratitude to the VKP group at CSUMB and the VKP National, I am indebted intellectually to Amy Driscoll who graciously transformed my understanding of the importance and nuances of teaching, learning, and assessment.

About VKP: In all, more than seventy faculty from twenty-two institutions participated in the Visible Knowledge Project over five years. Participating campuses included five research universities (Vanderbilt University, the University of Alabama, Georgetown University, the University of Southern California, Washington State University, and the Massachusetts Institute of Technology), four comprehensive public universities (Pennsylvania’s Millersville University, California State University (CSU)--Monterey Bay, CSU Sacramento, Ohio’s Youngstown State University, and participants from several four-year colleges in the City University of New York system, including City College, Lehman, and Baruch), and three community colleges (two from CUNY--Borough of Manhattan Community College and LaGuardia Community College, and California’s Cerritos College). In addition to campus-based teams, a number of independent scholars participated from a half dozen other institutions, such as Arizona State and Lehigh University.  The project began in June 2000 and concluded in October 2005.  We engaged in several methods for online collaboration to supplement our annual institutes, including an adaptation of the digital poster-tool created by Knowledge Media Lab (Carnegie Foundation), asynchronous discussion, and web-conferencing.  The VKP galleries and archives (http://crossroads.georgetown.edu/vkp/ ) provide a wealth of background information, including lists of participants, regular newsletters, and reports and essays by participants, as well as a number of related resources and meta-analyses.  For this article, the author gratefully acknowledges the students whose work is cited here. All students whose work is included have granted the author permission to use the material.

1. V. Wilson, Does Small Really Make a Difference? A Review of the Literature on the Effects of Class Size on Teaching Practice and Pupils' Behaviour and Attainment. (Edinburgh, Scotland: Scottish Council for Research in Education, 2002). [return to text]
2. D. Goulet and P. Freire, A New Moral Order: Studies in Development Ethics and Liberation Theology (Maryknoll, N.Y.: Orbis Books, 1974), viii.  For a classic account of dialogical / transformative education, see P. Freire and I. Shor, A Pedagogy for Liberation. Dialogues on transforming education (London: Macmillan, 1987) and P. Freire, Pedagogy of the Oppressed (Harmondsworth: Penguin, 1972) [return to text]
3. See A. Venables and R. Summit, Enhancing Scientific Essay Writing Using Peer Assessment. In Innovations in Education and Teaching International 40, no. 3 ( Aug 2003); D. Menchaca, “Cross-Training Teachers for Multiple Computer Spaces.” Kairos 7.3 (2002), http://english.ttu.edu/kairos/7.3/binder.html?response/techteach/index.htm; R. Johnson, The Next Frontier of the Student-Centered Classroom: Teaching Students To Recognize Quality Writing through the Use of Peer Evaluation (Iowa: ERIC, 2001). [return to text]
4. I could have used email or a blog for the same purpose. The idea is to have individual authors upload their papers to a centralized location. The reviewers can reach the materials and download on their own. [return to text]

Attachment	Size
Guiterrez_Focusing_on_Process.pdf	1.18 MB