Reliability in Evaluating Portfolios for Higher Education Teacher Accreditation.(Statistical Data Included)

Abstract

Data are reported regarding the reliability of evaluations of portfolios constructed for the purpose of gaining accreditation as a teacher in UK higher education (HE). Whilst the reliability of the evaluations lies towards the upper end of the spectrum of reliabilities found in U.S. studies, analysis of the data points to ways in which reliability might further be improved. The combination of evaluations into a single overall evaluation is shown to raise questions about the way in which the cumulation of performance scores should be undertaken.

An Approach to Authentic Assessment

Portfolios are now widely used for a variety of purposes relating to teacher development and appraisal, including self-presentation for promotion or tenure, personal accreditation as a teacher (Seldin, 1997), and accountability to super-ordinate authorities (such as school systems). Portfolios are also used for development and assessment in other professions including social work and nursing (Taylor, et al., 1999). Portfolios are seen as 'authentic,' in that they refer to collections of performances in naturalistic settings (though there is variation in expectation as to what should be included in a portfolio (Stecher, 1998; Simon and Forgette-Giroux, 2000). For that reason they are held to have advantages over other forms of assessment. However, Herman, et al., (1993, p.202) observed that "the measurement quality of portfolios is largely uncharted territory": there was a flurry of work on the reliability of portfolio assessments in the mid-1990s, but this work now seems substantially to have ceased without having advanced assessment methodology to any great extent.

Portfolios in the Accreditation of HE Teachers in the UK

The Open University (OU) in the UK runs courses for teachers in higher education, courses which lead to accreditation as teachers and to the award of a post-graduate qualification. The courses are accredited by the UK Staff and Educational Development Association (SEDA, 2000) and the Institute for Learning and Teaching (ILT 2000). SEDA specifies the seven or eight (depending on the course) outcomes to be achieved, and the six principles or values which must demonstrably underpin the achievement of these outcomes (see Appendix 1). The teacher is required to present a portfolio of evidence in support of his/her claim for accreditation. The portfolio contains two distinct types of material: evidence (lesson plans, graded student work and the like), and claims in which the course participant argues that s/he has met the outcomes. A portfolio for the course that forms the object of this study requires a total of seventy- five assessment judgements. Twenty of these are technicalities such as word and page count. Forty-six require academic judgement on whether particular elements are demonstrated in the portfolios. The overall judgement on each of seven outcomes is obtained in major part by combining judgement on the elements of that outcome, with limited discretion left to the assessor and scope for marginal failure of one element in an outcome to be condoned. To be judged to have passed the course, the teacher has to achieve a pass on all outcomes.

This large number of elements of assessment results in part from the two-dimensional matrix of seven outcomes, the attainment of each of which may need to be underpinned by up to six principles or values. The number of elements is greater than forty-two because each of the outcomes is subdivided into components. For example, the needs to reflect on one's practice, identify one's development needs and plan one's continuing professional development are all subsumed under Outcome 7.