Introduction to Many-Facet Rasch Measurement

← 8 | 9 →

Preface to the First Edition

This book grew out of times of doubt and disillusionment, times when I realized that our raters, all experienced professionals specifically trained in rating the performance of examinees on writing and speaking tasks of a high-stakes language test, were unable to reach agreement in the final scores they awarded to examinees. What first seemed to be a sporadic intrusion of inevitable human error, soon turned out to follow an undeniable, clear-cut pattern: Interrater agreement and reliability statistics revealed that ratings of the very same performance differed from one another to an extent that was totally unacceptable, considering the consequences for examinees’ study and life plans.

So, what was I to do about it? Studying the relevant literature in the field of language assessment and beyond, I quickly learned two lessons: First, rater variability of the kind observed in the context of our new language test, the TestDaF (Test of German as a Foreign Language), is a notorious problem that has always plagued human ratings. Second, at least part of the problem has a solution, and this solution builds on a Rasch measurement approach.

Having been trained in psychometrics and multivariate statistics, I was drawn to the many-facet Rasch measurement (MFRM) model advanced by Linacre (1989). It appeared to me that this model could provide the answer to the question of how to deal appropriately with the error-proneness of human ratings. Yet, it was not until October 2002, when I attended a workshop on many-facet Rasch measurement conducted by Dr. Linacre in Chicago, that I made up my mind to use this model operationally with the TestDaF writing and speaking sections. Back home in Germany, it took a while to convince those in charge of our testing program of the unique advantages offered by MFRM. But in the end I received broad support for implementing this innovative approach. It has been in place now for a number of years, and it has been working just fine.

In a sense, then, this book covers much of what I have learned about MFRM from using it on a routine basis. Hence, the book is written from an applied perspective: It introduces basic concepts, analytical procedures, and statistical methods needed in constructing proficiency measures based on human ratings of examinee performance. Each book chapter thus serves to corroborate the famous dictum that “there is nothing more practical than a good theory” (Lewin, 1951, p. 169). Though the focus of the MFRM applications presented herein is on language assessment, the basic principles readily generalize to any instance of ← 9 | 10 → rater-mediated performance assessment typically found in the broader fields of education, employment, the health sciences, and many others.

The present book emerged from an invited chapter included in the Reference Supplement to the Manual for Relating Language Examinations to the Common European Framework of Reference for Languages (CEFR; Council of Europe, 2009), Section H (Eckes, 2009a). Once more, I would like to thank the members of the Council of Europe’s Manual Authoring Group, Brian North, Sauli Takala (editor of the Reference Supplement), and Norman D. Verhelst, for helpful comments and suggestions on earlier drafts of that chapter. In addition, I received valuable feedback on the chapter from Rüdiger Grotjahn, Klaus D. Kubinger, J. Michael Linacre, and Carol M. Myford. When the chapter had evolved into this introduction, I was lucky enough to receive again feedback on the completely revised and expanded text, or parts of it, from Mike Linacre and Carol Myford. I highly appreciate their support and encouragement during my preoccupation with some of the more intricate and challenging issues of the MFRM approach. Of course, any remaining errors and shortcomings are mine.

I would also like to express my gratitude to my colleagues at the TestDaF Institute, Bochum, Germany, for many stimulating discussions concerning the design, analysis, and evaluation of writing and speaking performance assessments. Special thanks go to Achim Althaus, Director of the TestDaF Institute, who greatly supported me in striking a new path for designing a high-quality system of performance ratings. Sarah Kunert and Miriam Matenia, research assistants at the TestDaF Institute, helped with preparing the author and subject indexes.

Last, but not least, I would like to thank those persons close to me. My wife Andrea encouraged me to get the project started and provided the support to keep going. My children Laura and Miriam shared with me their experiences of rater variability at school (though they would not call it that), grumbling about Math teachers being unreasonably severe and others overly lenient, or about English teachers eagerly counting mistakes and others focusing on the skillful use of idiomatic expressions, to mention just a few examples. Looking back at my own schooldays, it is tempting to conclude that rater variability at school is one of the most reliable things in life. At the same time, this recurring variability pushed my motivation for finishing the book project to ever higher levels.

Indeed, my prime goal of writing this book was to introduce those who in some way or another employ, oversee, or evaluate rater-mediated performance assessments to the functionality and practical utility of many-facet Rasch ← 10 | 11 → measurement. To the extent that readers feel stimulated to adopt the MFRM approach in their own professional context, this goal has been achieved. So, finally, these are times of hope and confidence.

Thomas Eckes

March, 2011

← 11 | 12 →

← 12 | 13 →

Preface to the Second Edition

This second edition of my Introduction to Many-Facet Rasch Measurement is an extensive revision of the earlier book. I have been motivated by the many positive reactions from readers, and by learning that researchers and practitioners across wide-ranging fields of application are more than ever ready to address the perennial problems inherent in rater-mediated assessments building on a many-facet Rasch measurement approach.

In the present edition, I have revised and updated each chapter, expanded most chapters, and added a completely new chapter. Here I provide a brief outline of the major changes: Chapter 2 (“Rasch Measurement: The Basics”) discusses more deeply the fundamental, dichotomous Rasch model, elaborating on key terms such as latent variable, item information, and measurement invariance. Chapter 5 (“A Closer Look at the Rater Facet”) has been reorganized, dealing in a separate section with rater severity estimates and their precision; the section on rater fit statistics now includes a detailed discussion of the sample size issue. Further major amendments concern Chapter 6 (“Analyzing the Examinee Facet”), with new sections on examinee measurement results, examinee fit statistics, and criterion-specific score adjustment, and Chapter 7 (“Criteria and Scale Categories”), with new sections on criterion measurement results, manifest and latent rating scale structures, and indicators of rating scale quality. Chapter 8 (“Advanced Many-Facet Rasch Measurement”), probes more deeply into methods of confirmatory interaction analysis, focusing on approaches using dummy facets. The book now closes with Chapter 10 (“Summary and Conclusions”). In this chapter, I recapitulate relevant steps and procedures to consider when conducting a many-facet Rasch analysis, briefly discuss MFRM studies in a number of different fields of application, reconsider the implications of many-facet Rasch measurement for the validity and fairness of inferences drawn from assessment outcomes, and highlight the use of MFRM models within the context of mixed methods approaches to examining raters’ cognitive and decision-making processes.

I would like to thank my colleagues at the TestDaF Institute, University of Bochum, Germany, for keeping me attuned to the practical implications of many-facet Rasch measurement within a high-stakes assessment context. I am also grateful to research assistants Anastasia Bobukh-Weiß and Katharina Sokolski who diligently updated the author and subject indexes. My special thanks are due to Carol Myford and Mike Linacre who once more took their time to provide ← 13 | 14 → me with valuable feedback on this revised edition. As before, the errors that may remain are entirely my own.

The many-facet extension of the basic, dichotomous Rasch model ensures real progress by enhancing the validity and fairness of rater-mediated assessments across a steadily growing number of disciplines. Hopefully, this book will continue to play its part in further disseminating the rationale and practical utility of many-facet Rasch measurement.

Thomas Eckes

May, 2015

← 14 | 15 →

1.  Introduction

This chapter introduces the basic idea of many-facet Rasch measurement. Three examples of assessment procedures taken from the field of language testing illustrate the broader context of its application. In the first example, examinees respond to items of a reading comprehension test. The second example refers to a writing performance assessment, where raters evaluate the quality of essays. In the third example, raters evaluate the performance of examinees on a speaking assessment involving live interviewers. Having discussed key concepts such as facets and rater-mediated assessment, the general steps involved in adopting a many-facet Rasch measurement approach are pointed out. The chapter concludes with an outline of the book’s purpose and a brief overview of the chapters to come.

1.1    Facets of measurement