Category : Inter-rater reliability

Diverse opinions about cognitive process models

The survey Process Models in Judgment and Decision Making is an expert survey we conducted to find out about researchers’ opinion about the term “process model” and whether it applies to several models in the field. The main objective was to find out whether researchers’ agree or disagree about which models ought to be called “process model”. We contacted 118 authors of well-known decision-making models per email and decision scientists via mailing lists (SJDM, largest decision science list, www.sjdm.org; list […]

Inter-rater reliability for unequal raters per item: Fleiss-Cuzick’s kappa

Inter rater reliability If you want to obtain inter-rater reliability measures for dichotomous ratings, by more than two raters, but not all raters rated all items, Fleiss and Cuzick (1979) will be the referece you’ll find. For example, we asked researchers which model they consider a process model (dichotomous rating), and we asked about 60 researchers (more than two rater), of whom not everyone was familiar with every model (not all raters rated all items). They proposed a measure between […]