Contact
Diverse opinions about cognitive process models
659
post-template-default,single,single-post,postid-659,single-format-standard,bridge-core-2.4.3,ajax_fade,page_not_loaded,,qode_grid_1300,side_area_uncovered_from_content,overlapping_content,qode-content-sidebar-responsive,qode-theme-ver-22.8,qode-theme-bridge,disabled_footer_top,wpb-js-composer js-comp-ver-6.3.0,vc_responsive,elementor-default,elementor-kit-3194

Diverse opinions about cognitive process models

25 Dec Diverse opinions about cognitive process models

Posted at 00:16h in Inter-rater reliability, Results by

The survey Process Models in Judgment and Decision Making is an expert survey we conducted to find out about researchers’ opinion about the term “process model” and whether it applies to several models in the field. The main objective was to find out whether researchers’ agree or disagree about which models ought to be called “process model”.

We contacted 118 authors of well-known decision-making models per email and decision scientists via mailing lists (SJDM, largest decision science list, www.sjdm.org; list of the European Group of Process Tracing Studies, www.egproc.org).

Respondents

The following results will appear in detail in our manuscript

Jarecki, J. B., Tan, J. H., & Jenny, M. A. (2015). What is a cognitive process model? A disambiguation. Working Paper [SSRN]

63 scholars replied (we excluded two master-students and one person for non-serious answering). The majority were at the level of a professor, followed by researchers (=post-docs, senior researchers or lecturers), and 10 doctoral students. Most had experience with teaching methods classes and they were rather familiar with the models, e.g. professors rated between 16 and 115 of the 116 models (mw=50).

fig1-demogr

Agreement

We expect high inter-rater agreement between respondents if they agreed on which models are process models. Overall agreement measured by Fleiss-Cuzick’s kappa (Fleiss & Cuzick, 1979) yields κ = .26, which must be considered low — values above .60 indicate “good” agreement. Agreement is low as well when computing the measure per seniority level (for professors, researchers, or doctoral students separately, κ reaches maximally .32 < .60).

These two plots illustrate the fact. If scientists shared their opinions whether models are process models, we expect to see a distribution of opinions over all models that takes the form of a step function.

fig6.1_isPMexp

However, we see this distribution

fig6.2_isPMsurv

Note that for models which the majority regards as not process models there seems to be greater agreement (100 % in one case, the leftmost bar) than for models which seem to be considered process models by the majority (75 % agreement, the rightmost bar).

I am aware that it is possible to split these plots up by seniority — however I will refrain from showing more because these plots show percent agreement, which is an inadequate measure of rater agreement. Further the plots ignore the fact that some models are rated by many whereas other models are rated by few. The two plots above serve as mere illustration of the distribution of opinions.

If you have questions, drop me a mail.

Print page