Back
Full-text
Journal article

The Quality of Response Time Data Inference: A Blinded, Collaborative Assessment of the Validity of Cognitive Models.

  • Dutilh G University of Basel, Basel, Switzerland. gilles.dutilh@gmail.com.
  • Annis J Vanderbilt University, Nashville, USA.
  • Brown SD University of Newcastle, Callaghan, Australia.
  • Cassey P University of Newcastle, Callaghan, Australia.
  • Evans NJ University of Newcastle, Callaghan, Australia.
  • Grasman RPPP University of Amsterdam, Amsterdam, Netherlands.
  • Hawkins GE University of Newcastle, Callaghan, Australia.
  • Heathcote A University of Tasmania, Hobart, Australia.
  • Holmes WR Vanderbilt University, Nashville, USA.
  • Krypotos AM Utrecht University, Utrecht, the Netherlands.
  • Kupitz CN University of California, Irvine, USA.
  • Leite FP Ohio State University, Columbus, USA.
  • Lerche V University of Heidelberg, Heidelberg, Germany.
  • Lin YS University of Tasmania, Hobart, Australia.
  • Logan GD Vanderbilt University, Nashville, USA.
  • Palmeri TJ Vanderbilt University, Nashville, USA.
  • Starns JJ University of Massachusetts Amherst, Amherst, USA.
  • Trueblood JS Vanderbilt University, Nashville, USA.
  • van Maanen L University of Amsterdam, Amsterdam, Netherlands.
  • van Ravenzwaaij D University Groningen, Groningen, Netherlands.
  • Vandekerckhove J University of California, Irvine, USA.
  • Visser I University of Amsterdam, Amsterdam, Netherlands.
  • Voss A University of Heidelberg, Heidelberg, Germany.
  • White CN Missouri Western State University, St Joseph, USA.
  • Wiecki TV Brown University, Providence, USA.
  • Rieskamp J University of Basel, Basel, Switzerland.
  • Donkin C University of New South Wales, Sydney, Australia.
Show more…
  • 2018-02-17
Published in:
  • Psychonomic bulletin & review. - 2019
Cognitive modeling
Diffusion Model
LBA
Response Times
Validity
Adult
Cognition
Female
Humans
Male
Models, Psychological
Models, Statistical
Reaction Time
Reproducibility of Results
Single-Blind Method
English Most data analyses rely on models. To complement statistical models, psychologists have developed cognitive models, which translate observed variables into psychologically interesting constructs. Response time models, in particular, assume that response time and accuracy are the observed expression of latent variables including 1) ease of processing, 2) response caution, 3) response bias, and 4) non-decision time. Inferences about these psychological factors, hinge upon the validity of the models' parameters. Here, we use a blinded, collaborative approach to assess the validity of such model-based inferences. Seventeen teams of researchers analyzed the same 14 data sets. In each of these two-condition data sets, we manipulated properties of participants' behavior in a two-alternative forced choice task. The contributing teams were blind to the manipulations, and had to infer what aspect of behavior was changed using their method of choice. The contributors chose to employ a variety of models, estimation methods, and inference procedures. Our results show that, although conclusions were similar across different methods, these "modeler's degrees of freedom" did affect their inferences. Interestingly, many of the simpler approaches yielded as robust and accurate inferences as the more complex methods. We recommend that, in general, cognitive models become a typical analysis tool for response time data. In particular, we argue that the simpler models and procedures are sufficient for standard experimental designs. We finish by outlining situations in which more complicated models and methods may be necessary, and discuss potential pitfalls when interpreting the output from response time models.
Language
  • English
Open access status
hybrid
Identifiers
Persistent URL
https://sonar.ch/global/documents/247008
Statistics
Document views: 40 File downloads:
  • Full-text: 0