We conducted preregistered replications of 28 classic and contemporary published findings, with protocols that were peer reviewed in advance, to examine variation in effect magnitudes across samples and settings. Each protocol was administered to approximately half of 125 samples that comprised 15,305 participants from 36 countries and territories. Using the conventional criterion of statistical significance (p < .05), we found that 15 (54%) of the replications provided evidence of a statistically significant effect in the same direction as the original finding. With a strict significance criterion (p < .0001), 14 (50%) of the replications still provided such evidence, a reflection of the extremely high-powered design. Seven (25%) of the replications yielded effect sizes larger than the original ones, and 21 (75%) yielded effect sizes smaller than the original ones. The median comparable Cohen’s ds were 0.60 for the original findings and 0.15 for the replications. The effect sizes were small (< 0.20) in 16 of the replications (57%), and 9 effects (32%) were in the direction opposite the direction of the original effect. Across settings, the Q statistic indicated significant heterogeneity in 11 (39%) of the replication effects, and most of those were among the findings with the largest overall effect sizes; only 1 effect that was near zero in the aggregate showed significant heterogeneity according to this measure. Only 1 effect had a tau value greater than .20, an indication of moderate heterogeneity. Eight others had tau values near or slightly above .10, an indication of slight heterogeneity. Moderation tests indicated that very little heterogeneity was attributable to the order in which the tasks were performed or whether the tasks were administered in lab versus online. Exploratory comparisons revealed little heterogeneity between Western, educated, industrialized, rich, and democratic (WEIRD) cultures and less WEIRD cultures (i.e., cultures with relatively high
Background: Order and amount of information influence patients' risk perceptions, but most studies have evaluated patients' reactions to written materials. The objective of this study was to examine the effect of 4 communication strategies, varying in their order and/or amount of information, on judgments related to an audible description of a new medication and among patients who varied in subjective numeracy. Methods: We created 5 versions of a hypothetical scenario describing a new medication. The versions were composed to elucidate whether order and/or amount of the information describing benefits and adverse events influenced how subjects valued a new medication. After listening to a randomly assigned version, perceived medication value was measured by asking subjects to choose one of the following statements: the risks outweigh the benefits, the risks and benefits are equally balanced, or the benefits outweigh the risks. Results: Of the 432 patients contacted, 389 participated in the study. Listening to a brief description of benefits followed by an extended description of adverse events resulted in a greater likelihood of perceiving that the medication's benefits outweighed the risks compared with 1) presenting the extended adverse events description before the benefits, 2) giving a greater amount of information related to benefits, and 3) sandwiching the adverse events between benefits. These associations were only observed among subjects with average or higher subjective numeracy. Conclusion: If confirmed in future studies, our results suggest that, for patients with average or better subjective numeracy, perceived medication value is highest when a brief presentation of benefits is followed by an extended description of adverse events.
Kostopoulou O, Sirota M, Round T, Samaranayaka S, and Delaney BC
Medical Decision Making: An International Journal Of The Society For Medical Decision Making [Med Decis Making] 2017 Jan; Vol. 37 (1), pp. 9-16. Date of Electronic Publication: 2016 Apr 25.
Adult, Female, Humans, Information Seeking Behavior, Male, Middle Aged, Referral and Consultation, United Kingdom, Clinical Decision-Making, Neoplasms diagnosis, Physicians, Family psychology, and Thinking
MDM Policy & Practice [MDM Policy Pract] 2016 Sep 19; Vol. 1 (1), pp. 2381468316669361. Date of Electronic Publication: 20160919.
Background: Past research shows that people believe psychologically caused mental disorders are helped by different treatments than biologically caused mental disorders. However, it is unknown how people think about treatment when limited information is known to identify the disorder. Objective: Our objective was to explore how laypeople judged the helpfulness of treatments when a limited set of mental health symptoms is presented. Method: Across four experiments, Mechanical Turk and college undergraduate participants (N = 331) read descriptions displaying sets of three mental health symptoms and rated how helpful pharmaceuticals, counseling, or alternative medicine would be on a 0 (not at all helpful) to 100 (completely helpful) scale. We measured judgments for perceived mental and medical symptoms (Experiment 1) and how judgments were influenced by symptom severity (Experiment 2), duration (Experiment 3), and if alternative medicine and conventional treatments were used in conjunction (Experiment 4). Results: Perceived mental symptoms were rated as helped by counseling, while perceived medical symptoms were rated as helped by medication. Alternative medicine was never rated as extremely helpful. For example, in Experiment 1, counseling (mean [M] = 80.1) was rated more helpful than pharmaceuticals (M = 50.5; P < 0.001) or alternative medicine (M = 45.1; P < 0.001) for mental symptoms, and pharmaceuticals (M = 62.6) was rated more helpful than counseling (M = 36.1; P < 0.001) or alternative medicine (M = 47.5; P < 0.001) for medical symptoms. This pattern held regardless of severity, duration, or the adjunct use of alternative medicine. Limitations: We employed a general population sample and measured hypothetical treatment judgments. Conclusions: Mental health symptoms viewed as problems of the mind are thought to need different treatment than mental health symptoms seen as problems of the body.
Kostopoulou, Olga, Sirota, Miroslav, Round, Thomas, Samaranayaka, Shyamalee, and Delaney, Brendan C.
Kostopoulou, O, Sirota, M, Round, T, Samaranayaka, S & Delaney, B C 2016, ' The role of physicians' first impressions in the diagnosis of possible cancers without alarm symptoms ' Medical Decision Making, vol. 37, no. 1, pp. 9-16 . DOI: 10.1177/0272989X16644563
cognitive psychology, Health Policy, heuristics and biases, Original Articles, provider decision making, and family medicine
Background. First impressions are thought to exert a disproportionate influence on subsequent judgments; however, their role in medical diagnosis has not been systematically studied. We aimed to elicit and measure the association between first impressions and subsequent diagnoses in common presentations with subtle indications of cancer. Methods. Ninety UK family physicians conducted interactive simulated consultations online, while on the phone with a researcher. They saw 6 patient cases, 3 of which could be cancers. Each cancer case included 2 consultations, whereby each patient consulted again with nonimproving and some new symptoms. After reading an introduction (patient description and presenting problem), physicians could request more information, which the researcher displayed online. In 2 of the possible cancers, physicians thought aloud. Two raters coded independently the physicians' first utterances (after reading the introduction but before requesting more information) as either acknowledging the possibility of cancer or not. We measured the association of these first impressions with the final diagnoses and management decisions. Results. The raters coded 297 verbalizations with high interrater agreement (Kappa = 0.89). When the possibility of cancer was initially verbalized, the odds of subsequently diagnosing it were on average 5 times higher (odds ratio 4.90 [95% CI 2.72 to 8.84], P < 0.001), while the odds of appropriate referral doubled (OR 1.98 [1.10 to 3.57], P = 0.002). The number of cancer-related questions physicians asked mediated the relationship between first impressions and subsequent diagnosis, explaining 29% of the total effect. Conclusion. We measured a strong association between family physicians' first diagnostic impressions and subsequent diagnoses and decisions. We suggest that interventions to influence and support the diagnostic process should target its early stage of hypothesis generation.
Applied Psychological Measurement [Appl Psychol Meas] 2015 Jan; Vol. 39 (1), pp. 16-30. Date of Electronic Publication: 2014 Oct 20.
Heterogeneous item content is prevalent on psychological and educational tests that measure global traits or competencies, such as general intelligence and achievement tests. In such tests, the domain is broadly defined, so as to include many attributes and skills. Items often vary substantially in both the type and the number of attributes or skills that are involved in item solving. A hierarchical organization is often necessary to accommodate the heterogeneity of the test domain. For example, the mathematical achievement tests that are routinely administered in all U.S. states at the end of the school year typically have this hierarchical structure. The multicomponent latent trait model for diagnosis (MLTM-D) was developed for application to heterogeneous tests. MLTM-D is a confirmatory model that permits diagnosis at broad and more specific attribute or skill levels. In the current study, MLTM-D is applied to diagnose mastery at both the broad and specific skill levels in middle school mathematics. MLTM-D is presented, and methods involved in application are described.