HARKing (hypothesizing after the results are known) is an acronym coined by social psychologist Norbert Kerr[1] that refers to the questionable research practice of "presenting a post hoc hypothesis in the introduction of a research report as if it were an a priori hypothesis".[1][2] Hence, a key characteristic of HARKing is that post hoc hypothesizing is falsely portrayed as a priori hypothesizing.[3] HARKing may occur when a researcher tests an a priori hypothesis but then omits that hypothesis from their research report after they find out the results of their test. Post hoc analysis or post hoc theorizing then may lead to a post hoc hypothesis.
Types
editSeveral types of HARKing have been distinguished, including:
- THARKing
- Transparently hypothesizing after the results are known, rather than the secretive, undisclosed, HARKing that was first proposed by Kerr. In this case, researchers openly declare that they developed their hypotheses after they observed their research results.[4]
- CHARKing (or Pure HARKing)
- CHARKing[5] or "pure HARKing"[1] refers to the practice of constructing new hypotheses after the results are known and presenting them as a priori hypotheses.[1][5] CHARKing is often regarded as the prototypical form of HARKing.
- RHARKing
- RHARKing refers to retrieving old hypotheses from the existing literature after the results are known and presenting them as a priori hypotheses[5] Note that RHARKed hypotheses can be considered to be a priori hypotheses in the sense that they were developed and published prior to knowledge of the current research results.[5][6]
- SHARKing
- Suppressing a priori hypotheses after the results of tests of those hypotheses are known.[1][5]
- Active and passive HARKing
- Active HARKing occurs when researchers HARK prior to submitting their research report for publication. Passive HARKing occurs when researchers HARK in response to requests by editors and reviewers during the peer review process.[5]: 317
Prevalence among researchers
edit
Concerns about HARKing appear to be increasing in the scientific community, as shown by the increasing number of citations to Kerr's seminal article.[7] A 2017 review of six surveys found that an average of 43% of researchers surveyed (mainly psychologists) self-reported HARKing "at least once".[5] This figure may be an underestimate if researchers are concerned about reporting questionable research practices, do not perceive themselves to be responsible for HARKing that is proposed by editors and reviewers (i.e., passive HARKing), and/or do not recognize their HARKing due to hindsight or confirmation biases.
Researchers' motivation
editHARKing appears to be motivated by a desire to publish research in a publication environment that values a priori hypotheses over post hoc hypotheses and contains a publication bias against null results. In order to improve their chances of publishing their results, researchers may secretly suppress any a priori hypotheses that failed to yield significant results, construct or retrieve post hoc hypotheses that account for any unexpected significant results, and then present these new post hoc hypotheses in their research reports as if they are a priori hypotheses.[1][8][9][5][10]
Prediction and accommodation
editHARKing is associated with the debate regarding prediction and accommodation.[11] In the case of prediction, hypotheses are deduced from a priori theory and evidence. In the case of accommodation, hypotheses are induced from the current research results.[7] One view is that HARKing represents a form of accommodation in which researchers induce ad hoc hypotheses from their current results.[1][3] Another view is that HARKing represents a form of prediction in which researchers deduce hypotheses from a priori theory and evidence after they know their current results.[7]
Potential costs to science
editPotential costs of HARKing include:[1]: 211
- Translating Type I errors into hard-to-eradicate theory
- Propounding theories that cannot (pending replication) pass Popper's disconfirmability test
- Disguising post hoc explanations as a priori explanations
- Not communicating valuable information about what did not work
- Taking unjustified statistical licence
- Presenting an inaccurate model of science to students
- Encouraging questionable practices in other grey areas
- Making us less receptive to serendipitous findings
- Encouraging adoption of narrow, context-bound new theory
- Encouraging retention of too-broad, disconfirmable old theory
- Inhibiting identification of plausible alternative hypotheses
- Implicitly violating basic ethical principles
In 2022, Rubin provided a critical analysis of Kerr's 12 costs of HARKing. He concluded that these costs "are either misconceived, misattributed to HARKing, lacking evidence, or that they do not take into account pre- and post-publication peer review and public availability to research materials and data."[7]
HARKing and the replication crisis
editSome of the costs of HARKing are thought to have led to the replication crisis in science.[4] Hence, Bishop described HARKing as one of "the four horsemen of the reproducibility apocalypse," with publication bias, low statistical power, and p-hacking[12] being the other three.[13] An alternative view is that it is premature to conclude that HARKing has contributed to the replication crisis.[7][5][14]
The preregistration of research hypotheses prior to data collection has been proposed as a method of identifying and deterring HARKing. However, the use of preregistration to prevent HARKing is controversial.[3]
Ethical concerns
editKerr pointed out that "HARKing can entail concealment. The question then becomes whether what is concealed in HARKing can be a useful part of the 'truth' ...or is instead basically uninformative (and may, therefore, be safely ignored at an author's discretion)".[1]: 209 Three different positions about the ethics of HARKing depend on whether HARKing conceals "a useful part of the 'truth'".
The first position is that all HARKing is unethical under all circumstances because it violates a fundamental principle of communicating scientific research honestly and completely.[1]: 209 According to this position, HARKing always conceals a useful part of the truth.
A second position is that HARKing falls into a "gray zone" of ethical practice.[1][15] According to this position, some forms of HARKing are more or less ethical under some circumstances.[16][5][17][7] Hence, only some forms of HARKing conceal a useful part of the truth under some conditions. Consistent with this view, a 2018 survey of 119 USA researchers found that HARKing ("reporting an unexpected result as having been hypothesized from the start") was associated with "ambiguously unethical" research practices more than with "unambiguously unethical" research practices.[18]
A third position is that HARKing is acceptable provided that hypotheses are explicitly deduced from a priori theory and evidence, as explained in a theoretical rationale, and readers have access to the relevant research data and materials.[7] According to this position, HARKing does not prevent readers from making an adequately informed evaluation of the theoretical quality and plausibility of the HARKed hypotheses and the methodological rigor with which the hypotheses have been tested.[7][17] In this case, HARKing does not conceal a useful part of the truth. Furthermore, researchers may claim that a priori theory and evidence predict their results even if the prediction is deduced after they know their results.[7][19]
See also
editReferences
edit- ^ a b c d e f g h i j k Kerr, N. L. (1998). "HARKing: Hypothesizing after the results are known". Personality and Social Psychology Review. 2 (3): 196–217. doi:10.1207/s15327957pspr0203_4. PMID 15647155.
- ^ John, L. K.; Loewenstein, G.; Prelec, D. (2012). "Measuring the prevalence of questionable research practices with incentives for truth telling". Psychological Science. 23 (5): 524–532. doi:10.1177/0956797611430953. PMID 22508865. S2CID 8400625.
- ^ a b c Lishner, D. A. (2021). "HARKing: Conceptualizations, harms, and two fundamental remedies". Journal of Theoretical and Philosophical Psychology. 41 (4): 248–263. doi:10.1037/teo0000182. S2CID 234886782.
- ^ a b Hollenbeck, J. R.; Wright, P. M. (2017). "Harking, sharking, and tharking: Making the case for post hoc analysis of scientific data". Journal of Management. 43: 5–18. doi:10.1177/0149206316679487.
- ^ a b c d e f g h i j Rubin, M. (2017). "When does HARKing hurt? Identifying when different types of undisclosed post hoc hypothesizing harm scientific progress". Review of General Psychology. 21 (4): 308–320. doi:10.1037/gpr0000128. S2CID 149228437.
- ^ Vancouver, J. B. (2020). "Navigating the review process through the holier than thou". Industrial and Organizational Psychology. 13: 72–75. doi:10.1017/iop.2020.8. S2CID 219044743.
- ^ a b c d e f g h i Rubin, M. (2022). "The costs of HARKing" (PDF). British Journal for the Philosophy of Science. 73 (2): 535–560. doi:10.1093/bjps/axz050.
- ^ Mazzola, J. J.; Deuling, J. K. (2013). "Forgetting what we learned as graduate students: HARKing and selective outcome reporting in I–O journal articles". Industrial and Organizational Psychology: Perspectives on Science and Practice. 6 (3): 279–284. doi:10.1111/iops.12049. S2CID 142493071.
- ^ O'Boyle, E. H. Jr.; Banks, G. C.; Gonzalez-Mulé, E. (2017). "The chrysalis effect: How ugly initial results metamorphosize into beautiful articles". Journal of Management. 43: 367–399. doi:10.1177/0149206314527133. S2CID 145237761.
- ^ Cairo, A. H.; Green, J. D.; Forsyth, D. R.; Behler, A. C.; Raldiris, T. L. (2020). "Gray (literature) matters: Evidence of selective hypothesis reporting in social psychological research". Personality and Social Psychology Bulletin. 46 (9): 1344–1362. doi:10.1177/0146167220903896. PMID 32093574. S2CID 211475516.
- ^ Barnes, Eric Christian (2022), "Prediction versus Accommodation", in Zalta, Edward N.; Nodelman, Uri (eds.), The Stanford Encyclopedia of Philosophy (Winter 2022 ed.), Metaphysics Research Lab, Stanford University, retrieved 2022-10-28
- ^ Simmons, Joseph P.; Nelson, Leif D.; Simonsohn, Uri (17 October 2011). "False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant". Psychological Science. 22 (11): 1359–1366. doi:10.1177/0956797611417632. PMID 22006061.
- ^ Bishop, D. (2019). "Rein in the four horsemen of irreproducibility". Nature. 568 (7753): 435. Bibcode:2019Natur.568..435B. doi:10.1038/d41586-019-01307-2. PMID 31019328.
- ^ Mohseni, A. (2020). "HARKing: From misdiagnosis to misprescription" (PDF). Retrieved 12 November 2020.
{{cite journal}}: Cite journal requires|journal=(help) - ^ Butler, N.; Delaney, H.; Spoelstra, S. (2017). "The gray zone: Questionable research practices in the business school". Academy of Management Learning & Education. 16: 94–109. doi:10.5465/amle.2015.0201.
- ^ Leung, K. (2011). "Presenting post hoc hypotheses as a priori: Ethical and theoretical issues". Management and Organization Review. 7: 471–479. doi:10.1017/CBO9781139171434.009.
- ^ a b Vancouver, J. N. (2018). "In defense of HARKing". Industrial and Organizational Psychology. 11: 73–80. doi:10.1017/iop.2017.89. S2CID 149220988.
- ^ Sacco, D. F.; Bruton, S. V.; Brown, M. (2018). "In defense of the questionable: Defining the basis of research scientists' engagement in questionable research practices". Responsible Conduct of Research and Research Integrity. 13 (1): 101–110. doi:10.1177/1556264617743834. PMID 29179623.
- ^ Worrall, J. (2014). "Prediction and accommodation revisited". Studies in History and Philosophy of Science. 45: 54–61. Bibcode:2014SHPSA..45...54W. doi:10.1016/j.shpsa.2013.10.001. PMID 24984450.