Nullius in Verba

Episode 46: Invaliditas Externa Nov 01, 2024

In this episode, we discuss the paper "In defense of external invalidity" by Douglas Mook.

Shownotes

Mook, D. G. (1983). In defense of external invalidity. American Psychologist, 38(4), 379–387.
Mook, D. G. (1989). The myth of external validity. Everyday cognition in adulthood and late life, 25-43.
The case of Phineas Gage was written up: Harlow, J. M. (1848). Passage of an iron rod through the head. The Boston Medical and Surgical Journal (1828-1851), 39(20)

Prologus 46: In Defense of External Invalidity (D. G. Mook) Oct 25, 2024

A reading of the paper In Defense of External Invalidty by Douglas G. Mook, which will be discussed in the next episode.

Mook, D. G. (1983). In defense of external invalidity. American Psychologist, 38(4), 379–387. https://doi.org/10.1037/0003-066X.38.4.379

Episode 45: Apprenticiatus Oct 18, 2024

In this episode, we discuss the role of apprenticeship in training scientists and researchers. What’s the difference between traditional apprenticeship and cognitive apprenticeship? Does graduate training live up to its promise as an apprenticeship model? What can we do to improve the modeling of skills that are to be taught during graduate training?

Shownotes

Collins, A., Brown, J. S., & Holum, A. (1991). Cognitive apprenticeship: Making thinking visible. American educator, 15(3), 6-11.
Gabrys, B. J., & Beltechi, A. (2012). Cognitive apprenticeship: The making of a scientist. In Reshaping doctoral education (pp. 144-155). Routledge.
Casadevall, A., & Fang, F. C. (2016). Rigorous science: a how-to guide. MBio, 7(6), 10-1128.
Alvesson, M., Gabriel, Y., & Paulsen, R. (2017). Return to meaning: A social science with something to say. Oxford University Press.
Polanyi, M. (1958). Personal Knowledge: Towards a Post-Critical Philosophy (M. J. Nye, Ed.). University of Chicago Press.

Episode 44: Reprehensio Scientiae Aperta Oct 04, 2024

This is a live episode, recorded in Växjö, Sweden (Linnaeus university) on September 24, 2024, at the 5th meeting of the Open Science Community Sweden and the Swedish Reproducibility Network. Thanks to André Kalmendal at Mono (https://monovaxjo.se) for recording the episode.

Episode 43: Historia Casus Methodi Scientifica Sep 20, 2024

In this episode, we discuss the paper "A case history in scientific method" by B. F. Skinner

Shownotes

Skinner, B. F. (1956). A case history in scientific method. American psychologist, 11(5), 221.
Richter, C. P. (1953). Free research versus design research. Science, 118(3056), 91–93.
https://archive.org/details/WaldenTwoChapter01

Prologus 43: A Case Study in Scientific Method (Skinner) Sep 13, 2024

In preparation for the next episode, in which we discuss this paper, here is a reading of:

Skinner, B. F. (1956). A case history in scientific method. American Psychologist, 11(5), 221-233.

Episode 42: Aestimatio Scriptorum Sep 06, 2024

In today’s episode, we discuss critically reading and appraising scientific articles. How do we select which articles to read carefully? Which heuristics are useful for assessing paper quality? And do open science practices actually lead to better quality papers? Enjoy.

Shownotes

Bacon, F. (1625). Of Studies.
PNAS Submissions contributed by NAS members "The contributing member submits the manuscript to PNAS along with the names of at least two experts in the field of the paper who have agreed to review the work and brief comments about why each of those reviewers was chosen." https://www.pnas.org/pb-assets/authors/ifora-1720190309383.pdf
How many p-values just below 0.05 should we expect across multiple tests? https://daniellakens.blogspot.com/2014/05/the-probability-of-p-values-as-function.html
Lakens, D. (2024). When and How to Deviate From a Preregistration. Collabra: Psychology, 10(1), 117094. https://doi.org/10.1525/collabra.117094
TIER protocol: https://www.projecttier.org/tier-protocol/protocol-4-0/
Gino fraud investigation and excel meta-data: https://datacolada.org/109
REAPPRAISED checklist: https://www.nature.com/articles/d41586-019-03959-6
Yang, Z., & Hung, I. W. (2021). Creative thinking facilitates perspective taking. Journal of Personality and Social Psychology, 120(2), 278.

Episode 41: Sodalitates Academicae Aug 23, 2024

In this episode, we talk about academic societies, professional organizations, and academic advocacy groups, focusing primarily on the discipline of psychology. What are their roles and responsibilities? Is it necessary for researchers to join such organizations? And should we bring back scholarly soirees? Enjoy.

Shownotes

Royal Society Referee Reports

Psychological Science

APA Divisions

Consistori del Gay Saber

ReproducibiliTea

The Royal Society Soirées

Episode 40: Tabula de Ethicis Recensionibus Aug 09, 2024

In this episode, we discuss review boards for research with human subjects. Are they necessary? Are they efficient? Are scientists well equipped to make judgements about ethics? And are economists more ethical than psychologists?

Shownotes

Whitney, S. N. (2015). Balanced ethics review: A guide for institutional review board members. Springer.
Schrag, Z. M. (2010). Ethical imperialism: Institutional review boards and the social sciences, 1965–2009. JHU Press.
Kinsey Reports
- Masters & Johnson
How Institutional Review Boards can be (and are) Weaponized Against Academic Freedom
Weaponizing the IRB 2.0
Psychologists’ Involvement in Torture and the APA. Psychology Today.

Episode 39: Activismus Jul 27, 2024

In this episode, we discuss activism in science. How do political and personal values affect science? When is activism just part of the job? And should one be careful about activism in the classroom? Enjoy.

Shownotes:

Frisby, C. L., Redding, R. E., & O’Donohue, W. T. (2023). Ideological and Political Bias in Psychology: An Introduction. In Ideological and Political Bias in Psychology: Nature, Scope, and Solutions (pp. 1-14). Cham: Springer International Publishing.
McCaughey, M. (2023). The Trouble with Scholar-Activists. AAUP.
McCaughey, M. (2024). Against Scholar Activists. The James G. Martin Center for Academic Renewal.
Honeycutt, N., & Jussim, L. (2023). Political bias in the social sciences: A critical, theoretical, and empirical review. Ideological and Political Bias in Psychology: Nature, Scope, and Solutions, 97-146.
Sargent, R. M. (2012). From Bacon to Banks: The vision and the realities of pursuing science for the common good. Studies in History and Philosophy of Science Part A, 43(1), 82-90.
Weber, M. (1946). Science as a Vocation. In Science and the Quest for Reality (pp. 382-394). London: Palgrave Macmillan UK.

Episode 38 - Replicatio - II Jul 12, 2024

In this episode, we continue our discussion of replications. We talk about how to analyze replication studies, which studies are worth replicating, and what is the status of replications in other scientific disciplines.

Shownotes

Mack, R. W. (1951). The Need for Replication Research in Sociology. American Sociological Review, 16(1), 93–94. https://doi.org/10.2307/2087978
Smith, N. C. (1970). Replication studies: A neglected aspect of psychological research. American Psychologist, 25(10), 970–975. https://doi.org/10.1037/h0029774
Sidman, M. (1960). Tactics of Scientific Research: Evaluating Experimental Data in Psychology (New edition). Cambridge Center for Behavioral.
Ebersole, C. R., Mathur, M. B., Baranski, E., Bart-Plange, D.-J., Buttrick, N. R., Chartier, C. R., Corker, K. S., Corley, M., Hartshorne, J. K., IJzerman, H., Lazarević, L. B., Rabagliati, H., Ropovik, I., Aczel, B., Aeschbach, L. F., Andrighetto, L., Arnal, J. D., Arrow, H., Babincak, P., … Nosek, B. A. (2020). Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability. Advances in Methods and Practices in Psychological Science. https://doi.org/10.1177/2515245920958687
Isager, P. M., van Aert, R. C. M., Bahník, Š., Brandt, M. J., DeSoto, K. A., Giner-Sorolla, R., Krueger, J. I., Perugini, M., Ropovik, I., van ’t Veer, A. E., Vranka, M., & Lakens, D. (2023). Deciding what to replicate: A decision model for replication study selection under resource and knowledge constraints. Psychological Methods, 28(2), 438–451. https://doi.org/10.1037/met0000438
Aldhous, P. (2011). Journal rejects studies contradicting precognition. New Scientist. https://www.newscientist.com/article/dn20447-journal-rejects-studies-contradicting-precognition/
Stanley, D. J., & Spence, J. R. (2014). Expectations for Replications: Are Yours Realistic? Perspectives on Psychological Science, 9(3), 305–318. https://doi.org/10.1177/1745691614528518
Simonsohn, U. (2015). Small telescopes: Detectability and the evaluation of replication results. Psychological Science, 26(5), 559–569. https://doi.org/10.1177/0956797614567341
Nosek, B.A., Errington, T.M. (2017) Reproducibility in Cancer Biology: Making sense of replications. eLife 6:e23383. https://doi.org/10.7554/eLife.23383

Episode 37: Replicatio - I Jun 28, 2024

In the next two episodes, we will discuss replication studies, which are essential to building reliable scientific knowledge.

Shownotes

Mack, R. W. (1951). The Need for Replication Research in Sociology. American Sociological Review, 16(1), 93–94. https://doi.org/10.2307/2087978
Smith, N. C. (1970). Replication studies: A neglected aspect of psychological research. American Psychologist, 25(10), 970–975. https://doi.org/10.1037/h0029774
Sidman, M. (1960). Tactics of Scientific Research: Evaluating Experimental Data in Psychology (New edition). Cambridge Center for Behavioral.
Ebersole, C. R., Mathur, M. B., Baranski, E., Bart-Plange, D.-J., Buttrick, N. R., Chartier, C. R., Corker, K. S., Corley, M., Hartshorne, J. K., IJzerman, H., Lazarević, L. B., Rabagliati, H., Ropovik, I., Aczel, B., Aeschbach, L. F., Andrighetto, L., Arnal, J. D., Arrow, H., Babincak, P., … Nosek, B. A. (2020). Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability. Advances in Methods and Practices in Psychological Science. https://doi.org/10.1177/2515245920958687
Isager, P. M., van Aert, R. C. M., Bahník, Š., Brandt, M. J., DeSoto, K. A., Giner-Sorolla, R., Krueger, J. I., Perugini, M., Ropovik, I., van ’t Veer, A. E., Vranka, M., & Lakens, D. (2023). Deciding what to replicate: A decision model for replication study selection under resource and knowledge constraints. Psychological Methods, 28(2), 438–451. https://doi.org/10.1037/met0000438
Aldhous, P. (2011). Journal rejects studies contradicting precognition. New Scientist. https://www.newscientist.com/article/dn20447-journal-rejects-studies-contradicting-precognition/
Stanley, D. J., & Spence, J. R. (2014). Expectations for Replications: Are Yours Realistic? Perspectives on Psychological Science, 9(3), 305–318. https://doi.org/10.1177/1745691614528518
Simonsohn, U. (2015). Small telescopes: Detectability and the evaluation of replication results. Psychological Science, 26(5), 559–569. https://doi.org/10.1177/0956797614567341

Prologus 37: Replication studies: A neglected aspect of psychological research (N. C. Smith) Jun 21, 2024

Smith, N. C. (1970). Replication studies: A neglected aspect of psychological research. American Psychologist, 25(10), 970–975. https://doi.org/10.1037/h0029774

Episode 36: Leges Eponymae Jun 14, 2024

In this episode, we discuss a fun mix of eponymous laws, which are laws named after individuals who postulate them.

Shownotes

Campbell, D. T. (1979). Assessing the impact of planned social change. Evaluation and Program Planning, 2(1), 67–90. https://doi.org/10.1016/0149-7189(79)90048-X
Merton, R. K. (1995). The Thomas Theorem and the Matthews Effect. Social Forces, 74(2), 379–422.
Stigler, S. M. (1980). Stigler’s Law of Eponymy*. Transactions of the New York Academy of Sciences, 39(1 Series II), 147–157. https://doi.org/10.1111/j.2164-0947.1980.tb02775.x
Clarke, A. C. (Arthur C. (1962). Profiles of the future: An inquiry into the limits of the possible. New York : Bantam Books. http://archive.org/details/profilesoffuture00clar
Brandolini’s Law: Based on a tweet, after reading Kahneman Thinking fast and slow: https://twitter.com/ziobrando/status/289635060758507521
Preston, I. L. (1980). Researchers at the Federal Trade Commission—Peril and Promise. Current Issues and Research in Advertising, 3(1), 1–15.
Twyman’s Law: “The more unusual or interesting the data, the more likely they are to have been the result of an error of one kind or another.” Earliest scholarly reference is in Journal of the Royal Statistical Society, Series A, Vol 138, No 4, 1975. The Teaching of Statistics by A. S. C. Ehrenberg.
Bloch, A. (1990). Murphy's law complete: All the reasons why everything goes wrong. Arrow Books Limited.

Episode 35: Praedictio Clinica Versus Statistica May 31, 2024

In this final episode of the three-part series on the Philosophical Psychology lectures by Paul Meehl, we discuss lectures 6-8, which cover the ten obfuscating factors in "soft areas" of psychology and a host of advice Meehl provides for researchers, reviewers, editors, and educators on how to improve practice.

Shownotes

Krefeld-Schwalb, A., Sugerman, E. R., & Johnson, E. J. (2024). Exposing omitted moderators: Explaining why effect sizes differ in the social sciences. Proceedings of the National Academy of Sciences, 121(12), e2306281121. https://doi.org/10.1073/pnas.2306281121
Lakens, D., & Etz, A. J. (2017). Too True to be Bad: When Sets of Studies With Significant and Nonsignificant Findings Are Probably True. Social Psychological and Personality Science, 8(8), 875–881. https://doi.org/10.1177/1948550617693058

Episode 34: Aestimatio et Emendatio Theoriarum May 17, 2024

In this episode, we continue the discussion of Meehl's Philosophy of Psychology course, focusing on lectures 3, 4, and 5.

Shownotes

The quote "Don't make a mockery of honest ad-hockery" is probably from Clark Glymour: https://en.wikipedia.org/wiki/Clark_Glymour
Good, I. J. (1965). The Estimation of Probabilities: An Essay on Modern Bayesian Methods. M.I.T. Press.
Shepard, R. N. (1987). Toward a universal law of generalization for psychological science. Science, 237(4820), 1317–1323.

Prologus 34: Using scientific methods to resolve questions in the history and philosophy of science (Faust & Meehl) May 10, 2024

Faust, D., & Meehl, P. E. (1992). Using scientific methods to resolve questions in the history and philosophy of science: Some illustrations. Behavior Therapy, 23(2), 195–211. https://doi.org/10.1016/S0005-7894(05)80381-8

Episode 33: Risicae Theoreticae et Asterisci Tabulares May 03, 2024

Video lectures: https://meehl.umn.edu/video

Faust, D., & Meehl, P. E. (1992). Using scientific methods to resolve questions in the history and philosophy of science: Some illustrations. Behavior Therapy, 23(2), 195–211. https://doi.org/10.1016/S0005-7894(05)80381-8

Serlin, R. C., & Lapsley, D. K. (1985). Rationality in psychological research: The good-enough principle. American Psychologist, 40(1), 73–83. https://doi.org/10.1037/0003-066X.40.1.73

Meehl, P. E. (1990). Appraising and amending theories: The strategy of Lakatosian defense and two principles that warrant it. Psychological Inquiry, 1(2), 108–141. https://doi.org/10.1207/s15327965pli0102_1

Meehl, P. E. (1992). Cliometric metatheory: The actuarial approach to empirical, history-based philosophy of science. Psychological Reports, 71, 339–467.

Prologus 33: Paul E. Meehl Apr 26, 2024

In advance of the next three episodes discussing the Philosophical Psychology lectures by Paul E. Meehl, we present a brief reading from his autobiography in A history of psychology in autobiography.

Meehl, P. E. (1989). Paul E. Meehl. In G. Lindzey (Ed.), A history of psychology in autobiography (Vol. 8, pp. 337–389). Stanford, CA: Stanford University Press.

Episode 32: Impartialitas Apr 19, 2024

In this episode, we discuss objectivity and disinterestedness in science. We talk about norms, values, interests, and objectivity in research practice, peer review, and hiring decisions. Is it possible to be completely objective? Is objectivity a feature of epistemic products or epistemic processes? And most importantly, how would you objectively rate this podcast?

Shownotes

Armstrong, J. S. (1979). Advocacy and objectivity in science. Management Science, 25(5), 423–428.
Declaration of Interest by Stephen Senn: http://senns.uk/Declaration_Interest.htm
Djørup, S., & Kappel, K. (2013). The norm of disinterestedness in science; a restorative analysis. SATS, 14(2). https://doi.org/10.1515/sats-2013-0009
Elliott, K. C. (2017). A Tapestry of Values: An Introduction to Values in Science. Oxford University Press. https://doi.org/10.1093/acprof:oso/9780190260804.001.0001
Feyerabend, Paul. "How to defend society against science." Philosophy: Basic Readings (1975): 261-271.
Jamieson, K. H., McNutt, M., Kiermer, V., & Sever, R. (2019). Signaling the trustworthiness of science. Proceedings of the National Academy of Sciences, 116(39), 19231–19236. https://doi.org/10.1073/pnas.1913039116
Janack, M. (2002). Dilemmas of objectivity. Social Epistemology, 16(3), 267-281.
John, S. (2021). Objectivity in science. Cambridge University Press.
Merton, R. K. (1973). The sociology of science: Theoretical and empirical investigations. University of Chicago Press.
Mitroff, I. I. (1974). Norms and Counter-Norms in a Select Group of the Apollo Moon Scientists: A Case Study of the Ambivalence of Scientists. American Sociological Review, 39(4), 579–595. https://doi.org/10.2307/2094423
Mitroff, I. I. (1974). The subjective side of science: A philosophical inquiry into the psychology of the Apollo moon scientists (First Edition). Elsevier.
A Russian polar researcher has been charged trying to stab a colleague to death at a remote Antarctic base https://www.businessinsider.com/sergey-savitsky-alleged-attempted-murder-at-antarctic-bellingshausen-2018-10
Stamenkovic, P. (2023). Facts and objectivity in science. Interdisciplinary Science Reviews, 1–22. https://doi.org/10.1080/03080188.2022.2150807

Episode 31: Criticismus Apr 05, 2024

In this episode, we discuss the role of criticism in science. When is criticism constructive as opposed to obsessive? What are the features of fair and useful scientific criticism? And should we explicitly teach junior researchers to both give and accept criticism?

Shownotes:

Babbage, C. (1830). Reflections on the Decline of Science in England: And on Some of Its Causes.
Prasad, Vinay, and John PA Ioannidis. "Constructive and obsessive criticism in science." European journal of clinical investigation 52.11 (2022): e13839.
Lakatos, I. (1968, January). Criticism and the methodology of scientific research programmes. In Proceedings of the Aristotelian society (Vol. 69, pp. 149-186). Aristotelian Society, Wiley.
LOWI: https://lowi.nl/en/home/ As an independent advisory body it plays a role in the complaints procedure about alleged violations of principles of research integrity.
Holcombe, A. O. (2022). Ad hominem rhetoric in scientific psychology. British Journal of Psychology, 113(2), 434–454. https://doi.org/10.1111/bjop.12541
Daniel C. Dennett: I've Been Thinking https://wwnorton.com/books/9780393868050
Phillip Stark textbook chapter on logical fallacies: https://www.stat.berkeley.edu/~stark/SticiGui/Text/reasoning.htm
Gelman, A., & Tuerlinckx, F. (2000). Type S error rates for classical and Bayesian single and multiple comparison procedures. Computational Statistics, 15(3), 373–390. https://doi.org/10.1007/s001800000040
Popper, K. R. (1959). The logic of scientific discovery. Routledge.
PubPeer: https://pubpeer.com

Episode 30: Theoria Aedificans - Pars II Mar 22, 2024

In this episode, we continue discussing Dubin’s 8-step method for theory building. We discuss the measurement of theoretical constructs, using logical propositions to make falsifiable predictions from theories, and the importance of specifying boundary conditions.

Shownotes

Jaccard, J., & Jacoby, J. (2010). Theory Construction and Model-building Skills: A Practical Guide for Social Scientists. Guilford Press.
McGuire, W. J. (1973). The yin and yang of progress in social psychology: Seven koan. Journal of Personality and Social Psychology, 26(3), 446–456. https://doi.org/10.1037/h0034345
Simons, D. J., Shoda, Y., & Lindsay, D. S. (2017). Constraints on Generality (COG): A Proposed Addition to All Empirical Papers. Perspectives on Psychological Science, 12(6), 1123–1128. https://doi.org/10.1177/1745691617708630
Norm Macdonald: The Professor of Logic
Raven Paradox: https://platonicrealms.com/encyclopedia/Hempels-Ravens-Paradox
Pavlov, I. (1936). Bequest of Pavlov to the Academic Youth of His Country. Science, 83(2155), 369–370. https://doi.org/10.1126/science.83.2155.369

Episode 29: Theoria Aedificans - Pars I Mar 08, 2024

In this episode we discussed the 8-step method of theory building proposed by Robin Dubin in his classic 1969 book Theory Building.

Shownotes

Dubin, R. (1969). Theory building. Free Press. http://catalog.hathitrust.org/api/volumes/oclc/160506.html
Lynham, S. A. (2002). Quantitative Research and Theory Building: Dubin’s Method. Advances in Developing Human Resources, 4(3), 242–276. https://doi.org/10.1177/15222302004003003
Elms, A. C. (1975). The crisis of confidence in social psychology. American Psychologist, 30(10), 967.
Meehl, P. E. (1978). Theoretical Risks and Tabular Asterisks: Sir Karl, Sir Ronald, and the Slow Progress of Soft Psychology. Journal of Consulting and Clinical Psychology, 46(4), 806–834. https://doi.org/10.1037/0022-006X.46.4.806
Swedberg, R. (2014). The art of social theory. Princeton University Press.
Ben Wright: https://en.wikipedia.org/wiki/Benjamin_Drake_Wright
Yarkoni, T., & Westfall, J. (2017). Choosing Prediction Over Explanation in Psychology: Lessons From Machine Learning. Perspectives on Psychological Science, 12(6), 1100–1122. https://doi.org/10.1177/1745691617693393
Isaac, M. G., Koch, S., & Nefdt, R. (2022). Conceptual engineering: A road map to practice. Philosophy Compass, 17(10), e12879. https://doi.org/10.1111/phc3.12879

Episode 28: Scientia Cumulativa Feb 23, 2024

In this episode, we discuss the barriers to cumulative science, including inconsistent measurement tools, overreliance on single studies, and the large volume of research publications. Can replications, interdisciplinary collaborations, and prospective meta-analyses help us solve this issue? Can AI solve all our problems? And do most scientists treat their theories like toothbrushes?

Shownotes

Opening quote by George Sarton
- Sarton, G. (1927). Introduction to the History of Science (Vol. 376).
Is Science Cumulative? a Physicist Viewpoint: https://link.springer.com/chapter/10.1007/978-1-4020-6279-7_10
Psychological Methods. (2009). Special Issue: Multi-Study Methods for Building a Cumulative Psychological Science.

Walter Mischel, Becoming a Cumulative Science

Dorothy Bishop - Why we need cumulative science (AIMOS)
Watkins, J. W. (1984). Science and Skepticism. Princeton University Press.

Prologus 28: Chaos in the Brickyard (B. K. Forscher) Feb 16, 2024

A reading of: Forscher, B. K. (1963). Chaos in the Brickyard. Science, 142(3590), 339–339. https://doi.org/10.1126/science.142.3590.339

Episode 27: Vocans Ictus Tuos - Pars II Feb 09, 2024

In today’s episode, we continue our conversation about preregistration. How flexible can we be when we preregister, without increasing flexibility in our analysis? How well do people preregister, and what does a good preregistration look like? And how do we deal with deviations from preregistrations?

Shownotes

Dubin, R. (1969). Theory building. Free Press. His full quote is: "There is no more devastating commendation that the self-designated theorist makes of the researcher than to label his work purely descriptive".
Claesen, A., Gomes, S., Tuerlinckx, F., & Vanpaemel, W. (2021). Comparing dream to reality: An assessment of adherence of the first generation of preregistered studies. Royal Society Open Science, 8(10), 211037. https://doi.org/10.1098/rsos.211037
Akker, O. van den, Bakker, M., Assen, M. A. L. M. van, Pennington, C. R., Verweij, L., Elsherif, M., Claesen, A., Gaillard, S. D. M., Yeung, S. K., Frankenberger, J.-L., Krautter, K., Cockcroft, J. P., Kreuer, K. S., Evans, T. R., Heppel, F., Schoch, S. F., Korbmacher, M., Yamada, Y., Albayrak-Aydemir, N., … Wicherts, J. (2023). The effectiveness of preregistration in psychology: Assessing preregistration strictness and preregistration-study consistency. MetaArXiv. https://doi.org/10.31222/osf.io/h8xjw
Sequential analysis and alpha spending functions https://lakens.github.io/statistical_inferences/10-sequential.html
Bishop, D. V. M. (2018). Fallibility in Science: Responding to Errors in the Work of Oneself and Others. Advances in Methods and Practices in Psychological Science, 2515245918776632. https://doi.org/10.1177/2515245918776632
FDAAA Trial Tracker https://fdaaa.trialstracker.net
Ensinck, E., & Lakens, D. (2023). An Inception Cohort Study Quantifying How Many Registered Studies are Published. PsyArXiv. https://doi.org/10.31234/osf.io/5hkjz
Quantitude episode on preregistration https://quantitudepod.org/s3e07-in-defense-of-researcher-degrees-of-freedom/
Lakens, D. (2023). When and How to Deviate from a Preregistration. PsyArXiv. https://doi.org/10.31234/osf.io/ha29k

Episode 26: Vocans Ictus Tuos - Pars I Jan 26, 2024

In this two part episode we discuss the fine art of preregistration. We go back into the history of preregistration, its evolution, and current use. Do we preregister to control the Type 1 error rate, or to show that we derived our prediction from theory a priori? Can and should we preregister exploratory or secondary data analysis? And how severe is the issue of severe testing?

Shownotes

ClinicalTrials.gov
You can preregister on AsPredicted and the OSF
Johnson, M. (1975). Models of Control and Control of Bias. European Journal of Parapsychology, 36–44.
SPIRIT Checklist
Bishop, D. V. M. (2018). Fallibility in Science: Responding to Errors in the Work of Oneself and Others. Advances in Methods and Practices in Psychological Science, 1(3), 432–438. https://doi.org/10.1177/2515245918776632
FDA trials tracker: https://fdaaa.trialstracker.net
Ensinck, E., & Lakens, D. (2023). An Inception Cohort Study Quantifying How Many Registered Studies are Published. PsyArXiv. https://doi.org/10.31234/osf.io/5hkjz
van den Akker, O. R., van Assen, M. A. L. M., Enting, M., de Jonge, M., Ong, H. H., Rüffer, F., Schoenmakers, M., Stoevenbelt, A. H., Wicherts, J. M., & Bakker, M. (2023). Selective Hypothesis Reporting in Psychology: Comparing Preregistrations and Corresponding Publications. Advances in Methods and Practices in Psychological Science, 6(3), 25152459231187988. https://doi.org/10.1177/25152459231187988
Claesen, A., Gomes, S., Tuerlinckx, F., & Vanpaemel, W. (2021). Comparing dream to reality: An assessment of adherence of the first generation of preregistered studies. Royal Society Open Science, 8(10), 211037. https://doi.org/10.1098/rsos.211037
Bakan, D. (1966). The test of significance in psychological research. Psychological Bulletin, 66(6), 423–437. https://doi.org/10.1037/h0020412
Rosenthal, R. (1966). Experimenter effects in behavioral research. Appleton-Century-Crofts.
Johnson, M. (1975). Models of Control and Control of Bias. European Journal of Parapsychology, 36–44.
de Groot, A. D. (1969). Methodology. Mouton & Co.
Claesen, A., Lakens, D., Vanpaemel, W., & Dongen, N. van. (2022). Severity and Crises in Science: Are We Getting It Right When We’re Right and Wrong When We’re Wrong? PsyArXiv. https://doi.org/10.31234/osf.io/ekhc8

Episode 25: Reverentia Ad Auctoritatem Jan 12, 2024

In the first episode of 2024, we discuss the double-edged sword: reverence to authority. Should scientists respect others on whose shoulders they stand? Or should they be wary of appeal to authority? How should scientists deal with other sources of authority in science, like for example, the government or academic societies? And how can we differentiate true expertise from mere authority? Enjoy.

Shownotes

Frank, P. (1956). The role of authority in the interpretation of science. Synthese, 10, 335–338.
Barber, B. (1952). Science and the social order. Glencoe, Ill. : Free Press. http://archive.org/details/sciencesocialord0000barb
Barber, B. (1961). Resistance by Scientists to Scientific Discovery. Science, 134(3479), 596–602.
Kitcher, P. (1992). Authority, deference, and the role of individual reasoning in science. In E. Mcmullin (Ed.), The social dimensions of science. Notre Dame: The University of Notre Dame Press
Polanyi, M. (1962). The republic of science. Minerva, 1(1), 54–73
The practice of two-spaces after the end of a sentence comes from when type-writers used monospaced typefaces: https://slate.com/technology/2011/01/two-spaces-after-a-period-why-you-should-never-ever-do-it.html

Prologus 25: The Fixation of Belief (C. S. Peirce) Jan 05, 2024

The Fixation of Belief. Charles S. Peirce. Popular Science Monthly 12 (November 1877), 1-15.

http://peirce.org/writings/p107.html

Episode 24: Contra Creativitatem Epistolae - Pars II Dec 29, 2023

In this second installment of The Anticreativity Letters, we continue discussing the Tempter's tactics for stifling creativity and how to overcome them.

Episode 23: Contra Creativitatem Epistolae - Pars I Dec 15, 2023

In the first of a two-part episode, we discuss The Anticreativity Letters by Richard Nisbett, in which a senior "tempter" advises a junior tempter on ways to prevent a young psychologist from being a productive and creative scientist.

Nisbett, R. E. (1990). The anticreativity letters: Advice from a senior tempter to a junior tempter. American Psychologist, 45(9), 1078–1082.
BMJ Christmas issue: https://www.bmj.com/about-bmj/resources-authors/article-types/christmas-issue
Quote by Ira Glass: https://www.goodreads.com/quotes/309485-nobody-tells-this-to-people-who-are-beginners-i-wish

Prologus 23: The Anticreativity Letters (R. E. Nisbett) Dec 08, 2023

A reading of:

Nisbett, R. E. (1990). The anticreativity letters: Advice from a senior tempter to a junior tempter. American Psychologist, 45(9), 1078–1082. https://doi.org/10.1037/0003-066X.45.9.1078

Episode 22: Magisterium Dec 01, 2023

In today’s episode, we discuss the role of mentorship in academia. What are the characteristics of a good mentor-mentee relationship? What are the qualities of good mentors and good mentees? Does mentorship play a role in the development of scientific knowledge? And could mentors and mentees benefit from couples therapy?

Note: D.I.H.C is pronounced 'dick' but this is meant to be a family-friendly podcast :)

Shownotes

https://www.psychologytoday.com/us/blog/emotional-fitness/201303/10-things-your-relationship-needs-thrive

Roberts, L. R., Tinari, C. M., & Bandlow, R. (2019). An effective doctoral student mentor wears many hats and asks many questions. International Journal of Doctoral Studies, 14, 133.

Sarabipour, S., Niemi, N. M., Burgess, S. J., Smith, C. T., Filho, A. W. B., Ibrahim, A., & Clark, K. (2023). Insights from a survey of mentorship experiences by graduate and postdoctoral researchers (p. 2023.05.05.539640). bioRxiv. https://doi.org/10.1101/2023.05.05.539640

Episode 21: Verifica Sed Confide Nov 17, 2023

In this episode, we discuss the role of trust in science. Why should we verify but trust other scientists? What are the prerequisites for building trust within the scientific community? Who is ultimately responsible for verifying our claims and practices that bolster those claims? And should we give personality tests to everyone who enters academia?

Shownotes

Hardwig, J. (1991). The role of trust in knowledge. The Journal of Philosophy, 88(12), 693–708.

Hendriks, F., Kienhues, D., Bromme, R. (2016). Trust in Science and the Science of Trust. In: Blöbaum, B. (eds) Trust and Communication in a Digitized World. Progress in IS. Springer, Cham. https://doi.org/10.1007/978-3-319-28059-2_8

Strand, J. F. (2023). Error tight: Exercises for lab groups to prevent research mistakes. Psychological Methods, No Pagination Specified-No Pagination Specified. https://doi.org/10.1037/met0000547

Duygu Uygun-Tunç: Trust and criticism in science, Part I: Critical rationalism instead of organized skepticism: https://uyguntunc.wordpress.com/2020/10/30/trust-and-criticism-in-science-part-i-critical-rationalism-instead-of-organized-skepticism/

Vazire, S. (2017). Quality Uncertainty Erodes Trust in Science. Collabra: Psychology, 3(1), 1. https://doi.org/10.1525/collabra.74

Wicherts, J. M. (2011). Psychology must learn a lesson from fraud case. Nature, 480(7375), Article 7375. https://doi.org/10.1038/480007a

Fricker, E. (2002). Trusting others in the sciences: A priori or empirical warrant? Studies in History and Philosophy of Science Part A, 33(2), 373–383. https://doi.org/10.1016/S0039-3681(02)00006-7

Prologus 21: Role of Trust in Knowledge (J. Hardwig) Nov 10, 2023

In advance of our episode Verify but Trust, a reading of John Hardwig's paper The Role of Trust in Science.

Hardwig, J. (1991). The role of trust in knowledge. The Journal of Philosophy, 88(12), 693–708.

Episode 20: Recensio Aequalium Nov 03, 2023

In today’s episode, we discuss the peer review process---its history, its present, and its future. How does peer review work? How long has it existed in its current form? Should reviews be open and signed? Should reviewers be paid for their hard labor? Should we just abandon the peer review process, or does it have a positive role to play?

Shownotes

Peer Community in Registered Reports: https://rr.peercommunityin.org/

Suggestion to Darwin to publish a book about pigeons instead of The Origins of Species: https://www.darwinproject.ac.uk/letter/DCP-LETT-2457A.xml

Baldwin, M. (2018). Scientific Autonomy, Public Accountability, and the Rise of “Peer Review” in the Cold War United States. Isis, 109(3), 538–558. https://doi.org/10.1086/700070

Burnham, J. C. (1990). The evolution of editorial peer review. JAMA, 263(10), 1323–1329.

Episode 19: Quantifauxcation Oct 20, 2023

In this episode, we discuss Quantifauxcation, described by statistician Philip Stark as “situations in which a number is, in effect, made up, and then is given credence merely because it is quantitative.” We give examples of quantifauxcation in psychology, including errors of the third kind. We spend the second half of the podcast discussing how to develop quantitative measures that are meaningful and bridge the divide between qualitative and quantitative observations.

Shownotes

Statistics textbook by Philip Stark.

Stark, P. B. (2022). Pay No attention to the model behind the curtain. Pure and Applied Geophysics, 179(11), 4121–4145. https://doi.org/10.1007/s00024-022-03137-2

Burgess, E. W. (1927). Statistics and case studies as methods of sociological research, Vol 12(3), 103-120. (Thanks to Andy Grieve!)

Nick Brown's role in pointing out flaws in the positivity ratio.

Retraction notice of the positivity ratio paper.

Blog by Tania Lombrozo on nonsensical formulas in abstracts.

Kimball, A. W. (1957). Errors of the third kind in statistical consulting. Journal of the American Statistical Association, 52(278), 133–142. https://doi.org/10.1080/01621459.1957.10501374

Type III errors: Philip Stark’s post of Deborah Mayo’s blog

Brower, D. (1949). The problem of quantification in psychological science. Psychological Review, 56(6), 325–333. https://doi.org/10.1037/h0061802

Guttman scales

Wilson, M. (2023). Constructing measures: An item response modeling approach. Taylor & Francis.

Wilson, M., Bathia, S., Morell, L., Gochyyev, P., Koo, B. W., & Smith, R. (2022). Seeking a better balance between efficiency and interpretability: Comparing the likert response format with the Guttman response format. Psychological Methods. Advance online publication. https://doi.org/10.1037/met0000462

Bhatti, H.A., Mehta, S., McNeil, R., Wilson, M. (2023). A scientific approach to assessment: Rasch measurement and the four building blocks. In X. Liu & W. Boone (Eds.), Advances in Applications of Rash Measurement in Science Education. Springer Nature.

Prologus 19: Problem-Centering vs. Mean-Centering in Science (A. H. Maslow) Oct 13, 2023

In preparation for a discussion on Quantifauxcation, a reading of 'Problem-Centering vs. Means-Centering in Science' by Abraham H. Maslow (1946).

Maslow, A. H. (1946). Problem-Centering vs. Means-Centering in Science. Philosophy of Science, 13(4), 326–331. https://doi.org/10.1086/286907

Episode 18: Vitia Vocationalis Oct 06, 2023

In today’s episode, we discuss intellectual vices. How can we tell the difference between justified confidence and unjustified arrogance? How do we deal with feelings of envy or negative comparison with other scientists? What is the difference between building one’s career and careerism? And what do we do about scientists who do not care about the truth?

Shownotes

Azrin, N. H., Holz, W., Ulrich, R., & Goldiamond, I. (1961). The control of the content of conversation through reinforcement. Journal of the Experimental Analysis of Behavior, 4, 25–30. https://doi.org/10.1901/jeab.1961.4-25

Meehl, P. E. (1967). Theory-testing in psychology and physics: A methodological paradox. Philosophy of Science, 103–115. https://doi.org/10.1086/288135

Mitroff, I. I. (1974). Norms and Counter-Norms in a Select Group of the Apollo Moon Scientists: A Case Study of the Ambivalence of Scientists. American Sociological Review, 39(4), 579–595. https://doi.org/10.2307/2094423

Susan Blackmore: https://en.wikipedia.org/wiki/Susan_Blackmore

Episode 17: Snobismus Sep 22, 2023

In this episode, we discuss scientific snobbery and the ways in which it affects our interactions with and perceptions of other scientists. What are the reasons for hierarchies among different disciplines, institutions, and approaches to science? What are some ways in which snobbery manifests in science? And is it snobby to not want to present scientific posters? Enjoy.

Shownotes:

Ego and Math (3Blue1Brown)

M. V. Berry; Regular and irregular motion. AIP Conf. Proc. 15 September 1978; 46 (1): 16–120. https://doi.org/10.1063/1.31417

Episode 16: Vetus Crisi Replicatio Sep 08, 2023

In this episode, we continue our conversation on the replication crisis⏤Which methodological, theoretical, and practical concerns did psychologists raise half a century ago? What has changed, and what remains the same, during the current crisis?

Shownotes

Orne, M. T. (1962). On the social psychology of the psychological experiment: With particular reference to demand characteristics and their implications. American Psychologist, 17(11), 776–783. https://doi.org/10.1037/h0043424

Rosenthal, R. (1966). Experimenter effects in behavioral research. Appleton-Century-Crofts.

Gergen, K. J. (1973). Social psychology as history. Journal of Personality and Social Psychology, 26, 309–320. https://doi.org/10.1037/h0034436

Koole, S. L., & Lakens, D. (2012). Rewarding replications: A sure and simple way to improve psychological science. Perspectives on Psychological Science, 7(6), 608–614. https://doi.org/10.1177/1745691612462586

Greenwald, A. G. (Ed.). (1976). An editorial. Journal of Personality and Social Psychology, 33(1), 1–7. https://doi.org/10.1037/h0078635

Ring, K. (1967). Experimental social psychology: Some sober questions about some frivolous values. Journal of Experimental Social Psychology.

Ledgerwood, A., & Sherman, J. W. (2012). Short, sweet, and problematic? The rise of the short report in psychological science. Perspectives on Psychological Science, 7(1), 60–66.

Barber, T. X. (1976). Pitfalls in Human Research: Ten Pivotal Points. Pergamon Press.

Dunnette, M. D. (1966). Fads, fashions, and folderol in psychology. American Psychologist, 21(4), 343–352. https://doi.org/10.1037/h0023535

Babbage, C. (1830). Reflections on the Decline of Science in England: And on Some of Its Causes. B. Fellowes.

Prologus 16: Investigator Data Analysis Effect (T. X. Barber) Sep 01, 2023

Reading of the chapter "Investigator Data Analysis Effect" from the book:

Barber, T. X. (1976). Pitfalls in Human Research: Ten Pivotal Points. Pergamon Press.

Episode 15: Novum Crisi Replicati Aug 25, 2023

In this episode, we discuss the replication crisis in psychology which has been an important topic of discussion for the last decade. We revisit some key events from the start of the replication crisis, such as the publication of Daryl Bem's studies on precognition, the paper False Positive Psychology, and the Reproducibility Project and share personal anecdotes about how it was to live through the replication crisis.

Shownotes:

Bem, D. J. (2011). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology, 100(3), 407–425. https://doi.org/10.1037/a0021524

Ritchie, S. J., Wiseman, R., & French, C. C. (2012). Failing the Future: Three Unsuccessful Attempts to Replicate Bem’s ‘Retroactive Facilitation of Recall’ Effect. PLOS ONE, 7(3), Article e33423. https://doi.org/10.1371/journal.pone.0033423

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant. Psychological Science, 22(11), 1359–1366. https://doi.org/10.1177/0956797611417632

John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532. https://doi.org/10.1177/0956797611430953

Fiedler, K., & Schwarz, N. (2016). Questionable Research Practices Revisited. Social Psychological and Personality Science, 7(1), 45–52. https://doi.org/10.1177/1948550615612150
- NOTE: Daniel says in the podcast the paper below is by Fiedler and Strack - but it is by Fiedler and Schwarz.

Ebersole, C. R., Mathur, M. B., Baranski, E., Bart-Plange, D.-J., Buttrick, N. R., Chartier, C. R., Corker, K. S., Corley, M., Hartshorne, J. K., IJzerman, H., Lazarević, L. B., Rabagliati, H., Ropovik, I., Aczel, B., Aeschbach, L. F., Andrighetto, L., Arnal, J. D., Arrow, H., Babincak, P., …

Nosek, B. A. (2020). Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability. Advances in Methods and Practices in Psychological Science. https://doi.org/10.1177/2515245920958687

Luttrell, A., Petty, R. E., & Xu, M. (2017). Replicating and fixing failed replications: The case of need for cognition and argument quality. Journal of Experimental Social Psychology, 69, 178–183. https://doi.org/10.1016/j.jesp.2016.09.006

Simons, D. J., Shoda, Y., & Lindsay, D. S. (2017). Constraints on Generality (COG): A Proposed Addition to All Empirical Papers. Perspectives on Psychological Science, 12(6), 1123–1128. https://doi.org/10.1177/1745691617708630

Simonsohn, U. (2015). Small Telescopes Detectability and the Evaluation of Replication Results. Psychological Science, 26(5), 559–569. https://doi.org/10.1177/0956797614567341

Episode 14: Capax Mentis Aug 13, 2023

In this episode we reflect on the role of intelligence in scientist. How much does intelligence matter in science, and which other characteristics might play a role in doing good science? Do scientist need to be extremely intelligent or can anyone do science? And what is the role of stupidity in science?

Capax Mentis roughly translates to "capacity of mind."

Smriti stupidly messed up her audio so the quality isn't great. Apologies!

Shownotes

Schwartz, M. A. (2008). The importance of stupidity in scientific research. Journal of Cell Science, 121(11), 1771. https://doi.org/10.1242/jcs.033340
Bernal, J. D. (1939). The Social Function Of Science. Routledge.
Paul Medawar: Advice to a Young Scientist
Feynman talking about the uncomfortable feeling of confusion
A good scientist always keeps learning – Nobel Laureate Peter Doherty
Flatland (1884) by Edwin Abbott Abbott
A zero-order correlation simply refers to the correlation between two variables (i.e., the independent and dependent variable) without controlling for the influence of any other variables. Essentially, this means that a zero-order correlation is the same thing as a Pearson correlation.

Prologus 14: How can I tell if I am cut out to be a scientific research worker? (P. B. Medawar) Aug 04, 2023

As prologue to the next episode on how smart one needs to be to be a scientist, we present a reading of chapter 2 "How can I tell if I am cut out to be a scientific research worker?" by Peter B. Medawar from his 1979 book 'Advice to a young scientist'. Our next episode was inspired by the section "Am I brainy enough to be a scientist?"

https://www.google.nl/books/edition/Advice_To_A_Young_Scientist/3fg3DgAAQBAJ

Episode 13: Chmess Jul 28, 2023

In this episode we discuss Daniel Dennett's distinction between chess, or research worth doing, and 'chmess,' research not worth doing. We discuss ways to determine whether our research is chess or chmess, and how to avoid being sucked into lines of research we don't particularly care about.

Shownotes

Dennett, D. C. (2006). Higher-order truths about chmess. Topoi, 25, 39–41.
Dunnette, M. D. (1966). Fads, fashions, and folderol in psychology. American Psychologist, 21(4), 343.
- Folderol means 'a useless ornament or accessory', and is used to indicate something is 'nonsense'.
Dweck, C. S. (2022). Mindsets: From bathtubs to hot beliefs to social change. In Kassin, S. (Ed.) Pillars of Social Psychology: Stories and Retrospectives, 213–219. Cambridge University Press.
The Kardashian Index

Prologus 13: Chmess (D. C. Dennett) Jul 28, 2023

As prologue to the next episode on Chmess, we present a reading of a paper by Daniel C. Dennett:

Dennett, D. C. (2006). Higher-order truths about chmess. Topoi, 25, 39–41.

Episode 12: Virtutes Vocationalis Jul 14, 2023

In today's episode, we discuss vocational virtues⸺scientific principles that should guide the behavior of scientists. We discuss whether we agree with values put forth by numerous scientists, including Ivan Pavlov, Peter Medawar, Santiago Ramón y Cajal, Barry Schwartz, among others, and share our own.

Correction: At 56:24, Smriti mentions the book This is Biology, which is written by Ernst Mayr, not E.O. Wilson.

Shownotes

Pavlov, I. (1936). Bequest of Pavlov to the Academic Youth of His Country. Science, 83(2155), 369–370. https://doi.org/10.1126/science.83.2155.369
Schwartz, B. (2022). Science, scholarship, and intellectual virtues: A guide to what higher education should be like. Journal of Moral Education, 51(1), 61-72.
Robert T. Pennock: An Instinct for Truth: Curiosity and the Moral Character of Science
Merton, R. K. (1942). A Note on Science and Democracy. Journal of Legal and Political Sociology, 1, 115–126.
Schwartz, B. (1990). The creation and destruction of value. American Psychologist, 45(1), 7.
Paul Medawar: Advice to a Young Scientist
Ernst Mayr. This is Biology
Santiago Ramón y Cajal: Advice for a Young Investigator
Bernal, J. D. (1939). The Social Function Of Science. Routledge.
Weber, M. (1917/1958). Science as a Vocation. Daedalus, 87(1), 111–134.
E.O. Wilson: Letters to a Young Scientist

Prologus 12: Science, Scholarship, and Intellectual Virtues (B.J. Schwartz) Jul 07, 2023

As prologue to the next episode on vocational virtues, we present a reading of a paper by Barry Schwartz:

Schwartz, B. (2022). Science, scholarship, and intellectual virtues: A guide to what higher education should be like. Journal of Moral Education, 51(1), 61-72. https://doi.org/10.1080/03057240.2020.1772211 (Published online: 19 Jun 2020)

You can read the paper here.

An episode from Smriti's previous podcast with Paul Connor where they discussed the paper with Barry can be found here.

Episode 11: Inquisitionis Vastum Jun 30, 2023

In this episode, we discuss the topic of research waste. We discuss what it is it that is being wasted and whether we waste fewer scientific resources and talent through coordination, team science, and better planning.

Shownotes

Bacon, New Atlantis, 1626: https://www.gutenberg.org/files/2434/2434-h/2434-h.htm
Dennett, D. C. (2006). Higher-order truths about chmess. Topoi, 25(1–2), 39–41. https://doi.org/10.1007/s11245-006-0005-2
Chalmers, I., & Glasziou, P. (2009). Avoidable waste in the production and reporting of research evidence. The Lancet, 374(9683), 86–89.
Mao's Hundred Flowers Campaign: https://en.wikipedia.org/wiki/Hundred_Flowers_Campaign
Glasziou, P., & Chalmers, I. (2018). Research waste is still a scandal—An essay by Paul Glasziou and Iain Chalmers. BMJ, 363, k4645. https://doi.org/10.1136/bmj.k4645
AltmanDG. The scandal of poor medical research. BMJ 1994;308:283-4. 10.1136/bmj.308.6924.283 8124111
Bernal, J. D. (1939). The Social Function Of Science. Routledge. http://archive.org/details/in.ernet.dli.2015.188098
Duckworth, A. L., & Milkman, K. L. (2022). A guide to megastudies. PNAS Nexus, 1(5), pgac214. https://doi.org/10.1093/pnasnexus/pgac214
Almaatouq, A., Griffiths, T. L., Suchow, J., Whiting, M. E., Evans, J., & Watts, D. J. (2022). Beyond Playing 20 Questions with Nature: Integrative Experiment Design in the Social and Behavioral Sciences. PsyArXiv. https://doi.org/10.31234/osf.io/anjkm

Prologus 11: The Efficiency of Scientific Research (J.D. Bernal) Jun 23, 2023

In this bonus episode, Daniël reads Chapter 5 of John Desmond Bernal’s book The Social Function of Science, entitled The Efficiency of Scientific Research in preparation of our upcoming podcast episode on research waste.

You can read The Social Function of Science by Bernal at the Internet Archive: https://archive.org/details/in.ernet.dli.2015.188098

Episode 10: Probatio Significativitatis Hypothesis Nihili Jun 16, 2023

Shownotes

Wilson, E. B. (1923). The Statistical Significance of Experimental Data. Science, 58(1493), 93–100. https://doi.org/10.1126/science.58.1493.93

van Dongen, N. N. N., & van Grootel, L. (2021). Overview on the Null Hypothesis Significance Test. https://doi.org/10.31234/osf.io/hwk4n

Stark, P. B., & Saltelli, A. (2018). Cargo‐cult statistics and scientific crisis. Significance, 15(4), 40-43.

Uygun Tunç, D., Tunç, M. N., & Lakens, D. (2023). The epistemic and pragmatic function of dichotomous claims based on statistical hypothesis tests. Theory & Psychology, 09593543231160112. https://doi.org/10.1177/09593543231160112

Bakan, D. (1966). The test of significance in psychological research. Psychological Bulletin, 66(6), 423–437. https://doi.org/10.1037/h0020412

Cohen, J. (1990). Things I have learned (so far). American Psychologist, 45(12), 1304–1312. https://doi.org/10.1037/0003-066X.45.12.1304

Cohen, J. (1994). The earth is round (p < .05). American Psychologist, 49(12), 997–1003. https://doi.org/10.1037/0003-066X.49.12.997

Cohen, J. (1995). The earth is round ( p

Episode 9: Praeiudicium Publicandi Jun 02, 2023

In this episode, we discuss the issue of publication bias. We discuss issues like: Do we learn anything from null results, given the current state of research practices? Is poorly done research still worth sharing with the scientific community? And how can we move toward a system where null results are informative and worth publishing?

Shownotes

Bones, A. K. (2012). We Knew the Future All Along Scientific Hypothesizing is Much More Accurate Than Other Forms of Precognition—A Satire in One Part. Perspectives on Psychological Science, 7(3), 307–309. https://doi.org/10.1177/1745691612441216

Carter, E. C., & McCullough, M. E. (2014). Publication bias and the limited strength model of self-control: Has the evidence for ego depletion been overestimated? Frontiers in Psychology, 5. https://doi.org/10.3389/fpsyg.2014.00823

Greenwald, A. G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82(1), 1–20. https://doi.org/10.1037/h0076157

Fidler, F., Singleton Thorn, F., Barnett, A., Kambouris, S., & Kruger, A. (2018). The epistemic importance of establishing the absence of an effect. Advances in Methods and Practices in Psychological Science, 1(2), 237-244.

Pickett, J. T., & Roche, S. P. (2017). Questionable, Objectionable or Criminal? Public Opinion on Data Fraud and Selective Reporting in Science. Science and Engineering Ethics, 1–21. https://doi.org/10.1007/s11948-017-9886-2

Scheel, A. M., Schijen, M. R. M. J., & Lakens, D. (2021). An Excess of Positive Results: Comparing the Standard Psychology Literature With Registered Reports. Advances in Methods and Practices in Psychological Science, 4(2), 25152459211007468. https://doi.org/10.1177/25152459211007467

Sterling, T. D. (1959). Publication Decisions and Their Possible Effects on Inferences Drawn from Tests of Significance—Or Vice Versa. Journal of the American Statistical Association, 54(285), 30–34. JSTOR. https://doi.org/10.2307/2282137

The FDA Trial tracker to see which trials have not shared their results: https://fdaaa.trialstracker.net/

Episode 8: Scientia Cultus Sarcinarum May 19, 2023

In this episode, we discuss physicist Richard Feynman’s famous speech ‘Cargo Cult Science,’ which refers to work that has all the affectations of science without the actual application of the scientific method. We also discuss topics like: What is pathological science? How might cargo cult science and pathological be different from pseudo-science? How do we know whether or not we’re in a cargo cult, and what can we do to make sure we're not fooling ourselves?

Shownotes

Cargo Cult Science (Feynman, 1974)

Gergen, K. J. (1973). Social psychology as history. Journal of personality and social psychology, 26(2), 309–320.

Langmuir, I. (1989). Pathological science. Research-Technology Management, 32(5), 11-17.

Sabine Hossenfelder. No one in physics dare say so, but the race to invent new particles is pointless. The Guardian.

Young, P. T. (1932). Relative food preferences of the white rat. Journal of Comparative Psychology, 14(3), 297.

Young, P. T. (1941). The experimental analysis of appetite. Psychological Bulletin, 38(3), 129.

Prologus 8: Cargo Cult Science (R.P. Feynman) May 12, 2023

In this bonus episode, we present a reading of the famous speech by physicist Richard Feynman on "science that isn't science," Cargo Cult Science, which will be the topic of the next episode. Enjoy.

Episode 7: Corpora Regulatoria May 05, 2023

In this episode we discuss regulatory bodies their influence on the generation and dissemination of knowledge. Should regulatory bodies have the authority to affect the topics and methods of science? Is more highly regulated research actually better? And should we just give up on our own lines of research and become potato researchers?

Shownotes

Kitcher, P. (2001). Science, truth, and democracy. Oxford University Press.

Indian Council of Agricultural Research (ICAR)-Central Potato Research Institute (CPRI)

Emergence of the NIH

DARPA SCORE

Lakens, D. (2023). Is my study useless? Why researchers need methodological review boards. Nature, 613(7942), 9-9.

PSA: Legate, N., Ngyuen, T. V., Weinstein, N., Moller, A., Legault, L., Vally, Z., ... & Ogbonnaya, C. E. (2022). A global experiment on motivating social distancing during the COVID-19 pandemic. Proceedings of the National Academy of Sciences, 119(22).

Episode 6: Consentio Apr 21, 2023

In this episode, we discuss the importance of consensus in science, both as means of establishing true knowledge and for determining which research questions might be worth pursuing. We also discuss barriers to reaching consensus and the different frameworks that are currently employed for trying to reach consensus among important stakeholders.

Shownotes

The Popper quote is from: Popper, K. R. (1959). The logic of scientific discovery. Routledge.

The Polanyi quote is from: Polanyi, M. (1950). Freedom in Science. Bulletin of the Atomic Scientists, 6(7), 195–198. https://doi.org/10.1080/00963402.1950.11461263

Planck's Principle: A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die and a new generation grows up that is familiar with it.

The Many Smiles collaboration: Coles, N. A., et al., (2020). The Many Smiles collaboration: A multi-Lab foundational test of the facial feedback hypothesis. Nature Human Behaviour. https://doi.org/10.31234/osf.io/cvpuw

Paul Meehl's 50 year rule: Meehl, P. E. (1992). Cliometric metatheory: The actuarial approach to empirical, history-based philosophy of science. Psychological Reports, 71, 339–339.

Mulkay, M. (1978). Consensus in science. Social Science Information, 17(1), 107-122.

Deliberative Polling

Laudan, L. (1986). Science and values: The aims of science and their role in scientific debate. Univ of California Press.

Episode 5: Insanabile Scribendi Cacoethes Apr 07, 2023

In this episode, we discuss the insatiable itch to publish, starting with a quote from 1927 by sociologist Clarence Case on the dictum “Publish or perish.” We discuss ways in which individual goals to publish conflict with the broader scientific goal of producing useful knowledge. We also question the assumptions behind the notion that publishing less would be beneficial for science.

Shownotes

Case, C. M. (1927). Scholarship in sociology. Sociology and Social Research, 12, 323-340 (Publish or perish)
Phaf, R. H. (2020). Publish less, read more. Theory & Psychology, 30(2), 263–285. https://doi.org/10.1177/0959354319898250

The term "insanabile scribendi cacoethes" comes from: Merton, R. K. (1968). The Matthew effect in science: The reward and communication systems of science are considered. Science, 159(3810), 56-63.

Episode 4: Eminentia Mar 24, 2023

In this episode, we discuss the role of eminence in science. What ask questions like: What makes scientists eminent? What role does eminence play in science? Can eminence be spread across scientific teams instead of individuals? And how can we recognize and applaud scientists for their contributions, while avoiding conferring too many benefits on scientists who do become eminent?

Shownotes

Eminent psychologists of the 20th century

Intel - Our rock stars aren't like your rock stars

Episode 3: Confirmatio Praeiudicia Mar 10, 2023

In our third episode, we discuss confirmation bias, which affects not only how scientists generate and test their own hypotheses, but also how they evaluate the scientific evidence presented by others. We discuss guardrails against confirmation bias that are already in place, and others that could potentially improve scientific practice if adopted.

Shownotes

Wason, P. C. (1960). On the failure to eliminate hypotheses in a conceptual task. Quarterly Journal of Experimental Psychology, 12(3), 129-140.
Nickerson, R. S. (1998). Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology, 2(2), 175-220.
Mellers, B., Hertwig, R., & Kahneman, D. (2001). Do frequency representations eliminate conjunction effects? An exercise in adversarial collaboration. Psychological Science, 12(4), 269-275.
Coles, N. A., March, D. S., Marmolejo-Ramos, F., Larsen, J. T., Arinze, N. C., Ndukaihe, I. L., ... & Liuzza, M. T. (2022). A multi-lab test of the facial feedback hypothesis by the many smiles collaboration. Nature Human Behaviour, 1-12.
Dutilh, G., Sarafoglou, A., & Wagenmakers, E. J. (2021). Flexible yet fair: Blinding analyses in experimental psychology. Synthese, 198(23), 5745-5772.

Sarafoglou, A., Hoogeveen, S., & Wagenmakers, E. J. (2023). Comparing analysis blinding with preregistration in the many-analysts religion project. Advances in Methods and Practices in Psychological Science, 6(1), 25152459221128319.
Faster-than-light neutrino anomaly

Episode 2: Scepticismus Mar 03, 2023

In our second episode, we discuss the role of skepticism in science, a topic that relates closely to the title of our podcast. Given that the scientific enterprise is essentially an exercise in organized skepticism, how can we maintain a healthy amount of skepticism while also ensuring that scientists don't slip into cynicism or nihilism?

Shownotes

Opening quote by Imre Lakatos from Science and Pseudoscience. Hear it from the man himself.
Ego depletion
Ioannidis, J. P. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124. https://doi.org/10.1371/journal.pmed.0020124
Wacholder, S., Chanock, S., Garcia-Closas, M., El Ghormli, L., & Rothman, N. (2004). Assessing the probability that a positive report is false: an approach for molecular epidemiology studies. Journal of the National Cancer Institute, 96(6), 434-442. DOI: 10.1093/jnci/djh075

Quote by Debra Mayo. Original reference: Mayo, D. G. (2018). Statistical inference as severe testing: How to get beyond the statistics wars. Cambridge University Press.

Episode 1: Motivus Feb 24, 2023

In our first episode, we discuss a quote from the preface to The Instauratio Magna (of which Novum Organum is a part), in which Bacon claims that scientists should be motivated to do science for the betterment of mankind, and not for personal motives like fame, fortune, or even fun.

Here is the tweet (by Heidi Seibold) on academia not being aligned with good scientific practices.

An unedited transcript of the episode can be found here.

Episode 0: Introductio Feb 19, 2023

In this introductory episode, Daniël and Smriti share which podcasts they like, why they are starting their own, and how their connection to each other is also tied to podcasting. They also talk about the theme of the podcast, which is inspired by Francis Bacon’s delineation of the scientific method 400 years ago.

Our TOPPODCAST Picks

Follow Us

Stay Connected

Related Podcasts

Links

Stay Connected