Journal Club: "ReproducibiliTea: Big Little Lies: p-hacking and effective countermeasures"

The Journal Club: “ReproducibiliTea” at LMU Munich continues with its fifth meeting to discuss an interesting paper about p-hacking strategies and effective countermeasures. The organisation team is happy to announce that the paper’s author, Angelika Stefan, will give an introductory talk to this session and participate in the open discussion afterwards:

When: Friday, 10.06.2022, 2:30pm
Where: Via Zoom (Meeting ID: 917 8852 1090 / Password: Replicate)
Language: English
Paper: Stefan, A., & Schönbrodt, F. D. (2022, March 16). Big Little Lies: A Compendium and Simulation of p-Hacking Strategies. https://doi.org/10.31234/osf.io/xy2dk

Participants can use the following key questions to prepare for the discussion:

• What do you think: How prevalent are p-hacking-strategies in your field/community (e.g. in student thesis, in published research)?
• Which potential solutions (described on pages 37-45) do you find the most convincing?
• Which questions about QRPs do you still have? Which questions do you think are not answered by the scientific community, yet (both empirical questions e.g. specific prevalence rates and theoretical questions)?

You can find a short abstract for the introductory talk by Angelika Stefan (University of Amsterdam) below:

Big Little Lies: p-hacking and effective countermeasures

A decade ago, John et al. (2012) found a high prevalence of questionable research practices (QRPs) among psychological researchers. This result was confirmed by many other surveys among researchers in the social sciences. QRPs range from accidental mistakes to clear scientific misconduct and its prevalence undermined the credibility of the social sciences. One of the most prominent QRPs is p-hacking which includes a broad range of different strategies to render non-significant testing results significant. But if QRPs and especially p-hacking are widespread, as previous research suggests: How problematic are the different strategies in fact? How big is the impact of different p-hacking-strategies in detail? And what countermeasures are most effective?

In this talk, Angelika Stefan will give a short, general introduction to QRPs. She will discuss p-hacking in more detail and distinguish it from other QRPs. Then, she will show some examples of simulations from this session’s paper that examined the impact of different p-hacking strategies. Based on the simulations’ results, possible countermeasures and solutions will be introduced and discussed.

About the speaker:

Angelika Stefan is a researcher and doctoral candidate at the University of Amsterdam and belongs to the Department of Psychological Methods within the Faculty of Social and Behavioural Sciences. She finished her Masters in Economic, Organizational and Social Psychology in 2018 at LMU Munich. Her work focuses on the application of Bayesian Statistics and the general dealing with methodological flexibility in the Social Sciences. She is an expert for Open Science, led many workshops on Open Science-related topics and organizes the ReproducibiliTea at her university in Amsterdam.

Concept of the Journal Club

The interdisciplinary journal club “ReproducibiliTea” is launched every 3-4 weeks and addresses main questions and current topics concerning Open Science and the credibility of research findings. The idea - originated from Great Britain - was already developed in many other countries using the same name. During on-site meetings tea, juice and small snacks are provided to live up to the slogan. However, some meetings are held online due to organisational reasons. No matter where the sessions take place, everybody is welcome to bring their own food and drinks and thus connect the journal club with their lunch break.

The Journal Club consists of open talk sessions for discussing Open Science, reproducibility, and other related issues and is explicitly open to all disciplines and institutions to create the most interesting interchange. Each session starts with a short talk about the paper and then continues with an open discussion in which you can contribute your own thoughts and questions.

During a longer break between 2020 and 2021, the concept was slightly revised, and the organisation team decided to invite the speaker(s) of the respective paper that is discussed in each session. The author herself/himself starts with the introductory talk including a short summary of the paper. Afterwards, there is enough time for direct interaction with the author(s) and, as before, an open discussion with all of the participants.

You are warmly welcome to join the next meetings. You can simply send an e-mail to reproducibilitea.munich@gmail.com to join the mailing list to get information about the next meetings and chosen papers. For each session it would be helpful to have read the announced paper, but there is no obligation to do so.

The organisation team is looking forward to seeing you at one of the next Journal Club sessions.

Organisation team:

• Laura Goetz, Medical Student at LMU Munich
• Stephan Nuding, Psychology Student at LMU Munich
• Leonhard Schramm, Psychology Student at LMU Munich