Workshop: "Making the black box transparent: Multiverse analysis and visualizations in R"

When: March 15th, 2024, 9:00-12:00
Where: Leopoldstraße 13, room 2201
Registration: https://www.pretix.osc.lmu.de/lmu-osc/multiverse-W/1
Material: https://osf.io/7qr4x/

Abstract: To describe the multitude of researchers’ methodological choices when analyzing data, Gelman and Loken (2013) coined the term “garden of forking paths”. Multiple defensible alternative options are available for selection throughout the study design, data preprocessing and data analysis workflow. In line with a traditional analysis approach, researchers will select one sole workflow of defensible options and often do not specifically disclose how the decisions were made. However, a multiple comparison problem occurs even if only one constructed dataset following one single workflow of decisions is used for statistical inferences. The reason is that theoretically a large variety of workflows is possible, but researchers do not correct their hypothesis tests for potential comparisons that they did not explicitly carry out.

Recently, statistical approaches have been proposed to explore and integrate multiple inferential results accumulated along forking paths. Multiverse analysis (Steegen et al., 2016) refers to repeated hypothesis testing on the multitude of datasets resulting from different defensible decisions regarding, for example, variable selection, categorization, transformation, outlier selection, etc. After combining all potential choices by simple rules of combinatorics, contradictory combinations are eliminated. Datasets created on the basis of the remaining set of combinations are then submitted to statistical analyses along a looped pipeline. Results provide as many outcomes of statistical tests as datasets created in the multiverse. These can then be visualized to understand how statistical conclusions depend on methodological decisions or they can be statistically integrated.

In this workshop, unexperienced attendees who attended the lecture on the previous day will be guided through an implementation example of multiverse analyses on the R Software for Statistical Computing. There will be time for exercises and discussions. Workshop materials and necessary R packages will be shared in advance and participants are expected to join the workshop with their own laptops.

Instructor: Andrea Hildebrandt, Professor for Psychological Methods and Statistics at the Carl von Ossietzky Universität Oldenburg, Department of Psychology and Cassie Short, Postdoc at the Psychological Methods and Statistics Lab of the Carl von Ossietzky Universität Oldenburg, Department of Psychology

Participants: Scientists from any discipline who apply statistical models in the framework of regression analysis, without prior experience with multiverse analysis implementations.

Prerequisite: Basic statistics (including GLM) and statistical programming skills in R are required. R software must be installed prior to the workshop. Participants must also have attended the lecture on the previous day: https://www.pretix.osc.lmu.de/lmu-osc/multiverse-L/

Logistics: The event will be held in person, in English. Participation in the workshop is free and open to all. Priority will be given to, in order, members of Departments sponsoring the Open Science Center (Department of Psychology, Business Administration, Biology), other LMU members, and non-LMU members. Non-members will receive confirmation whether their attendance is possible at least 2 days before the workshop.