Department of Psychology/Methods of Plasticity Research

Nicolas LangerUniversity of Zurich

Department of PsychologyMethods of Plasticity Research (MethLab)

Center for Reproducible Science UZH

Hands on: Preregistration platforms and templates

Preregistration

The Four Horsemen of the Reproducibility Apocalypse

2

https://en.wikipedia.org/wiki/Four_Horsemen_of_the_Apocalypse#/media/File:Apocalypse_vasnetsov.jpg

Low powerHARKing P-hacking

Publication bias

• HARKing (hypothesizing after results are known) involves generating a hypothesis from the data and then presenting it as a priori.

• Low statistical power increases the chances of missing true discoveries and reduces the likelihood that obtained positive effects are real.

• P-hacking (exploiting researcher degrees of freedom) manifests in two general forms: collecting data until analyses return statistically significant effects, and selectively reporting analyses that reveal desirable outcomes.

• Publication bias occurs when journals reject manuscripts on the basis that they report negative or undesirable findings.

Research Cycle

SNF, Horizonte, No 130; September 2016 3https://www.horizonte-magazin.ch/2016/08/31/

The Four Horsemen of the Reproducibility Apocalypse within the Research Cycle

DATA ANALYSIS--

INTERPRETATION

PUBLICATION--

PEER REVIEW

LITERATUREREVIEW

HYPOTHESISGENERATION

--DESIGN STUDY

DATACOLLECTION

Low Power

p-hacking

HAR

Kin

g

Publication biasJohn et al. (2012)

John

et a

l. (2

012)

Kerr

(199

8)

Cohen (1962)Sedlmeier & Gigerenzer (1989)Graves (2001)

Fanelli (2010)

Chambers, 2017 4

The Influence of Open Science

The 6 principles of Open Science • prevent low power• no HARKing• no/less p-hacking• no publication bias• incentives for replication studies• archiving of data & materials

Open Science Tools:- Power Analysis- Preregistration- Open Code & Data

Replication

5

https://de.wikiversity.org/wiki/Wikiversity:Fellow-Programm_Freies_Wissen%C2%B4/Open_Science

Preregistration

Idea:Researchers decide on research plan (hypotheses, experimental procedures, and main analyses) before data collection, in order to exclude the possibility of a-posteriori modification.

- time-stamped & frozen

- Preregistration contains unique URL

2-stage publication process of a research project:

1. Preregistration

2. Final publication

register-your-project

Research Colloquia:1. Synopsis of MA thesis2. Presentation of the results

6

Why Preregistration?

• Preregistration shifts the focus of research more towards the research question, the correct experimental procedures and the methods. Thus, the outcome of the experiment is less important.

• enables the distinction between exploratory and confirmatory research

• promotes transparent and reproducible research (credibility of research)

• reduces/prevents:- HARKing- p-hacking - publication bias (Registered Reports)

Preregistration

7

Various forms of preregistration

Preregistration Templates: https://osf.io/zab38/wiki/home/

Other preregistration forms:• Replication Recipe Preregistration (Brandt et al., 2013)• Replication Recipe Post-Completion (Brandt et al., 2013)• Preregistration in Social Psychology (van't Veer & Giner-Sorolla, 2016)• Qualitative Research Preregistation• fMRI Preregistration Template

Open-Ended Registration: no requirements. You will be asked to write a summary description of the project.

OSF-Standard Pre-Data Collection Registration: Questions: 1. Data already collected? 2. have you already looked at data?

AsPredicted.org: Online input mask with 9 questions: hypothesis; dependent variables; conditions; Data analysis; outliers & exclusion criteria; sample size (power analysis); type of study

OSF Preregistration: 25 subject areas that need to be answered. Preregistration Prize ($1,000 prize, by December 31, 2018)

Registered Report: Preregistration is reviewed via a journal (peer-review process) and labelled with “IPA” (in-principle acceptance). If the study is carried out correctly, it is guaranteed that it will be published, regardless of the result. (see also https://cos.io/rr). 125 journals already participate: https://docs.google.com/spreadsheets/d/1D4_k-8C_UENTRtbPzXfhjEyu3BfLxdOsn9j-otrO870/edit#gid=0

common to all: carry out before data collection!

Inpu

t req

uire

men

ts

8https://clinicaltrials.gov/Alternatives: https://www.who.int/ictrp/network/primary/en/

Possible Content of PreregistrationsIntroduction

– Description of the research questions

– Review of the existing literature (priors, effect sizes)

Methods

– Hypothesis

– Statistical power analysis (effect sizes should be justified: literature, Warning:publication bias: potentially cut published effect size in half (Franco et al., 2015))

– Description of the sample (inclusion and exclusion criteria, outliers)

– Experimental procedure (detailed enough so replication is possible)

– Description of data collection (exclusion due to technical problems)

– Description of the analysis plan (planned data preprocessing & statistics, covariates, correction for multiple comparisons, quality checks (e.g. ceiling effects) )

– exploratory or confirmatory approach

time stamp & frozen

possible pilot-data 9

An Example of Preregistration in AsPredicted.org

https://aspredicted.org/nfj4s.pdf 10

https://osf.io/cp98d/Materials for preregistration demo:

www.aspredicted.orgby Uri Simonsohn, Joe Simmons, and Leif Nelson

Open Science Framework (OSF)

OSF is a free, open source project management platform (repository) that supports scientists throughout a project cycle.

• Repository to store & share: data, code, materials,....

• Collaborations-platform

• OSF helps scientists to structure (partly) private or public research projects, to make them quotable and to give access to the research community.

• enables to preregister a project- including code, materials, data

www.osf.io

11

https://clinicaltrials.gov/Alternative:

12

https://osf.io/prereg/

Preregistration in OSF

http://help.osf.io/m/registrations/l/546603-create-a-preregistration

https://osf.io/rr/Registered Report in OSF

https://osf.io/n78ae/Demo Project:

Preregistration in OSF

13

Examples of existing preregistrations in OSF

https://osf.io/e6auq/wiki/Example%20Preregistrations/

QUESTIONS

14

https://goo.gl/forms/hZfEK4vt6dIzjPFu1

Participate in our preregistered survey

Q & AWhy is it important to distinguish between exploratory and confirmatory analyses?

• Mistaking exploratory results for confirmatory tests leads to misplaced confidence in the replicability of reported results.

• Failing to appreciate the difference can lead to overconfidence in post hoc explanations (postdictions) and inflate the likelihood of believing that there is evidence for a finding when there is not. (overestimated effect-sizes)

• Presenting postdictions as predictions can increase the attractiveness and publishability of findings by falsely reducing uncertainty. Ultimately, this decreases reproducibility.

• Preregistration does not favor prediction over postdiction; its purpose is to make clear which is which.

• In prediction (cofirmatory), data are used to confront the possibility that the prediction is wrong. In postdiction (exploratory), the data are already known and the postdiction is generated to explain why they occurred. Formally speaking, analyses conducted on the data that are not part of the preregistration inform postdiction.

• Testing predictions is vital for establishing diagnostic evidence for explanatory claims. Testing predictions assesses the uncertainty of scientific models by observing how well the predictions account for new data.

• With preregistration, prediction is achieved because selection of tests is not influenced by the observed data, and all conducted tests are knowable. 15Nosek et al., 2018

Q & AHow to deal with deviations from preregistrations?

Sometimes the best laid plans are difficult to achieve. What should we do:

Deviations do not necessarily rule out testing predictions effectively. If the outcomes have not yet been observed, the researcher can document the changes to her preregistration without undermining diagnosticity.

The researcher can transparently report changes that were made and why. Most of the design and analysis plan is still preserved, and deviations are reported transparently, making it possible to assess their impact.

16Nosek et al., 2018

Q & AHow to deal discovery of assumption violations during analysis

Someone discovers that the distribution of one of her variables has a ceiling effect and another is not normally distributed. These violate the assumptions of her preregistered tests.

• For some kinds of analysis, it is possible to define stages and preregister incrementally. For example, a researcher could define a preregistration that evaluates distributional forms of variables to determine data exclusions, transformations, and appropriate model assumptions that do not reveal anything about the research outcomes. After that, the researcher preregisters the model most appropriate for testing the outcomes of interest.

• Another method is to preregister a decision tree. The decision tree defines the sequence of tests and decision rules at each stage of the sequence. For example, the decision tree might specify testing a normality assumption and, depending on the outcome, selection of either a parametric or nonparametric test.

• A final option is to establish standard operating procedures (SOPs) that accompany one or many preregistrations.

17Nosek et al., 2018

Q & AWhat if data are preexisting?

• The extent to which testing predictions is possible on preexisting data depends on whether decisions about the analysis plan are blind to the data.

• e.g. an economist can create predictions of government data that exist but have not been released.

• This transparency provides insight about potential biasing influences for, at minimum, subjective assessment of credibility.

• It is also possible to embrace discovery in a single large study and have some hypothesis testing of interesting possibilities. In a process known as cross-validation, the dataset is split in two. One part is used for exploratory analysis to develop the models and predictions; the other part is sealed until exploration is complete (54). Sealing a dataset and preregistering the outcomes of the discovery process before unsealing converts postdictions from the initial dataset to predictions for the holdout dataset.

Longitudinal Studies

• Each year, some variables are newly observed. Preregistrations in advance of the new observations gain some benefits of blinding.

18Nosek et al., 2018

Q & AEmbargo in OSF:• Registrations can be public or embargoed for up to four years. Embargoed registrations will

be made public automatically when the embargo expires.

• Optional "Embargo End Date” can be chosen (calendar)

• When preregistration is submitted it is initially in a "pending" state: • When a project is registered, all administrators on the project are notified via email and given the chance to approve or cancel the

preregistration.

• The preregistration goes into effect:

– either when all administrators have approved the preregistration

– or when 48 hours have passed (whichever comes first)

• If one administrator disapproves the preregistration, the preregistration will be cancelled. You will be notified via email if an administrator disapproves the preregistration.

Embargo In AsPredicted.org: • If all participating authors approve the preregistration, it is saved and stays private until

an author acts to make it public, or it remains private forever.

19

Q & AWithdraw a Registration in OSFWithdrawing a registration removes its contents from OSF. Withdrawn registrations will maintain basic metadata (e.g. title, contributors, DOI, etc.) about the registration, as well as a justification for its withdrawal, but the registration form, files, and other content will be removed.

Please note that registration withdrawal is irreversible.

Justification for Withdrawal can be made

Withdraw In AsPredicted.org: • Pre-registrations are permanent

• Use “I’m just trying things out” button (self-destruction within 24 hours)

• Keep it private

20

Q & ACan I not simply submit multiple similar preregistrations (AsPredicted.org)AsPredicted.org has several algorithms that routinely run analyses on private and public submissions seeking to identify possible abuse and errors. When a pattern deserving closer attention is identified, members of the AsPredicted team are notified and at least one member reads portions or the totality of the suspected submissions;

If no obvious benign explanation for the activity flagged by the algorithm is identified, that member of the AsPredicted team emails the participating authors asking for an explanation. If no satisfactory explanation is provided, a disclaimer may be added to the public .pdf (if already public), and the corresponding private pre-registrations may be blocked from being made public. Moreover, suspicious activity may lead AsPredicted.org to block involved authors from creating new pre-registrations. So far, all instances flagged by the algorithms have had benign explanations.

Nosek et al., 2018:

“Finally, the website https://aspredicted.org/ provides a simple form for preregistration, but it is not itself a registry because users can keep their completed forms private forever and selectively report preregistrations. However, researchers can post completed forms to a registry to meet the preservation and transparency standards.”

21

Q & A

Length of preregistration on AsPredicted.org

• The 3200 character limit is a recommendation, submissions with more characters are allowed. The system does enforce a limit based on the number of pages the final .pdf document would have (the link between number of characters and pages is imperfect due to various factors).

In particular: If the pre-registration would be 1 page long, it can be submitted as is.If the pre-registration would be 2 pages long, you will get a warning message and be asked to consider shortening it, but will be allowed to proceed.If the pre-registration would be 3 pages long, you will not be allowed to submit it.

22

Q & APreregistration of projects in OSF:

23

How to distinguish between projects and registrations:1. A watermark indicating that the registration is read-only2. A banner indicating that the registration is frozen, and linking back to the project

Materials, Analysis Code and Data can be preregistered in OSF.

Q & A

Effects of Preregistration

24

Q & AHistory of Preregistration (preregistration is not a new concept)

25

Results of the survey

Statements 1, 2, 3, and 4 assign probabilities to parameters or hypotheses, something that is not allowed within the frequentistframework. Statements 5 and 6 mention the boundaries of the CI (i.e., 0.1 and 0.4), whereas, as was stated above, a CI can beused to evaluate only the procedure and not a specific interval. The correct statement, which was absent from the list, is the following: “If we were to repeat the experiment over and over, then 95 % of the time the confidence intervals contain the true mean.” (Hoekstra et al., 2014)

N = 27

Thank you for your attention

Science isn’t brokenIt’s just a hell of a lot harder than we give it credit for.

27https://www.pinterest.ch/researchugent/research-integrity/