The Psychological Science Accelerator Calls for Study Submissions

patrick-tomasso-406271-unsplash.jpg

The Psychological Science Accelerator (PSA), a network of over 500 labs collaborating to collect psychological data from large-scale international samples of people, is currently accepting study proposals from all areas of psychological science. Anyone can submit a proposal, regardless of PSA membership status, research area, methodology, or career stage. We especially encourage submissions in research areas including but not limited to clinical psychology, community psychology, developmental psychology, and neuroscience. The deadline for submission is September 15th, 2019.

The mission of the PSA is to accelerate the accumulation of reliable and generalizable evidence in psychological science, reducing the distance between the truth about human behavior and mental processes and our current understanding. For a full overview of the PSA, please see our paper introducing our policies and procedures (https://psyarxiv.com/785qu/). For information on previous successful submissions, please see each study’s status page, with descriptions and links to study materials and preprints (https://psysciacc.org/studies/).

Capture

Proposed studies can test novel hypotheses or focus on the replication of previous findings, can be basic or applied in focus, and can be exploratory or confirmatory in nature. Because accepted studies will likely involve considerable use of resources, the study selection process begins with the evaluation of a stage 1 Registered Report style submission. Once submitted, the proposals will be masked and checked for completeness. Then, our Study Selection Committee will conduct an initial evaluation of all proposed studies (see below for more information on the feasibility check portion of this initial evaluation), and vote on their acceptability for further consideration. Studies that pass this check will be fully evaluated by 6-11 peer reviewers, shared with all members of the PSA network to be rated, and either provisionally accepted or rejected by the Study Selection Committee.

Studies that are provisionally accepted enter a second stage of preparation and final evaluation before the PSA officially accepts the project, makes a public announcement, and recruits data collection labs. During this second stage of preparation and evaluation, the PSA and proposing authors collectively conduct a needs assessment. The goal is to identify all of the resources the provisionally-accepted project will need in terms of methodology, funding, translation, ethics, data management, project management, and logistics. This is an in-depth feasibility evaluation that will ensure that the project, if officially accepted, can be carried out to a high standard for rigor.

Projects that pass the in-depth feasibility evaluation proceed to the preparation phase. During this phase, the PSA assigns key PSA personnel to the project, including a project manager, methodologist, data manager, ethics manager, and translation manager in accordance with the needs assessment (these roles can be filled by proposing authors if they have the skills and capacity to do so). This group will then meet with the proposing authors to 1) establish a collaboration agreement, 2) identify who will be the lead communicator with data collection labs, and 3) establish a clear deadline for the proposing authors to convert their PSA proposal into a journal-quality draft of a Stage 1 Registered Report (RR). This journal-quality draft RR represents the final record of the motivation for the study and the methods to be used, including analysis plan. PSA directors will review the RR to make the final decision about whether to officially accept the project and make a public announcement to recruit data collection labs (with a link to the RR). Projects that are officially accepted move to the next stage of preparation, in which proposing authors will revise the RR based on feedback on the draft sent out to data collection labs and obtain ethics approval at their primary institution.

Even after official acceptance, PSA studies may go through revisions. For instance, reviewers at journals may suggest additional measures or procedural changes. Accepted proposals are not unchangeable. However, proposing authors and the PSA will make every effort to minimize the likelihood of post-acceptance changes. If the changes necessary after acceptance are too many in number or too significant, the PSA may ask proposing authors to withdraw their project and resubmit it during the next call for studies. Changes that are most likely to trigger a request for resubmission are those that increase the burden on data collecting sites.

After official acceptance, all review materials (submissions, peer reviews, network ratings, and Study Selection Committee decision letters) will be made publicly available for accepted submissions, but not rejected submissions.

Feasibility

All feasibility decisions are made with respect to our current, and ever-changing, resources. Although the PSA is comprised of hundreds of labs from around the world who have agreed, in principle, to volunteer some of their resources to PSA projects, we may not be able to accommodate all types of designs. A few important feasibility considerations that may result in a study not moving beyond the feasibility check include:

  • Does the study require specialized equipment (e.g., eye-tracking, EEG) or proprietary experimental software (e.g., E-Prime) to be used at the data collection sites?
  • Can experimental materials and analysis scripts be shared easily and made publicly available?
  • Does the study require “hard-to reach” samples (e.g., children, minority groups, clinical populations, etc.)?
  • Is the target sample size per site, number of data collection sites, duration of the individual data collection sessions, and number of data collection sessions required fully justified and balanced so as to not overburden the PSA network?
  • Is the likelihood and severity of risk to participants kept to a minimum, such that the risk is not greater than what participants would face normally and would not require special consideration or deliberation from an ethics board?

Characteristics of strong submissions

Beyond simply being feasible given current PSA resources, strong submissions will also:

  • Accurately and clearly describe literature relevant to the study’s goals and design, such that researchers unfamiliar with the subject can understand the basic concepts behind the theory/phenomenon and the purpose of the research.
  • Clearly articulate the purpose of the research, relevant research questions, and hypotheses (if confirmatory).
  • Clearly articulate the research design, with a focus on sound methodology appropriate to the research questions, including adequate power analysis to justify sample size.
  • Provide examples of relevant material, for example websites, experimental scripts (e.g., E-prime, Inquist, OpenSesame), precise experimental design, and/or stimuli.
  • Accurately and clearly describe an analysis strategy appropriate to the research questions and design. Pilot or simulated data and working analysis scripts are ideal for clarity.
  • Make a compelling case for the importance of large-scale collaborative data collection for the project.

Submission Format and Guidelines

The following components are required for all submissions:

  • Cover Page, including the title of the study, date of the latest draft, and keywords
  • Abstract of up to 150 words
  • Main body submission text of up to 5,000 words
  • A version of the submission with cover page included
  • A masked version of the submission without the cover page
  • References
  • Supplementary materials

The following guidelines are intended to assist you in the preparation of your study submission to the Psychological Science Accelerator. Submissions normally include a description of the key background literature and motivation for the study, hypotheses, study procedures, proposed statistical analysis plan, a statistical power analysis, and pilot data (wherever applicable).

Introduction

A review of the relevant literature that motivates the research question and a full description of the study aims and hypotheses.

Method

A full description of proposed sample characteristics, including criteria for data inclusion and exclusion (e.g., outlier extraction). Procedures for objectively defining exclusion criteria caused by technical errors or for any other reasons must be specified, including details of how and under what conditions data would be replaced.

A description of study procedures in sufficient detail to allow another researcher to repeat the methodology exactly, without requiring further information.

Proposed analysis pipeline, including all preprocessing steps, and a precise description of all planned analyses, including appropriate correction for multiple comparisons. Specify all covariates or regressors. Specify analysis decisions that are contingent on the outcome of prior analyses.

Results

Studies involving Neyman-Pearson inference must include a statistical power analysis. Estimated effect sizes should be justified with reference to the existing literature or theory. Because publication bias inflates published estimates of effect size, power analysis should be based on the lowest available or meaningful estimate of the effect size.

In the case of highly uncertain effect sizes, a variable sample size and interim data analysis is permissible but with inspection points stated in advance, appropriate Type I error correction for ‘peeking’ employed, and a final stopping rule for data collection outlined.

For studies involving analyses with Bayes factors, the predictions of the theory must be specified so that a Bayes factor can be calculated. Authors should indicate what distribution will be used to represent the predictions of the theory and how its parameters will be specified.

Full descriptions must be provided of any outcome-neutral criteria that must be met for successful testing of the stated hypotheses. Such quality checks might include the absence of floor or ceiling effects in data distributions, positive controls, or other quality checks that are orthogonal to the experimental hypotheses.

Supplemental Materials

Include full questionnaires, stimuli, and materials needed to conduct the study. Pilot data can be included to establish proof of concept, effect size estimations, or feasibility of proposed methods. Simulated data and analysis scripts are ideal for clarity of the exclusion criteria and analysis plan.

These guidelines were adapted from https://osf.io/pukzy.

 

Submissions can be made via this google form. Submissions will be accepted until 23:59 in the last time zone on earth on September 15th, 2019. If you have any questions, do not hesitate to email the PSA Director (Chris Chartier, cchartie@ashland.edu).

News from the Accelerator – May 2019

This month we announce an upcoming call for studies, seek contributors to several key committees, and give mini progress reports on all 6 of our current studies.

New Call for Studies Coming Soon

It is almost time for another round of study selection (our 3rd)! On June 15th we will release a detailed call for studies post with our updated submission guidelines. The deadline for study submissions will be September 15th. To begin planning now, you can review our prior call for studies, or reply to this email with any presubmission inquiries. We are excited to see what great ideas are submitted this round!

Committees Seeking Contributors

Several committees could use additional members to better support our current studies, provide training for the PSA network, and to prepare for our next round of study review and selection.

  • Ethics Committee: The Ethics Committee is seeking representation from the following broad geographic regions: Africa, Asia, Australia, and Central and South America. We are currently a group of 7; however, we are almost exclusively located in the US and Europe, and we want to meet the diverse needs of all PSA-represented regions and beyond. Currently, we have the following roles: 1) provide reviews of proposed PSA projects with an eye toward ethics, 2) provide reviews of accepted PSA project IRB and ethics review board materials prior to distribution to the larger PSA group, 3) provide ethics-oriented consultation to ongoing PSA projects, 4) work with other PSA committees to ensure ethical practices. Zoom-based meetings are approximately monthly. For more information and/or to submit a CV, please contact Erica Musser at emusser@fiu.edu.
  • Study Selection Committee: The Study Selection Committee serves a vital role in the workflow of the PSA by facilitating the review of proposed projects and ultimately deciding on their acceptance to be run by our network. We are currently seeking 2-4 new members to aid in our next round of selection in late summer and early fall of this year. Committee members must commit to serve for 1 year but may choose to resign at any point thereafter. Unlike many of the other PSA committees, the work of the Study Selection Committee is concentrated to a few months per year. The PSA solicits study proposal submissions on an ad hoc basis. After each call for submissions is closed, the Study Selection Committee evaluates the submissions for feasibility and general quality. We then vote on whether each proposal merits review. Suitable proposals are then reviewed by experts in and outside of the network. The Study Selection Committee compiles and synthesizes this feedback. The committee then makes the final selections on the basis of reviewers’ responses, evaluations from the PSA network, and the PSA’s mission, values and capacity. If you would like to join the committee, please submit a CV to committee co-chair and PSA Assistant Director Kathleen Schmidt (kathleenschmidt1@gmail.com or kathleen.schmidt@siu.edu).
  • Training Committee: The Training Committee is recruiting committee coordinators to help organize and provide trainings for the PSA and larger science community. If you are interested in joining our team, please read about our openings and apply! Link: https://forms.gle/yiS89d22bNA6P5hRA
  • Translation Coordinator for Study 004, The Accelerated CREP Project: We are looking for a translation coordinator for PSA 004 (Accelerated CREP) to coordinate and monitor the translations according to PSA’s translation policies. The Accelerated CREP is a pedagogical project conducted by student teams in collaboration with faculty supervisors. The preprint of the registered report (under review and hopefully very close to an in principle acceptance at AMPPS) is available at https://psyarxiv.com/zeux9   . The study is currently translated into nine different languages and most of these are ready for the next step in the translation process (back translations and controlling for equivalence). The translation coordinator is expected to coordinate the remainder of the translation process and monitor them. The translation coordinator will earn authorship on the resulting manuscript.

Current Studies

  • 001 FACE PERCEPTION: We have collected data from over 10,000 participants and are on track to finish data collection in September. 001 Preprint
  • 002 OBJECT ORIENTATION & 003 GENDERED PREJUDICE: The 003 pre-registration is being finalized by the lead team and data collection for the bundled projects will commence in June! 002 Preprint
  • 004 TRUE BELIEF: The Stage 1 Registered Report is under review (following 2 rounds of “revise and resubmit”) at Advances in Methods and Practices in Psychological Science. 004 Preprint
  • 005 STEREOTYPE THREAT: The Stage 1 Registered Report draft is nearly complete and will be submitted to Nature Human Behavior very soon.
  • 006 TROLLEY PROBLEM: The Stage 1 Registered Report is under review at Nature Human Behaviour. 006 Preprint

Postdoctoral Researcher Position

Tomorrow (June 1) is the last day to apply for the PSA post-doc position. We (a team led by Hans IJzerman) are looking for a talented and motivated postdoctoral researcher. The position is for two years and will be hosted at LIP/PC2S at Université Grenoble Alpes. The position’s primary focus is to support the Psychological Science Accelerator (PSA: https://psysciacc.org) in grant writing. The first priority for the postdoctoral researcher is to help write a Synergy Grant (https://erc.europa.eu/funding/synergy-grants), where the proposed co-PIs are John Ioannidis (Stanford University), Denny Borsboom and Eric Jan Wagenmakers (University of Amsterdam), Lisa DeBruine and Ben Jones (University of Glasgow), and Hans IJzerman (Université Grenoble Alpes). The postdoctoral researcher will also be involved in PSA-related research and will be involved in some supervision of students who do PSA-related research. The candidate we search should be an excellent writer and should have ample experience with open science (and be familiar with R and/or Python). To read more about context in which the postdoctoral researcher will be embedded, please go to www.corelab.io.

News from the Accelerator – April 2019

This month we have a few updates on our 6 studies, our search for funding, and other interesting projects that PSA members may be interested in.

001 FACE PERCEPTION

We have collected data from over 9,000 participants! There are two regions that we could still use extra collection in: Africa and Central America & Mexico. If you, or a colleague of yours, would be interested in contributing to data collection in either region, please let us know by replying to this email. Materials are ready, and this would be a relatively quick way to contribute to a PSA study and earn authorship on the Nature Human Behaviour paper! Additionally, we have funds for a few (3-5) small ($200-$400) data collection grants if such an award would make joining possible for you.

001 Preprint

002 OBJECT ORIENTATION & 003 GENDERED PREJUDICE

Labs continue to secure IRB approval and contributors are translating materials into their local languages. The last major step prior to opening data collection for these studies is a pre-registration of the 003 analysis plan, which should be complete in the next few weeks!

002 Preprint

004 TRUE BELIEF

We received a second, and relatively minor, revise and resubmit from Advances in Methods and Practices in Psychological Science. Braeden Hall is leading the revision efforts, and we think we are quite close to securing an in principle acceptance!

004 Preprint

005 STEREOTYPE THREAT

Labs are securing IRB approval, the lead team is finalizing the adaptive design that will be used to distribute participants across conditions, and all collaborators are currently providing feedback on the Stage 1 Registered Report manuscript.

006 TROLLEY PROBLEM

After multiple rounds of contributor feedback, Bence and Balazs submitted the Stage 1 Registered Report manuscript to Nature Human Behaviour. Fingers crossed for a quick and positive peer review experience, and congratulations to all co-authors on making it to this big milestone for the project! Labs are securing IRB approval at a fast rate, and we are just about to begin our translation process in earnest, led by Chuan-Peng Hu, who will be coordinating translations for the study.

006 Preprint

GRANT APPLICATIONS

  • We should hear back on our NSF submission in the next few months. If we receive this grant, we will be able to hire 2 full time research coordinators.
  • The line-up is set for our Synergy grant submission. Hans IJzerman is leading a team including EJ Wagenmakers, Lisa DeBruine, Ben Jones, Denny Borsboom, and John Ioannidis in drafting this submission.
  • Bence Bagó, Balazs Aczel, and Rink Hoekstra are working on a grant application for the NWO replication stream to further expand the 006 study to more non-student samples.
  • Neil Lewis Jr. is leading a team to apply for 005 project specific funding through the William T. Grant Foundation.

INTERESTING OPPORTUNITIES AND PROJECTS SEEKING COLLABORATORS

  • The EEG Many Labs study leaders now welcome study nominations for the project. You can nominate a study using this form.
  • In-Mind Foundation is reorganizing and is seeking Assistant Directors. Here is the interest form.
  • Jess Flake (PSA Data and Methods Assistant Director) is hiring a post-doc in quant psych and modeling. Ad here.

News from the Accelerator – March 2019

March has been a particularly active month for the PSA, and we are building serious momentum collecting data, preparing multiple studies to begin data collection, and establishing new PSA policies around data management and ethics approvals.

The network also continues to grow (556 researchers, 446 laboratories, 65 countries)! Nick Coles is regularly updating this map to reflect our ever-expanding roster of member researchers. If you don’t see yourself or your lab on the map, let us know!

Congratulations and thank yous are in order for Chuan-Peng Hu and Ilker Dalgar who will both be taking up roles as Assistant Directors of the PSA for the Translation and Cultural Diversity Committee. We will be working with Chuan-Peng and Ilker to, among other things, identify translation coordinators for all PSA projects and to further refine and improve our translation procedures. Thank you to Oscar Oviedo Trespalacios who served this role for many months and was integral in creating our initial PSA translation policies!

Study Updates

  • 001 FACE PERCEPTION. We’ve collected data from over 7,000 participants! This study is right on track to wrap up by the planned end of data collection in September.
  • 002 OBJECT ORIENTATION & 003 GENDERED PREJUDICE. Labs are securing IRB approval and contributors are translating materials into their local languages.
  • 004 TRUE BELIEF. Still under review. Fingers crossed!
  • 005 STEREOTYPE THREAT. IRB materials have been drafted and put through a final review with the ethics committee. They will be shared with labs very soon.
  • 006 TROLLEY PROBLEM. Over 80 labs have submitted IRB documents and 40 have approval. We plan to submit the project as a stage 1 registered report to Nature Human Behaviour on April 15th.

 Funding

  • Hans IJzerman is organizing a team to submit a European Research Council Synergy Grant application, and is recruiting researchers who are interested in leading one of the proposed research centers in Europe. You can read more about this opportunity and indicate your interest in getting involved here.
  • We have also started a crowd-funding campaign to pool monthly contributions into small (at first) stipends to support PSA labor that is currently being completed on a volunteer basis. Examples include translation, data management, project management, ethics review (and on and on and on). You can see the ongoing campaign info, and contribute if you are willing and able, at the PSA Patreon page. Thanks to those who have already generously donated, we can begin the process of selecting recipients and paying these stipends next month!

Policy Documents

  • A group led by Patrick Forscher has drafted a very nice and extensive set of data management guidelines for the PSA. This policy document has been voted on and approved by the Data and Methods Committee, but they remain open to your feedback on all things data management and data release. One important element of the guidelines is to identify a data manager for every PSA project. If you are interested in data management, you can check out this tweet thread to learn a bit more and find out how to get involved in a data management role for the PSA.
  • The Ethics Committee is drafting a brief policy document to clarify our plans for documenting situations where labs do not require local ethical approval to conduct a PSA study. You can see the draft policy statement, and the conversation surrounding it in this google doc. Please feel free to provide your feedback in the comments.

Interesting Links and Opportunities

There are several cool posts up on StudySwap seeking collaborators or offering up equipment:

PSA Talks

  • Lisa DeBruine gave a talk about the PSA at the Open Science IMC conference.
  • Braeden Hall gave a talk on both the CREP and the PSA as part of an ECR-led Open Science in Action symposium.
  • Ben Jones gave a talk on the PSA at Abertay University

Thanks for spreading the word!

PSA Facebook

PSA Twitter

News from the Accelerator – February 2019

This month (our 18th month!) we have made progress on PSA policy documents, our search for funding (don’t get too excited…no big news yet), and all 6 of our in-progress studies. Also of note, we’re now up to 428 unique labs! Thank you so much for everyone who has gotten involved in the PSA!

PSA Policies

  • The membership policy was ratified in our most recent vote and is now official PSA policy. This will lead to slight changes in our onboarding process for the PSA generally and specific PSA studies. In the near future we will distribute a one-time form for those already in our membership roster to affirm the code of conduct and the mission of the PSA (be on the lookout for that!).
  • We now call for final comments and votes on our Personnel, Appointments, and Elections Policy Document. PSA members will receive the draft policy via email and have until March 15th, 2019 to vote.

Updates from the Funding Committee

The funding committee has been working extremely hard to identify and pursue promising funding opportunities for the PSA. We all owe a big thanks to Neil Lewis Jr., Thuy-vy Nguyen, and Hans IJzerman for their efforts in organizing our funding search and attempts to date. Here are some notable attempts:

  • We have requested an NSF grant (under review)
  • We requested a Marie Curie grant (but it was rejected 😦 )
  • In the Netherlands, we will collaborate with Marcel van Assen to request an Open Competition Grant
  • Diego Forero is requesting an internal grant at his University to create a PSA hub
  • EU PSA researchers are working on requesting a Synergy Grant. Two centers for this grant have been identified, but we will want to identify two more. Those interested in participating can contact Hans IJzerman.

PSA researchers can identify smaller grant opportunities and contact the funding committee (Neil Lewis and Thuy-Vy Vu) to investigate possibilities. We are always looking for ways to support the PSA and to support PSA members through grants, large and small.

Finally, central to our future funding requests is to work on a long-term vision document, detailing what we are able to accomplish with our current support and capacity, but also what we would plan to do with additional support to make an even bigger impact on psychological science. Chris Chartier is taking the lead on this, and will share drafts in the coming months.

Study Updates

Now that we have 6 studies concurrently in preparation or data collection, some of us are having trouble keeping them straight when discussing their status! So, we’ve started using 3-digit codes and 2-word titles for our in-progress studies. Here are some mini-updates on all 6 PSA studies:

  • 001 FACE PERCEPTION. The study is now up and running, in at least demo mode, for 22 of our data collection languages. You can test out all versions via these demo links. We have now collected data from 3,497 total raters across 9 world regions and are well on our way to meeting our September target for completing data collection!
  • 002 OBJECT ORIENTATION & 003 GENDERED PREJUDICE. These two studies are “bundled” for a shared data collection session. Now that 002 has an in principle acceptance in hand from Psychonomic Bulletin and Review, we have decided to pursue results-blind review (rather than a registered report) for 003. We have initiated IRB and translation processes for these studies in the hopes of initiating data collection quite soon.
  • 004 TRUE BELIEF. We are still in the “revise and resubmit” process for this study at Advances in Methods and Practices in Psychological Science. We hope to have news of an in principle acceptance to share soon!
  • 005 STEREOTYPE THREAT. The leadership team for this study has made considerable recent progress on making final design decisions and we will be sharing IRB materials with data collection labs in the coming weeks.
  • 006 TROLLEY PROBLEM. This study now includes a team of 130 labs across 40 countries! We are hoping to submit as a registered report to Nature Human Behaviour. Our next step is to initiate IRB submission in all data collection labs.

Spreading the Word

Members have presented, or otherwise shared, information about the PSA with wide audiences this month.

  • Hannah Moshontz presented at the SIPS pre-conference session of SPSP in Portland, Oregon. You can see her slides here.
  • Gianni Ribeiro presented to an open science meetup group at IGDORE in Bali.DytUYbBUUAEUOyB
  • Lisa DeBruine gave an excellent interview on the Everything Hertz podcast.
  • Ben Jones published an article on Psychology Today discussing first impressions and how the PSA is working to make the literature less WEIRD.
  • Chris Chartier, Nick Coles, and Hans IJzerman published an article on Science Trends calling for more researchers to join us in building up the PSA.

Interesting Links and Opportunities

Finally, here are some links to other projects or opportunities that PSA members may find interesting (feel free to pass such links along to Chris for future newsletters):

PSA Facebook

PSA Twitter