Second Special Call for Studies – Studying Generalizability with Global Samples

Deadlines

  • Initial Proposals (Optional) due by February 15, 2023.
  • Full Proposals due by May 15, 2023.

Overview

The Psychological Science Accelerator (PSA), supported by the John Templeton Foundation (JTF), welcomes study proposals to test the generalizability of phenomena related to JTF strategic priorities with large, global samples. 

The PSA is a distributed network of researchers from the behavioral sciences across the globe. Our more than 2400 collaborators represent 73 countries from all six populated continents. The mission of the PSA is to accelerate the accumulation of reliable and generalizable evidence in psychological science. This call for studies will select projects that fit within the 5 topic areas, broadly construed, that are prioritized by the JTF. Both members and non-members of the PSA can submit proposals.

JTF Priority Topics

To be eligible for this call, projects must investigate research questions related to one or more of the strategic priorities of the JTF in university and community samples across the globe. Below is a list of those priorities. Each is followed by some example research questions and topics. These are meant to just be examples; they are not an exhaustive list of the research questions that could be encompassed by each priority.

  • The dynamics of religious change. Why do religions flourish or lose adherents? What kinds of features of religious organizations drive or repel members? Why and how do people switch religious identities?
  • Intellectual humility. Which factors enhance or inhibit intellectual humility? How can we better study the concept of intellectual humility?
  • Religious cognition. What is the nature of religious belief and how can it be better measured? What causes religious or spiritual experiences and what are the effects of those experiences? How do individuals develop and revise their religious beliefs?
  • The science of character virtue. In particular, what helps individuals to develop curiosity and love? What are the consequences of those virtues? How do religions and perceptions of supernatural agents affect virtue and morality development?
  • Health, religion, and spirituality. Are there associations between religious/spiritual beliefs, experiences, practices, and identities, and physical and mental health? If so, are there underlying causal mechanisms?

Study Incubation

Unlike previous PSA calls, this call includes the option of submitting a short initial proposal that will be considered for study incubation. See section below for requirements.

The intention of incubation is to support proposing authors in developing a full proposal. This extra step is designed to remove barriers to submission and increase the competitiveness of proposals from researchers who are under-represented in psychological science (broadly defined and self-identified). 

Authors of all initial proposals will be given written feedback and the opportunity to meet with someone from the grant team to discuss their projects. A subset of these proposals will also be selected for further incubation. Proposing authors of these submissions will be provided additional guidance in the form of meetings, full proposal draft feedback, and ad hoc support.

Short proposals may be submitted at any point up until February 15, 2023, and will be evaluated on a rolling basis for feedback and support needs.

Proposal Selection

All full proposals, regardless of whether they were a part of study incubation, will have the same requirements and will be given equal consideration during the project selection process.

PSA projects are selected using a rigorous review process. Researchers submit detailed proposals for consideration. These proposals resemble Stage 1 Registered Reports (e.g., Chambers, 2013), and contain a theoretical introduction and hypothesis, a description of the planned sample and methods, and an analysis plan to test those hypotheses (see below section for requirements).

The proposals then undergo several rounds of review, overseen by the PSA’s Study Selection Committee (SSC). First, each proposal is screened for feasibility given the PSA’s current capacity and resources (Initial feasibility and quality review). For instance, a proposal that requires dozens of collection sites with fMRI machines is likely to be rejected based on infeasibility. Simultaneously, the SSC screens out submissions deemed low quality. Proposals that pass this screening are then sent out for peer review (Reviewer identification & review submission). Each proposal is reviewed by 5-10 reviewers, consisting of both PSA members and external experts. These reviewers are identified based on their ethical, methodological, and/or theoretical expertise related to a given proposal. All submissions are also rated quantitatively by members of the PSA network (Network rating). The SSC then synthesizes the reviewer and network feedback to select projects for the PSA to pursue (Study selection decision). See Evaluation Criteria section for more details and anticipated timeline.

Funding and Personnel Support for Studies Resulting from this Call

Projects selected through this special call will be advertised to members of the  PSA network in collaboration with the proposing authors. Each study will be supported by $40,000 of direct funding for data collection. Specifically, these funds will be distributed to network data collection teams, primarily for the purposes of compensating participants from their local communities.

The proposing author teams will also be collectively supported by 3 full-time scientific staff members (a Research Coordinator [Erin Seivers], a Postdoctoral Researcher [Dr. Priya Silverstein], and a Research Scientist [Dr. Kathleen Schmidt]), an additional ⅓ time senior staff member (Dr. Christopher R. Chartier), a ½ time staff member equivalent team of undergraduate research assistants, several members of PSA committees (at their discretion), and members of the network at large (at their discretion).

Submission Requirements

Initial Proposals (Optional – these will be considered for incubation)

Initial proposals should be 500-1000 words in length and include rationale for the project and an overview of the research design. As part of the submission form, proposing authors will provide details about their research including type, investigation category, relevant JTF priority/ies, and subfield. They will also be asked to describe how the additional support provided by the study incubation process would help them overcome barriers to submitting a full proposal or otherwise benefit their submission.

Full Proposals (Required)

The following manuscript components are required for all full proposals:

  • Cover Page, including the title of the study, date of the latest draft, and keywords
  • Abstract of up to 150 words
  • Main body submission text of up to 5,000 words
  • A version of the submission with a cover page included
  • A masked version of the submission without the cover page
  • References
  • Supplementary materials

The following guidelines are intended to assist you in the preparation of your proposal submission to the PSA. Submissions typically include a description of the key background literature and motivation for the study, hypotheses, study procedures, proposed statistical analysis plan, a statistical power analysis, and pilot data (wherever applicable).

Introduction

A review of the relevant literature that motivates the research question and a full description of the study aims and hypotheses.

Method

A full description of proposed sample characteristics, including criteria for data inclusion and exclusion (e.g., outlier extraction). Procedures for objectively defining exclusion criteria caused by technical errors or for any other reasons must be specified, including details of how and under what conditions data would be replaced.

A description of study procedures in sufficient detail to allow another researcher to repeat the methodology exactly, without requiring further information.

Analysis Plan

Proposed analysis pipeline, including all preprocessing steps, and a precise description of all planned analyses, including appropriate correction for multiple comparisons. Specify all covariates or regressors. Specify analysis decisions that are contingent on the outcome of prior analyses.

Studies involving frequentist inference must include a statistical power analysis. Estimated effect sizes should be justified with reference to the existing literature or theory. Because publication bias inflates published estimates of effect size, power analysis should be based on the lowest available or meaningful estimate of the effect size.

In the case of highly uncertain effect sizes, variable sample size and interim data analysis is permissible but with inspection points stated in advance, appropriate Type I error correction for ‘peeking’ employed, and a final stopping rule for data collection outlined.

For studies involving analyses with Bayes factors, the predictions of the theory must be specified so that a Bayes factor can be calculated. Authors should indicate what distribution will be used to represent the predictions of the theory and how its parameters will be specified.

Full descriptions must be provided of any outcome-neutral criteria that must be met for successful testing of the stated hypotheses. Such quality checks might include the absence of floor or ceiling effects in data distributions, positive controls, or other quality checks that are orthogonal to the experimental hypotheses.

Supplemental Materials

Include full questionnaires, stimuli, and materials needed to conduct the study. Pilot data can be included to establish proof of concept, effect size estimations, or feasibility of proposed methods. Simulated data and analysis scripts are recommended to provide clarity about the exclusion criteria and analysis plan.

Please also describe within the supplemental materials the details of what open research practices will be followed and how. This section should include information about plans for data sharing, analysis code sharing, pre-print use, and Registered Report manuscript submission and/or pre-registration. Other practices which are common in PSA studies are open materials, use of open source tools, and open research workflows. 

These guidelines were adapted from https://osf.io/pukzy.

Evaluation Criteria

We anticipate that three studies will be selected and implemented through this call. 

Appropriateness for this special call for studies is one of the key criteria for evaluation. Your full proposal should clearly state how your project aligns with the goals of the call and discuss why your research topic would specifically benefit from global data collection (e.g., theoretical reasons to predict global variation). 

Upon submission of your full proposal, you will be asked to provide details about your project to categorize the submission and help the study selection committee assess its fit and feasibility. These items are listed below (see also the submission form link).

  • What type of research are you proposing?
  • What category of investigation is your project?
  • To which JTF strategic priority does your research relate? Select all that apply.
  • What area of psychology best describes the research?
  • Target number of data collection sites
  • Target number of participants per data collection sites
  • Does your project require specialized equipment (e.g., eye-tracking, EEG) or proprietary experimental software (e.g., E-Prime) to be used at the data collection sites?
    • If yes: Please explain and justify the use of specialized equipment or proprietary software in your research. Can you provide the collection sites with these resources? Do you only need a minority of sites to participate who already have these resources?
  • Can experimental materials and analysis scripts for your project be easily shared and made publicly available?
    • If no: Please explain and justify why your materials or analysis scripts cannot be publicly shared.
  • Does your project require “hard-to reach” samples (e.g., children, traditionally marginalized group members, clinical populations, etc.)?
    • If yes: Please explain and justify your use of “hard to reach” samples in your research. Do you only need a minority of sites to participate who already have access to these populations? If not, how will they recruit participants? Does research with this population involve additional ethical considerations?
  • Does the study expose participants to more risk than they would encounter in everyday life?
    • If yes: Please explain and justify the potential risk to individuals who participate in your proposed study.

In evaluating submissions for this special call for studies, we will prioritize projects whose designs are most well suited for promoting generalizability. In particular, we will prioritize studies that seek global samples (as opposed to studies that wish to sample from one or a few countries), studies with very high statistical power, and studies that are more strongly grounded in previous research (such as replication and generalization studies). All of these criteria are intended to maximize our chances of producing generalizable insights on phenomena-of-interest.

In addition, we will evaluate proposals for this special call together so that we can ensure variation across the projects. In particular, we will strive to select a package of studies that, collectively, represent a range of JTF priorities, study designs (e.g., experimental vs. correlational), and researcher backgrounds (e.g., research specialty, researcher location, etc.). This way, we will avoid overrepresenting a given topic, type of research, or type of researcher in this initiative. 

Timeline

  • Short Proposal (optional – to be considered for incubation) latest deadline: February 15, 2023
  • Short Proposal feedback and incubation decision: March 1, 2023 (latest date).
  • Full Proposal deadline: May 15, 2023.
  • Initial feasibility and quality review: May 16 – May 22, 2023.
  • Reviewer selection: May 22 – May 29, 2023.
  • Network rating solicitation: May 29, 2023.
  • Review submission deadline: June 12, 2023.
  • Network rating deadline: June 12, 2023.
  • Study selection decision: June 26, 2023.

Possible submission outcomes include “desk rejection” upon initial review, rejection upon full review, provisional acceptance, or an invitation to revise and resubmit.

Following a successful period of needs assessment, preparation, personnel identification, lab recruitment, and pre-registration, provisionally selected studies will commence data collection in late 2023 and end data collection in late 2024.

Please use this form to submit your optional initial proposal.

Please use this form to submit your required full proposal.

For pre-submission inquiries, please email Kathleen Schmidt at kathleenschmidt1@gmail.com or Chris Chartier at cchartie@ashland.edu . They are happy to meet with proposing authors and/or answer questions about what is likely to make for a strong submission.

Using crowdfunding to finish the largest stereotype threat project with Black American participants ever

(Donate to the project here)

In 2018, the Psychological Science Accelerator selected an unusual study to conduct. Instead of using the PSA network to achieve an international sample, this study focused on a hard-to-reach study population: Black American college students. Compared to previous studies, the project was also unusually applied-facing, as it focused on a theory that was developed to understand, and hopefully address, performance gaps in settings like college where strong performance could affect a person’s life trajectory. 

The proposed project aimed to address potential weaknesses in the evidence for the theory while simultaneously demonstrating how we can use big team science to achieve the sample sizes needed for informative quantitative tests of theories about hard-to-reach populations. The budding PSA network saw merit in this project, and at least one journal has agreed, as the project was accepted in principle as a Registered Report at Nature Human Behaviour.

Four years later and data collection is going strong. As of October 31, 2022, we have recruited 1025 Black college students into our project. This is a stunning accomplishment, and already puts our project among the largest ever studies of stereotype threat among Black American college students. This accomplishment has only been possible due to financial contributions from the project leadership team, which we have redistributed to data collection labs to defray the costs of participant recruitment.

However, 1025 is still short of our goal of a minimum of 2000 participants. We now believe we need crowdfunding to reach our data collection goal.

You can donate to this project here. Any donations we receive that exceed our financial need will be donated to the PSA at project’s end, as we have relied heavily on PSA resources to make this project happen.

Help us finish an amazing – and, for psychology, historic – project, further demonstrating the potential of big team science for solving some of behavioral science’s most pressing problems.

– The PSA Stereotype Thread administrative team

Help us select our next project(s)!

Project proposal submissions to the PSA’s special call for studies that have passed the initial phase of study selection are now available for network evaluations. PSA members can read and provide feedback on the proposals in a survey on the Canvas membership site. Specifically, you will be asked to rate the submission as a potential PSA project and indicate your interest in contribution to the project if selected. Providing feedback is completely optional, but much appreciated, and will not take much of your time. The deadline to submit evaluations is September 30th. 

Special Call for Studies – Studying Generalizability with Global Samples DEADLINE EXTENDED

Note: Proposals submitted by the original deadline (July 20) are already being reviewed, but any submissions received by the extended deadline (August 31) will be given full consideration.

Updated Timeline

  • Submission deadline (extended): August 31, 2022.
  • Feasibility and quality review: September 1 – 7, 2022.
  • Reviewer selection: September 8-14, 2022.
  • Network rating solicitation: September 15, 2022.
  • Review submission deadline: September 29, 2022.
  • Network rating deadline: September 29, 2022
  • Study selection decision deadline: October 12, 2022.

Overview

The Psychological Science Accelerator (PSA), supported by the John Templeton Foundation (JTF), welcomes study proposals to test the generalizability of phenomena related to JTF strategic priorities with large, global samples.

The PSA is a distributed network of researchers from the behavioral sciences across the globe. Our more than 1300 collaborators represent 84 countries from all six populated continents. The mission of the PSA is to accelerate the accumulation of reliable and generalizable evidence in psychological science. This call for studies will select projects that fit within the 5 topic areas, broadly construed, that are prioritized by the JTF.

JTF Priority Topics

To be eligible for this call, projects must investigate questions related to one or more of the strategic priorities of JTF. Below is a list of those priorities. Each is followed by some example research questions and topics. These are meant to just be examples; they are not an exhaustive list of the research questions that could be encompassed by each priority.

  • The dynamics of religious change. Why do religions flourish or lose adherents? What kinds of features of religious organizations drive or repel members? Why and how do people switch religious identities?
  • Intellectual humility. Which factors enhance or inhibit intellectual humility? How can we better study the concept of intellectual humility?
  • Religious cognition. What is the nature of religious belief and how can it be better measured? What causes religious or spiritual experiences and what are the effects of those experiences? How do individuals develop and revise their religious beliefs?
  • The science of character virtue. In particular, what helps individuals to develop curiosity and love? What are the consequences of those virtues? How do religions and perceptions of supernatural agents affect virtue and morality development?
  • Health, religion, and spirituality. Are there associations between religious/spiritual beliefs, experiences, practices, and identities, and physical and mental health? If so, are there underlying causal mechanisms?

Proposal Selection

PSA projects are selected using a rigorous review process. Researchers submit detailed proposals to the PSA for consideration. These proposals resemble 5,000 words Stage 1 Registered Reports (e.g., Chambers, 2013), and contain a theoretical introduction, a description of the planned sample and methods, as well as hypotheses and an analysis plan to test those hypotheses. 

The proposals then undergo several rounds of review, overseen by the PSA’s Study Selection Committee (SSC). First, each proposal is screened for feasibility given the PSA’s current capacity and resources. For instance, a proposal that requires dozens of collection sites with fMRI machines is likely to be rejected based on infeasibility. Simultaneously, the SSC screens out submissions deemed low quality. Proposals that pass this screening are then sent out for peer review. Each proposal is reviewed by 5-10 reviewers, consisting of both PSA members and external experts (see Supporting Materials for peer review rating criteria). These reviewers are selected based on their methodological and/or theoretical expertise related to a given proposal. All submissions are also rated quantitatively by the entire PSA network. The (SSC) then aggregates two sources of info, the comments of the reviewers and ratings from the network, before selecting projects for the PSA to pursue.

Funding and Personnel Support for Studies Resulting from this Call

Projects selected through this special call will be advertised to members of the  PSA network in collaboration with the proposing authors. Each study will be supported by $40,000 of direct funding for data collection. The proposing author teams will also be collectively supported by 3 full-time scientific staff members (a Research Coordinator, a Postdoctoral Researcher, and a Research Scientist), an additional ⅓ time senior staff member (Dr. Christopher R. Chartier), a ½ time staff member equivalent team of undergraduate research assistants, several members of PSA committees (at their discretion), and members of the network at large (at their discretion).

Submission Requirements

The following components are required for all submissions:

  • Cover Page, including the title of the study, date of the latest draft, and keywords
  • Abstract of up to 150 words
  • Main body submission text of up to 5,000 words
  • A version of the submission with a cover page included
  • A masked version of the submission without the cover page
  • References
  • Supplementary materials

The following guidelines are intended to assist you in the preparation of your study submission to the PSA. Submissions normally include a description of the key background literature and motivation for the study, hypotheses, study procedures, proposed statistical analysis plan, a statistical power analysis, and pilot data (wherever applicable).

Introduction

A review of the relevant literature that motivates the research question and a full description of the study aims and hypotheses.

Method

A full description of proposed sample characteristics, including criteria for data inclusion and exclusion (e.g., outlier extraction). Procedures for objectively defining exclusion criteria caused by technical errors or for any other reasons must be specified, including details of how and under what conditions data would be replaced.

A description of study procedures in sufficient detail to allow another researcher to repeat the methodology exactly, without requiring further information.

Analysis Plan

Proposed analysis pipeline, including all preprocessing steps, and a precise description of all planned analyses, including appropriate correction for multiple comparisons. Specify all covariates or regressors. Specify analysis decisions that are contingent on the outcome of prior analyses.

Studies involving Neyman-Pearson inference must include a statistical power analysis. Estimated effect sizes should be justified with reference to the existing literature or theory. Because publication bias inflates published estimates of effect size, power analysis should be based on the lowest available or meaningful estimate of the effect size.

In the case of highly uncertain effect sizes, variable sample size and interim data analysis is permissible but with inspection points stated in advance, appropriate Type I error correction for ‘peeking’ employed, and a final stopping rule for data collection outlined.

For studies involving analyses with Bayes factors, the predictions of the theory must be specified so that a Bayes factor can be calculated. Authors should indicate what distribution will be used to represent the predictions of the theory and how its parameters will be specified.

Full descriptions must be provided of any outcome-neutral criteria that must be met for successful testing of the stated hypotheses. Such quality checks might include the absence of floor or ceiling effects in data distributions, positive controls, or other quality checks that are orthogonal to the experimental hypotheses.

Supplemental Materials

Include full questionnaires, stimuli, and materials needed to conduct the study. Pilot data can be included to establish proof of concept, effect size estimations, or feasibility of proposed methods. Simulated data and analysis scripts are recommended to provide clarity about the exclusion criteria and analysis plan.

These guidelines were adapted from https://osf.io/pukzy.

Evaluation Criteria

Two studies will be selected and implemented through this year’s call. The project will also include a second round of submissions, with two additional studies to be selected, in 2023.

In your submission, you should clearly state how your project aligns with the goals of this special call and discuss why your research topic would specifically benefit from global data collection (e.g., theoretical reasons to predict global variation). 

In evaluating submissions for this special call for studies, we will prioritize projects whose designs are most well suited for promoting generalizability. In particular, we will prioritize studies that seek global samples (as opposed to studies that wish to sample from one or a few countries), studies with very high statistical power, and studies that are more strongly grounded in previous research (such as replication and generalization studies). All of these criteria are intended to maximize our chances of producing generalizable insights on phenomena-of-interest.

In addition, we will evaluate proposals for this special call together so that we can ensure variation across the projects. In particular, we will strive to select a package of studies that, collectively, represent a range of JTF priorities, study designs (e.g., experimental vs. correlational), and researcher backgrounds (e.g., research specialty, researcher location, etc.). This way, we will avoid overrepresenting a given topic, type of research, or type of researcher in this initiative. 

Possible submission outcomes include “desk rejection” upon initial review, rejection upon full review, provisional acceptance, or an invitation to revise and resubmit for next year’s call.

Following a successful period of needs assessment, preparation, personnel identification, lab recruitment, and pre-registration, provisionally selected studies will commence data collection in early 2023 and end data collection in late 2023.

Please use this form to submit your proposal.

For pre-submission inquiries, please email Chris Chartier at cchartie@ashland.edu. He is happy to meet with proposing authors and/or answer questions about what is likely to make for a strong submission.

Nominations for PSA leadership positions

The Psychological Science Accelerator (PSA) is accepting nominations for two Associate and six Assistant Directors. Associate Directors serve on the Board of Directors and are elected by all PSA members for 4-year terms. Assistant Directors co-lead (with another Assistant Director) specific committees for 3-year terms and are elected by vote by the Board of Directors. 

The positions are open to all members of the PSA–and current non-members are also eligible if they sign-up for a [free] PSA membership. Members can either nominate themselves or others for each role. There are no term limits. For more information about PSA elections policies, see this document.

Nominations are due July 29 2022. To nominate yourself or others, please (1) login to the PSA membership portal, (2) navigate to the PSA Membership course page, (3) click on the Quizzes tab, and (4) complete a short nomination form.

All current members of the PSA should have received an email invitation to the Canvas membership site within the past year. If you know you have a PSA ID, you can reset your password by going to https://canvas.psysciacc.org/. If you aren’t sure what email to use, contact psa.membersite@gmail.com. If you do not have an account, you can sign up for an account using this form: https://member.psysciacc.org/

See below for more information about the open positions.

Associate Directors

Associate Directors serve on the Board of Directors, which serves as the steering body of the PSA. Typically, they meet 1-2 times a month with an assigned committee, help the committee perform essential tasks, and serve as a liaison between the committee and the Board of Directors. They also attend a Board of Directors meeting every three weeks.The estimated time commitment is 4-5 hours per month. For more information, please contact the current Director (ncoles@stanford.edu).

Assistant Director of Project Monitoring

The Project Monitoring Committee consists of two assistant directors and a larger team who actively serve as project monitors of PSA studies and provide general feedback about PSA project procedures. The committee meets twice a month, wherein committee members share project updates, troubleshoot problems, and discuss project management strategies. Assistant Directors are responsible for coordinating these twice-a-month meetings and for regularly communicating updates to the PSA Board of Directors. For more information, contact the current Assistant Director (david.vaidis@u-paris.fr).

Assistant Director of the Study Selection Committee 

The Study Selection Committee consists of two assistant directors and a small team that is tasked with reviewing and selecting projects for the PSA. Study proposals that pass an initial quality and feasibility check by the committee are then reviewed by members of the advisory committees and internal and external content experts. For more information, contact the current Assistant Director (kathleenschmidt1@gmail.com).

Assistant Director of the Funding Committee 

The Funding Committee works with the PSA Board of Directors on grant applications, money management, and the developing of the PSA funding strategy. The Funding Committee also offers occasional support to PSA study investigators in funding applications. For more information, contact the current Assistant Director (nlewisjr@cornell.edu).

Assistant Director of the Ethics Committee

The PSA Ethics Committee helps ensure high quality assessment and identification of ethical risks or challenges, as well as protection and safeguarding of ethical practices of PSA research projects. In the broadest sense, the committee is responsible to: 1) provide reviews of proposed PSA projects with an eye toward ethics, 2) provide ethics-oriented consultation to selected PSA projects (e.g., co-develop the material to ensure hassle-free approval in the majority of PSA labs), 3) curates ethics procedures towards successful implementation of ongoing PSA projects (especially related to the IRB and ethics review board materials prior to distribution to the larger PSA group), 4) work with other PSA committees to ensure ethical practices. For more information, contact dana.basnightbrown@gmail.com.

Assistant Director of the Translation and Cultural Diversity Committee

The Translation and Cultural Diversity committee coordinates translations for each PSA project, expands the PSA community by recruiting translators, and helps ensure that PSA members from every country are treated fairly and equally.

Assistant Director of the Community Building Committee

The Community Building and Network Expansion Committee (CBNEC) is tasked with engaging, assessing and expanding the PSA Network (“improve the reach of and access to the PSA, both internally and externally” Moshontz et al., 2018 pg. 511). The Assistant Directors provide the long-term vision for the committee and coordinates its daily/monthly operations.