Recognizing Invisible Labor in the PSA

One of the biggest threats to the sustainability of large-scale collaborations is the rewards system in place for scholarly labor.  Scholarly labor is rewarded with credit for scholarly products. Typically, people scan authorship lists to observe whom to credit, with the lion’s share going to the first- and last-listed authors.

This disproportionate assignment of credit serves as a disincentive for collaborating on papers with author lists that are too long: if the bulk of the credit goes to the first and last authors, middle authorship is only worthwhile when earned through a minimal investment of effort.  This disincentive weighs especially heavily on multi-site projects, which require dozens, if not hundreds, of people to pull off successfully.

The problem is still harder in a standing organization like the PSA.  Maintaining an organization requires a lot of administrative labor – labor that is often invisible if there aren’t mechanisms in place to surface it.  In academia, we see this invisible labor problem on a small scale in the typical treatment of academic lab managers: these people are necessary to produce the science that graces academic journals, but are seldom, if ever, properly credited.

Given that the success of the PSA depends on ongoing administrative labor, we would like to find ways to disrupt the perverse incentive structure that inhibits standing collaborations in academia.  The PSA already endorses a contributorship model of awarding credit for scholarly products and uses the CRediT taxonomy to show project contributions.  We will also be unveiling the results of some ongoing initiatives to implement CRediT when we share the results of our first study.  

One additional initiative that we’ve started is to award stipends for the administrative roles that people fill for PSA studies.  For now, we will only be giving six of these stipends, and the stipends themselves are relatively small – $400 – but, if we are able to secure regular sources of funding for the PSA, paying our staff is one of our highest priorities.  You can help us increase the size and number of these stipends by donating to our Patreon (and you can read more about our ongoing Patreon campaign here).

Finally, we want to make some of the invisible contributions to the PSA more visible by highlighting some of the contributors.  Below, we profile five of these people, each at a different place in their scientific career, all of whom have made outstanding contributions to the PSA and its projects.  Only by recognizing the value of contributions like theirs will we subvert the system that heaps credit on the few at the expense of the many.

Profiles of five major contributors to the PSA

Nicholas Coles

Nicholas Coles is a fifth year Social Psychology PhD student at the University of Tennessee.  He’s a member of both the Project Monitoring and Community Building committees at the PSA.

unnamed

Nick has been a rockstar Project Monitor for PSA001 (face perception).  As the first project monitor for any PSA project, he has helped define what project monitors do, and has made a myriad of tracking sheets, and forms, and other materials that have served as templates for similar tools in other PSA projects. As the PSA001 project monitor, Nick has also served as the primary point of contact for an author list of nearly 200 people. Nick’s efforts have helped produce a data collection effort spanning 11,000 participants, 48 countries, and 28 languages.  As if that weren’t enough, Nick also developed a web application that displays an interactive map of the PSA network.

If you want to find out more about Nick and his work, check out his website.

Anna Szabelska

Anna finished her PhD in Cognition at Queen’s University Belfast and is looking for her next adventure. She was recently accepted to the NASA Datanauts project, a program that applies data science methods to NASA datasets. This program, along with her work for the PSA, is the coolest thing that’s happened in her professional life.

pasted image 0

Anna has been an exceptionally active member of the PSA from the very start. She was a founding member of the Data and Methods Committee and helped draft its bylaws. She has also co-led a project to create a standard psychology dataset format (Psych-DS), which would enable projects ranging from automated meta-analysis to standardized data analysis tools. She has also played a critical role on PSA002 (object orientation), and in that capacity, she has helped define what methodologists do in PSA studies. One of the more interesting outcomes from that work is a novel meta-science initiative – prediction markets to determine how accurate experts are in predicting whether the object orientation effect replicates in a given language. Finally, Anna has shared her enthusiasm for the PSA with anyone who is willing to listen, giving official talks on the PSA for RLadies Dublin, Women Who Code, and Google Women Techmakers, and co-organizing a PSA workshop and unconference.

You can find more about Anna (including her CV) at her LinkedIn profile.

Jeremy Miller

Jeremy Miller is a Professor of Psychology at Willamette University in Salem, OR. Jeremy is on sabbatical and has generously volunteered a chunk of his sabbatical time to working with the PSA. 

pasted image 0 (1)

Jeremy is the Project Monitor for two PSA projects, PSA002 (object orientation) and PSA003 (gendered prejudice), which have been bundled together for the purpose of efficient data collection. Jeremy coordinates communications between the many, many parties involved in pulling off a project involving 49 labs, 16 languages, and two separate projects, with an eye toward ensuring all parties adhere to the PSA’s ever-evolving policies. His effectiveness in this role is informed by his experience as the head of a data collection lab for PSA001 (face perception). Jeremy also serves on the Project Monitor committee, working with Project Monitors across PSA projects to ensure that the PSA uses its resources effectively and efficiently. 

You can read more about Jeremy and his lab at his lab website.

Marton Kovacs

Marton is a second-year master’s student at Eotvos Lorand University, Hungary and is a huge fan of both the PSA and the broader movement to improve psychological research. He is planning to apply to PhD programs to do meta-scientific research, focusing specifically on ways to increase research efficiency by minimizing human error.

pasted image 0 (2)

Even though Marton is relatively early in his scientific career, he has played a critical role in swiftly moving the PSA006 (moral dilemmas) from conception to a submitted Registered Report. As Data Manager, he drafted the data management plan that will ensure that the data from more than 130 participating labs are credible, transparently shared, and efficiently collected. He is also helping develop tools that will help solve some of the unique problems that large-scale collaborations face. This includes the problem of properly crediting contributions mentioned in this post: Marton is developing a Shiny app that helps authors create human- and machine-readable contributorship information so that the “invisible contributions” to multi-author projects are made visible.

You can find out more about Marton on his website

Sophia Christin Weissgerber

Sophia C. Weissgerber recently started a postdoc position in cognitive psychology at the University of Kassel in Germany. She is survey manager for PSA004 (true belief), a partnership with the Collaborative Replication and Education Project (CREP), which uses multi-site replications to provide training and professional growth experiences for students and instructors. Since Sophia is a huge fan of the CREP-project, she likes to expose her 3rd semester students to real-world hands-on research experience, for example replication of Griskevicius et al. (2010).

pasted image 0 (3)

Sophia performed a feat of programming wizardry for PSA004, implementing an experimental design in the online platform SocSciSurvey for a project involving 54 sites (and counting), as well as 26 countries and 12 languages. Each site requires a unique survey and survey link, so Sophia also does extensive coordination with the participating sites to personalize each survey for each site. This labor is critical for the project’s success. Sophia also enjoys participating in other PSA-projects, e.g. PSA001 (face perception). She is really excited about the PSA community and work (quote: “super-awesome”) and together with Hans IJzerman, Rick Klein, and Anna van ‘t Veer, she is currently working on a non-technical primer on how to conduct code review in psychological science.

You can read more about Sophia and her work on her website.

Announcing PSA Member Lab Grants for 2019-2020

Opportunity is distributed unequally in human affairs. Science is a human affair, so this dictum applies to science as it applies to any other human endeavor.

At the PSA we recognize this as an unfortunate current characteristic of science, but we also think that science will be both fairer and of higher quality if opportunity were distributed more evenly. Working toward this aspiration is central to our core principles. However, we have some ways to go to meet it: of the over 500 labs that have joined our network, only 14% were in South and Central America, 5% in Eastern Europe, 6% in the Middle East, 4% in Asia, and 1% in Africa. 

Last year, we started an initiative to move us closer to our aspiration: we issued three small grants of $400 to under-resourced labs PSA labs. These grants helped these labs collect data and earn authorship on one of our studies.

This year, we are continuing this program. While the exact number of grants we will be able to issue is uncertain, we hope to issue six grants of $400 each to PSA labs that wish to collect data for one of our studies. These grants will be funded using money collected from our Patreon if we meet our support goal (see this previous post for more details).  

We will select recipients based on the following two criteria:

  1. Demonstrated need. We define “need” to mean that the data collection could not happen without the provision of the grant. As an additional provision, and as an attempt to improve lab representation from non-Western countries, at least three of the grants will go to labs from those regions.
  2. Benefit to the PSA study in question. We will prioritize labs that provide a unique benefit to the PSA study in which they are participating. For example, in a study of how a psychological process varies across cultures, we will prioritize labs that allow the recruitment of participants from a culture that is not already represented in the study.

Our current crop of accepted studies deal with topics ranging from language comprehension, to stereotype threat, to moral decision-making. Five of these six studies (002 through 006) are actively recruiting labs. We also have an active call for studies that ends September 15, 2019 that will give us a new wave of studies covering yet more topics. This provides prospective labs with the opportunity to join studies, and earn authorship on manuscripts, that match their interests.

If you are interested in applying for one of these grants, fill out this form. We hope this initiative will play a small but measurable role in redistributing scientific opportunities in a fairer way.

Capture

The PSA’s Patreon Sustainability Drive

The PSA owes its success to the enthusiasm of its members.  That enthusiasm has led to an outpouring of volunteer work hours. In 2019, we estimate that these hours total 19,390, equivalent to approximately 10 full-time people per year. 

However, enthusiastic volunteerism has its limits. First, some items, such as software licenses and compensation for participants, cannot be bought solely with volunteer labor. Second, volunteer time necessarily comes at the expense of time spent on other activities — activities that more directly benefit one’s career. Thus, for the PSA to maximize its positive impact, it needs to develop a broad set of reliable funding streams, both to enable special projects that require direct funding and to guard against the eventuality that enthusiasm for the PSA diminishes.

That’s where you come in. We have recently created a Patreon page to allow interested people to make regular contributions to help the PSA fulfill its goals. The exact items that we fund with the Patreon will change on a regular basis; however, we have made a pledge that all the money we receive through this Patreon page will be reinvested into the PSA in a way that directly supports our mission and is in line with our core principles.

In the coming year, we hope to use the Patreon to fund two items, both of which will be discussed further in a blog post of their own later this week:

  1. Six grants of $400 to PSA member labs who wish to collect data for one of our studies but lack the resources to do so
  2. Six stipends of $400 to help compensate people who take on study-specific administrative roles

We will feel comfortable funding these if our Patreon reaches the threshold of $400 per month by Tuesday September 3, 5pm UTC. We will provide regular updates as to our progress.

If you are able to donate a monthly contribution and you think our goal is worthy, consider becoming a patron. Even $1 a month gets us closer to your goal. Help us reach our goal and thereby incrementally advance the improvement of psychological science!

Highlights of the PSA’s First Two Years

On August 26, 2017, Chris Chartier wrote a blog post posing a simple question: what might happen if psychology adopted the collaborative “big science” model seen in physics and built a “CERN for psychological science”?

2018 gave us a glimpse of the answer: a rapidly-expanding network of over 300 psychology labs from over fifty countries. The first year was a flurry of activity as, in quick succession, the PSA accepted four studies and had its first academic paper accepted. The network’s rapid expansion in its first year speaks to the hunger for ambitious solutions to two of the problems that psychology currently faces: lower-than-desired levels of replicability and generalizability of psychological findings.

The PSA’s second year has marked the beginnings of a transition from idea to action.  The network has continued to expand, now standing at over 500 labs from over 70 countries.  We have selected 2 new studies, collected data from over 11,000 participants for our first study, initiated data collection in 3 other studies, received in principle acceptance for 2 registered report manuscripts, and submitted another 2 for stage 1 peer review. 

This productivity in the selection, preparation, and conduct of our studies was matched by a substantial organizational infrastructure development. PSA members named Assistant Directors to lead the committees that do the bulk of its administrative and coordination work, initiated a project to formalize its study pipeline, drafted a coherent set of policies dictating how the PSA manages and releases its data, made the PSA’s first full-time hire (Patrick Forscher at Université Grenoble Alpes), announced the first round of PSA elections, created a Patreon, submitted several grant proposals, and drafted a vision statement outlining the PSA’s long-term priorities. If the PSA’s first year was focused on growth and excitement, its second year was focused on turning plans into action and establishing sustainability, and we hope our third year is one of securing large-scale funding to more effectively pursue our mission with six key aims: 

  1. Collect samples that represent all people 
  2. Train and support our global network of researchers
  3. Expand the PSA’s capacity to include more complex behavioral studies
  4. Release large-scale data-sets that are maximally informative for secondary re-use
  5. Improve psychological science through meta-scientific and instructional contributions
  6. Expand capacity to conduct more, and bigger, studies

Below we have created a chronological timeline of important events in the PSA. All of these accomplishments represent the work of literally hundreds of researchers. We can’t thank them enough for joining us on this journey to accelerate psychological science. Here’s to many more productive and rewarding years ahead!

Timeline of important PSA events (with links)

Year one

Year two

The PSA is Two Years Old. Let’s Celebrate!

This coming Monday the Psychological Science Accelerator turns two. To celebrate, we’re planning a week of exciting announcements, blogposts, and hackathons. Today we just wanted to post a full schedule for the week, with brief descriptions of each event in case you wanted to follow along or join in. Watch our twitter account for frequent updates and links throughout the week.

Blog schedule

Monday

Highlights from our first 2 years

Tuesday

Updates on our Patreon grassroots funding campaign

Wednesday

Announcing a new batch of PSA member lab grants for 2019/2020 and how you can apply

Thursday

Celebrating and making more visible the sometimes invisible work of PSA members filling study-specific support roles (e.g., project monitoring, data management, translation coordination)

Friday

Releasing and soliciting feedback on the first draft of our vision statement for the PSA (precursor to a 5-year strategic plan)

Saturday

Announcing Study 001’s data release plan, including incentives for pre-registered secondary analyses

Hackathon schedule

Monday

Vision Statement Draft Feedback and Editing (Chris Chartier, 10:30 UTC, 6:30am EST)

Google drive migration (Patrick Forscher, 17:00 UTC, 1:00PM EST) 

Tuesday

Meta-science policies (Nick Fox, 18:00 UTC, 2:00pm EST)

Wednesday

Member site and roster info form (Erin Buchanan, 13:30 UTC, 9:30am EST)

Thursday

Election logistics (Charlie Ebersole, 16:30 UTC, 2:30pm EST)

Friday

Patreon and Donor Campaign (Chris Chartier, 13:00 UTC, 9:00am EST)

Synergy hack-a-thon (Patrick Forscher, 15:00 UTC, 11:00am EST)

We hope to see you during the week!