will rutter

Peer Feedback: Studying peer review in the life sciences

In January 2018, ASAPbio engaged me to perform generative research on life scientists' experience of peer review along with evaluative user research for ASAPbio’s proposed journal-independent peer review service titled Peer Feedback. In brief, the proposed system involved scientists submitting their papers to the Peer Feedback organization for peer review before submission to any journals. The reviews would then be posted publicly with the paper on a pre-print server, and could be forwarded with the paper to any journal with the intent that technical review would not have to be repeated. This Peer Feedback system, as conceived, was intended to mitigate the conflation of impact and technical rigor perceived by ASAPbio in the life science publication system. I conducted a mixed-methods user study including a survey and 45-minute in-depth interviews using a Wizard of Oz prototype to examine scientists’ reactions to various situations they may encounter with the Peer Feedback System. I received 291 responses to the survey and interviewed 17 scientists at institutions throughout the Americas and Europe. Unfortunately, the survey and interviews showed the system was not likely to be well received or utilized for various reasons. My final report, delivered in early February 2018, was well received despite the negative news about the Peer Feedback proposal. This portfolio piece includes samples of the study materials and report as well as photos of in-progress analysis.

Select each image below to enlarge and read a brief description of it.