APA’s Executive Lead Psychologist for Science Adrienne Stith is calling on the National Institutes of Health (NIH) to withdraw or substantially revise a proposal (PDF, 322KB)opens in new window that would replace final overall impact scores and percentiles with three broad categories: “Most Competitive,” “Competitive,” and “Not Discussed.” On behalf of APA, in comments responding to NIH’s Request for Information on Proposed Changes to Reporting Outcomes from NIH Peer Reviewopens in new window, Stith warned that the change would reduce transparency without resolving variability or potential bias in peer review.
According to the NIH proposal, the agency would continue using numerical scores internally, while applicants, institutions, program staff, NIH Institute and Center leaders, and Advisory Councils would receive only a categorical outcome. NIH claims that the change is intended to reduce overly mechanical reliance on paylines and small score differences that may imply more precision than peer review can support.
While APA agrees that small differences in scores should not be treated as precise or determinative, reducing scores to three broad categories would eliminate meaningful distinctions when considered alongside written critiques and programmatic judgment help applicants and decision-makers understand review outcomes. Scores, percentiles, critiques, and panel summaries serve different but complementary purposes; they help applicants identify needed revisions and allow institutions and the public to assess consistency across review panels, disparities across fields or applicant groups, changes over time, and instances in which final funding decisions depart from peer review assessments. Narrative critiques alone are not calibrated to a common scale and therefore cannot fully replace this information.
Reduced information may also have unequal effects. Applicants with less institutional experience, fewer mentoring resources, or limited access to informal guidance may be less able to interpret a broad category or determine how to strengthen a future application. Broad categories may also obscure meaningful differences across fields. Behavioral and psychological science, including prevention, implementation, population, and interdisciplinary research, often uses methods, endpoints, and pathways to impact that differ from those common in laboratory-based biomedical research. Any new system should be evaluated to ensure that it does not systematically disadvantage these approaches. NIH can reduce overreliance on small score differences without withholding information applicants, decision-makers, and the public need to evaluate the fairness and integrity of the funding process.
Specifically, APA recommended NIH:
- Retain overall impact scores and percentiles during any transition and report new categories alongside them.
- Assess narrower score ranges or decile bands as alternatives that reduce false precision without eliminating useful distinctions.
- Standardize narrative feedback so applicants can distinguish major concerns from minor or readily addressable issues.
- Require an auditable decision record that explains how peer review, strategic priorities, portfolio balance, and budget considerations informed award decisions.
- Pilot and independently evaluate any new system before full implementation, including its effects on equity, applicant behavior, administrative workload, and behavioral and psychological science.
For more information, email Angela L. Sharpe.