Highlighting High Quality Comments

Goal: Improve the quality of users’ comments Evidence rating (PDN): Likely Civic signal: Understand — Promote thoughtful conversation When to use: Reactive

What it is

In a comments section of an online publication, editors select a high quality comment to be highlighted and acknowledged, for example as an "Editors' Pick".

Intended impact

By acknowledging a commenter for their high quality contribution - and in doing so, giving them a social reward - that commenter is incentivized to continue creating similar high quality comments. Editors may also signal to other users and commenters (whose comments were not highlighted) the types of comments that are valued and, subsequently, incentivize others to aim for similar levels of comment quality.

Evidence that it works

Wang & Diakopolous (2022) compared the subsequent comments of readers who received a NYTimes "Editor's Pick" with those of readers who wrote comments of similar quality (to the "pick"). From a natural experiment (so lacking a randomized control), the results are persuasive that recipients of "picks" continue to create higher quality content than they would otherwise. There was less evidence, however, that commenters who merely viewed a "pick" went on to create higher quality comments.

Why it matters

Special Considerations

• This approach might yield different results in communities where quality of responses is less valued; in other words, the effectiveness of this intervention might be mediated by the culture and values of a community. The existence of shared values might be a prerequisite for this intervention to work.

• This intervention may have limited impact as there may be a ceiling to the number of commenters who can receive a pick; that number is constrained by the number of users who create adequately high comments as well as the threshold beyond which being "picked" is no longer a prized acknowledgement. In the Wang & Dialopoulos paper, for example, 10% of all commenters received a pick.

• The highlighting effect is rather short-lived (i.e., lasting for around three subsequent comments) This is not only notable from a purely practical perspective but also because it might tell us something about how this intervention works. Discussions with researcher Yixue Wang suggest that this intervention is very externally rewarding focused. That is, the highlighting is less about internally stirring up people’s perception and feeling of individual social responsibility in a comment section. More so the observed short-term effect speaks to the interpretation that picked commentators want to be picked again and, therefore, produce high-quality content. This is important to note because external reward does not just fade quickly (in the study users lost interest in high-quality comments after having not received highlighting for their next 2 or 3 comments) but can also turn sour when commentators grow frustrated because they do not get highlighted again even though they invest more time in a comment (to make it high-quality).

• Being transparent regarding the procedure by which a comment has been picked may be considered; in other contexts procedural transparency has been shown to promote prosocial behavior.

Citations

• *Highlighting High-quality Content as a Moderation Strategy: The Role of New York Times Picks in Comment Quality and Engagement* — Yixue Wang, Nicholas Diakopoulos (ACM Transactions on Social Computing, January 29, 2022). https://doi.org/10.1145/3484245

---

From the Prosocial Design Network's Digital Intervention Library (https://www.prosocialdesign.org/library/highlighting-high-quality-comments), licensed CC BY-SA 4.0. Reformatted for the RT Commons; text otherwise as published. Library citation: Prosocial Design Network (2024), Digital Intervention Library, https://doi.org/10.17605/OSF.IO/Q4RMB. Anything built from this pattern should credit the Prosocial Design Network and the researchers cited above.

Details

• goal: Improve the quality of users’ comments
• citations: [object Object]
• image alt: A Life of Brian comment that's been edited, and assigned a ribbon to show it's the top pick from a moderator.
• image url: https://cdn.prod.website-files.com/647f696dc653a97c26f42a0b/6764872028d3e8192284c733_647f696dc653a97c26f42ba2_top_pick.webp
• pdn rating: Likely
• when to use: Reactive
• civic signal: Understand
• pdn library doi: 10.17605/OSF.IO/Q4RMB
• civic signal goal: Promote thoughtful conversation