Posting Content and Comments
Offering users the opportunity to create content aligned with safety policies and reform that content before it becomes visible or submitted for review gives content creators direct feedback on the content they create in real time.
How does this mitigate hate?
A significant percentage of the toxic content that is created on social platforms is created by users who often don’t post toxic content, but instead are just not in a good mood and don’t react well to content that triggers an emotional reaction in them.
Providing a bit of friction to the process of posting and commenting allows users to pause before they inappropriately post content that they ordinarily wouldn’t say to others in person or offline.
When to use it?
This pattern is best suited in social, content creation tools as an opportunity for users to reexamine the platform’s safety policies, and re-form their content before posting. While it also offers the opportunity for users to consider how their post / reaction may be considered harassment rather than contribute to the conversation.
Platforms should use nudges to discourage users’ attempts to engage in abusive behavior. One way to do this is to use automation to proactively identify content as potentially abusive and nudge users with a warning that their content may violate platform policies and encourage them to revise it before they post.
How does it work?
These tools work to flag and categorize potentially harmful comments before a human can review them, helping to manage the workload and reduce the visibility of toxic content.
Every time a user submits content/comments, it goes through the moderation algorithm.
If the system identifies the content as risky/offensive or is misaligned with the publisher’s Community Guidelines instead of rejecting it automatically or sending for additional human review, a message is presented to the user urging them to take another look at what they wrote.
The user can either choose to edit their comment and repost or they can post it anyway, accepting the outcome.
Users get one chance per comment to rethink their message before it is submitted, in order to deter them from gaming the system. (1)
Advantages
The feature uses the nudge concept, a known theory in behavioral sciences that proposes positive reinforcement and indirect suggestions as ways to influence the behavior and decision making of groups or individuals. (2)
Disadvantages
Some of the users that receive feedback will likely try to game the system by manipulating the spelling of the word, adding punctuation, etc.
If implemented incorrectly it may hinder freedom of expression, ideas, etc.
Ido Goldberg, OpenWeb’s SVP of product, says this kind of adaptive behavior was one of the main concerns in designing their real-time feedback feature. “There’s this window for abuse that’s open to try to trick the system,” he says. “Obviously we did see some of that, but not as much as we thought.”(3)
Considerations
Create a limit to the amount of chances users have to fix their comment. This will keep users from gaming the system through manipulating the word. In addition, have a delay of time before the user can try to post again as a penalty and to keep them from gaming the system.
If a user is repeatedly ignoring nudges and trying to trick the system, it warrants stronger tools such as auto suspension. OpenWeb allows publishers to suspend a user from participating in the conversation if they attempt to post a rejected comment multiple times.(4)
Examples
Perspective API nudge example
OpenWeb – real time feedback example
Hate Free Plug-in in action on twitter
References
Cupples, Sarah. “Frictionless Design, Frictionless Racism.” Medium, February 6, 2021. https://uxdesign.cc/frictionless-racism-1097022d07f8.
Goldberg, Ido, Guy Simon, and Kusuma Thimmaiah. “Nudge Theory Examples in Online Discussions.” OpenWeb, September 20, 2020. https://www.openweb.com/blog/openweb-improves-community-health-with-real-time-feedback-powered-by-jigsaws-perspective-api.
Jigsaw. “Helping Authors Understand Toxicity, One Comment at a Time.” Medium, February 10, 2021. https://medium.com/jigsaw/helping-authors-understand-toxicity-one-comment-at-a-time-f8b43824cf41.
Lougmani, Chaymae. “Hate Free Plugin: Reducing Cyber Harassment in Social Media.” Medium, February 2, 2021. https://blog.prototypr.io/we-won-international-alert-peacehack-and-heres-why-d41b4534dbe3.
Pardes, Arielle. “To Clean up Comments, Let AI Tell Users Their Words Are Trash.” Wired, September 22, 2020. https://www.wired.com/story/comments-section-clean-up-let-ai-tell-users-words-trash/.
“Perspective API.” www.perspectiveapi.com. Accessed September 17, 2021. https://www.perspectiveapi.com/#/home.
(1)(2)(3)(4) Vilk, Viktorya, Elodie Vialle, and Matt Bailey. “No Excuse for Abuse: What Social Media Companies Can Do Now to Combat Online Harassment and Empower Users.” Edited by Summer Lopez and Suzanne Nossel. PEN AMERICA. PEN America, March 31, 2021. https://pen.org/report/no-excuse-for-abuse/.
Collaboration
Written in collaboration with PenAmerica.