Effective Incident Management For Trust & Safety Teams

The goal of Trust & Safety teams is to protect users and platforms through thoughtful policy development, creation and maintenance of safe and secure platforms, and effective protocols and mechanisms for content moderation. The most effective measures for doing so are to have proactive detection and preventative measures for ensuring threat actors and their tactics don’t migrate from off-platform to on-platform. However, even the best platforms will deal with incidents from time to time. 

For this reason, platforms need to have workflows that specifically address incident management and the protocols to be followed during and after their occurrence. While each platform will have a different interpretation and approach to what constitutes an incident and how it should be handled appropriately, there are common themes for Trust & Safety teams to be aware of.

Incidents

Incidents are pieces of malicious content that have failed to be moderated before or immediately after being posted or indexed, that violate the platform’s policies, and that have gained some degree of user attention which poses a reputational risk to the platform. In order to deal with these types of incidents, platforms need to have pre-existing mechanisms in place to remove, report, and mitigate the risk of such violations. 

Incident Management Protocols 

Companies must have protocols in place to handle these situations, and the backbone of any incident management protocol is effective triaging. With strong triaging and reporting protocols, Trust & Safety teams can handle them swiftly. Such protocols should take into account a variety of factors: 

• Has this incident spread to the press?
• Has the incident spread to social media? 
• How many users have been exposed to this incident? 
• Have users flagged or filed reports regarding the specific content, and if so, how many? 
• Does the incident concern sensitive events, like elections, a health crisis, or social unrest?
• Can this content, if kept online, cause harm to users? 
• Does this content pose a reputational or regulatory risk to the platform? 

Identifying the way in which incidents are categorized by severity before they occur can be a key factor in the prioritization of incidents, and can help Trust & Safety teams do their jobs effectively when under pressure.

Incident Analysis

When an incident occurs, if the content is proven violative of a platform’s policy, it will be removed and the incident will be closed. This is what’s called a ‘point fix’.  The removed content may be a comment, an account, or a page that has been de-indexed.

If the content is proven not to violate a policy, it will stay online. However, the incident may still require deeper assessment. Why was this content flagged if it doesn’t violate a policy? Is there a gap in the existing policy that allows violative content to slip through the cracks? Are there external factors that may affect the allowance of this type of content to remain online, even if it violates policy? Such an analysis may inform policy change, which is why incidents must always be further assessed.

Depending on a variety of factors, online platforms can have anywhere from a handful to thousands of incidents per month. During sensitive times, like election season or in the midst of social unrest or a public health crisis, platforms may see a sizable spike in incidents necessitating point fixes. A strategically-minded Trust & Safety team will look at these incidents and try to gauge commonality among them and the potential presence of an underlying problem that allowed such incidents to occur. 

Answering these queries may identify a root cause that allowed these incidents to happen, and may point to a policy that isn’t complete, a product that isn’t built right, or a safety mechanism that is not optimized.

This is where the real power of incident management lies: the insights from incidents that inform policy and product improvements. Trust & Safety teams should focus on proactive prevention of threats, but as we all know, violations occur despite the best efforts. Thoughtful incident management acknowledges that it could not prevent the incident today, but by identifying root causes and systemic problems, it may indeed prevent the incident tomorrow.

Next Steps 

Given the potential ramifications of incidents for platforms both small and large, effective, proactive management and clear protocols must be considered high priority. By employing systems that deny threat actors the ability to post malicious or illegal content, whether on social networks, e-commerce sites, dating apps, or other types of UGC platforms, companies can build and maintain public trust, ensure compliance with the law, and create a broadly safe and secure digital environment for users. ActiveFence offers detection and reporting services that deliver intelligence on emerging threats before they take hold. Having access to this kind of intel early on is exactly the kind of tool that can enable platforms to deny the spread of malicious content. 

The principle of Safety by Design applies here especially; companies have the opportunity to design platforms that put proactivity at the forefront. Assuming responsibility as a service provider, encouraging user empowerment and autonomy, and giving importance to transparency and accountability will be effective strategies for creating and maintaining a safe platform. 

Proper proactive policies are an invaluable tool for platforms to be able to prevent incidents and therefore mitigate harm. Building these policies and carrying them out effectively requires intelligence collection, detection, and action; ActiveFence is proud to provide these services to platforms.

Learn about our offerings and see how our tailored solutions can be a tool in your platform’s crisis management protocol.