Manage and orchestrate the entire Trust & Safety operation in one place - no coding required.
Take fast action on abuse. Our AI models contextually detect 14+ abuse areas - with unparalleled accuracy.
Watch our on-demand demo and see how ActiveOS and ActiveScore power Trust & Safety at scale.
The threat landscape is dynamic. Harness an intelligence-based approach to tackle the evolving risks to users on the web.
Don't wait for users to see abuse. Proactively detect it.
Prevent high-risk actors from striking again.
For a deep understanding of abuse
To catch the risks as they emerge
Disrupt the economy of abuse.
Mimic the bad actors - to stop them.
Online abuse has countless forms. Understand the types of risks Trust & Safety teams must keep users safe from on-platform.
Protect your most vulnerable users with a comprehensive set of child safety tools and services.
Stop online toxic & malicious activity in real time to keep your video streams and users safe from harm.
The world expects responsible use of AI. Implement adequate safeguards to your foundation model or AI application.
Implement the right AI-guardrails for your unique business needs, mitigate safety, privacy and security risks and stay in control of your data.
Our out-of-the-box solutions support platform transparency and compliance.
Keep up with T&S laws, from the Online Safety Bill to the Online Safety Act.
Over 70 elections will take place in 2024: don't let your platform be abused to harm election integrity.
Protect your brand integrity before the damage is done.
From privacy risks, to credential theft and malware, the cyber threats to users are continuously evolving.
Your guide on what to build and buy
It goes without saying that all content moderation decisions should be fair and equal, but sometimes these decisions are much trickier, especially when the users and posts in question have massive followings and influence. While one post might be a blip on the radar, moderating another might make global news. In order to ensure that all users, no matter their relative power, receive fair treatment, Trust & Safety teams need to take into account various considerations to keep their platforms trustworthy.
Powerful, influential users are generally good for business. Using their influence to share mostly positive (or at least not violative) content, these individuals create communities and accrue user engagement, ultimately driving high revenues for both themselves and the platforms they use. However, it is when these accounts begin participating in violative activity that Trust & Safety teams are faced with more significant dilemmas than usual.
Recently, Ye (formerly known as Kanye West) had several posts removed and his Instagram and Twitter accounts temporarily restricted for violating the content policies on both platforms. While not the first time Ye has lashed out on these platforms – not even the first time this year, in fact – this is a slightly softer reaction to content violations than what former president Donald Trump experienced, having been banned from Twitter following the January 6 insurrection.
Typically, the enforcement of content policy takes a number of different forms, ranging from adding warning labels to denote sensitive content to posts to removing them, suspending users, or banning them outright. Some platforms even add a layer of context or informative label to seemingly controversial posts that might not otherwise violate a policy. In the cases of Ye and Trump, platforms took the typical approach of post removal, account suspension, and, in Trump’s case, an indefinite ban.
It’s the goal and responsibility of Trust & Safety teams to keep platforms safe and secure, and sometimes that means needing to hold users accountable for their actions. While this seems like a pretty mundane issue – bad behavior has consequences – it becomes more complex when those committing the violations have massive followings. It’s the responsibility and goal of Trust & Safety teams to keep user safe, but that mission gets challenged sometimes, like in the cases of Ye and Trump.
It’s no secret that power helps you get away with things offline, and online, it appears that sometimes holds true, too. Back in 2019, one outlet published a story on the conundrum of moderating high-profile users like those mentioned above and found that Meta moderated its users differently depending on their level of influence. Rules were relaxed when it came to powerful users, and certain personalities were effectively immune to them. The company was privately holding influential users to different standards. In practice, this isn’t so uncommon: Twitter, in their explanation about Trump’s ban from the platform, stated that it was the context of his tweets and the influence he had that resulted in his removal, not necessarily the use of explicitly violative language. This indicates that had it been a user with a small following sending out the same tweets, that person might still be on the platform today: it wasn’t just the content that was problematic in Trump’s case, it was his level of influence.
Powerful accounts represent high user engagement and business opportunities for platforms and posters alike. The problem is threefold: platforms want to keep users engaged as much as possible, and moderating accounts or posts with large engagement puts this at risk by creating the possibility for on-platform backlash, fomenting bad press, and even resulting in a loss of revenue. It’s no secret that high user engagement means financial gain for platforms, so one of their chief interests will be keeping people online. But moderating can anger users and cause them to troll or spam Trust & Safety teams, to leave a platform en masse, or to make negative noise off-platform. That being said, choosing not to moderate high-profile users can have the same effect: showing that different users are treated unequally won’t do anything to boost a platform’s credibility. Damned if you do, damned if you don’t, as the saying goes.
The question here isn’t about what policies a platform should have, but if they should be applied evenly to all users, regardless of status. The short answer is yes, but it’s not an answer without its complications.
Trust & Safety teams are tasked with the difficult balance of ensuring free and open engagement in online communities with guaranteeing user safety. So from the outset, they need to have adequate transparency.
The rationale behind content moderation decisions should be clear to users so that when issues arise, there’s well-established clarity about how they’re handled. That means laying out the process for each level of moderation, whether it’s a first-strike system that’ll ban someone outright upon a violation, or if the go-to method is to add warning labels to potentially violative posts while leaving them up. When it comes to especially tough decisions in high-stakes situations, teams should have internal processes to ensure correct choices are being made, bringing different opinions to the table to guarantee biases are free from the final say. Being upfront about moderation mistakes if and when they happen is also a good practice for Trust & Safety teams to implement. Those moments present learning opportunities not only for the platforms but for other companies in the industry as well.
Trust & Safety doesn’t exist in a vacuum, and powerful users are on every site and app out there: policies can and should evolve given the changing world we live in. Across the board, Trust & Safety teams need to be in agreement that policy is policy, regardless of a user’s power on a platform or off it. Situations will always arise that challenge this notion, as we’ve seen recently. But encouraging moderators to treat all users the same is the key to maintaining trustworthiness and transparency. Just as laws in the ‘real world’ are meant to be applied equally, so too, should the policies on platforms.Â
Looking for better ways to moderate influential users? Explore our comprehensive content moderation solutions.
The decision to build or buy content moderation tools is a crucial one for many budding Trust and Safety teams. Learn about our five-point approach to this complex decision.
Recognized by Frost & Sullivan, ActiveFence addresses AI-generated content while enhancing Trust and Safety in the face of generative AI.
Building your own Trust and Safety tool? This blog breaks down for Trust and Safety teams the difference between building, buying, or using a hybrid approach.