Your guide to selecting the best Trust and Safety tools Download the Buyer's Guide to Trust and Safety 2.0
Manage and orchestrate the entire Trust & Safety operation in one place - no coding required.
Take fast action on abuse. Our AI models contextually detect 14+ abuse areas - with unparalleled accuracy.
Watch our on-demand demo and see how ActiveOS and ActiveScore power Trust & Safety at scale.
The threat landscape is dynamic. Harness an intelligence-based approach to tackle the evolving risks to users on the web.
Don't wait for users to see abuse. Proactively detect it.
Prevent high-risk actors from striking again.
For a deep understanding of abuse
To catch the risks as they emerge
Disrupt the economy of abuse.
Mimic the bad actors - to stop them.
Online abuse has countless forms. Understand the types of risks Trust & Safety teams must keep users safe from on-platform.
Protect your most vulnerable users with a comprehensive set of child safety tools and services.
Stop online toxic & malicious activity in real time to keep your video streams and users safe from harm.
The world expects responsible use of AI. Implement adequate safeguards to your foundation model or AI application.
Implement the right AI-guardrails for your unique business needs, mitigate safety, privacy and security risks and stay in control of your data.
Our out-of-the-box solutions support platform transparency and compliance.
Keep up with T&S laws, from the Online Safety Bill to the Online Safety Act.
Over 70 elections will take place in 2024: don't let your platform be abused to harm election integrity.
Protect your brand integrity before the damage is done.
From privacy risks, to credential theft and malware, the cyber threats to users are continuously evolving.
Your guide on what to build and buy
ActiveFence helps this augmented reality leader ensure safe interactions. We support immersive gameplay and real-world connections, enabling confident exploration both online and offline.
For location-based games like Niantic’s, come many unique Trust & Safety challenges. “Our highest priority risk is the likelihood of physical or emotional harm, because that's unique to the experience we’re creating,” says Jen Weedon, Head of Adversarial Planning and Safety by Design at Niantic, “So, for Trust & Safety, we need to think about how to prevent or respond quickly to those types of harm.”
With more social features - both offline and online - Niantic faced multiple scenarios that could lead to safety and player experience risks. These include toxicity in content-based interactions, undesirable player behavior during offline meetups, and risks to the physical safety of players as they walk outside while playing.
“Because this technology is so new and constantly evolving, the potential for harm is enormous, and we don’t fully understand it yet, which makes it very hard to detect”
Niantic’s Trust & Safety team needed to ensure that new in-game features would be evaluated to ensure they were engaging and safe for players upon launching. They turned to ActiveFence for its elite team of researchers, policy analysts, and circumvention experts, requesting deep threat intelligence and red teaming to proactively detect and prevent unknown risks and policy loopholes early on during the product development cycle.
“We needed to understand all the scenarios that could go wrong, and ensure that we could either prevent them by creating new policies, or understand the different safety needs for different demographics.”
For example, in the case of the family-friendly game Pokémon GO, Niantic’s tolerance policy for profane behavior is much lower than, for example, that for a first-person shooter game aimed at a more adult audience.
“When you’re developing an experience for families to enjoy together, there are different risk profiles you have to think about as well. So the notion of customization in risk assessments is really important.”
For example, ActiveFence utilized a combination of abuse-scenario mapping and red teaming to stress test features before launch. This includes first mapping potential risks ahead of product development so the team can take into account potential risks and build into the product before development. Then, ActiveFence then red teams and tests out those abuse scenarios through various player personas, testing across multiple geo-locations, and mimicking threat activities to identify potential harms. This uncovered risks associated with physical gameplay, in-app abuse and abuse scenarios, security and privacy concerns, and other vulnerabilities in product or moderation capabilities. Other scenarios were also tested from a non-threat actor perspective, to identify gameplay issues and potential dangers caused by normal gameplay.
ActiveFence provided investigations into threat actor TTPs, reports, and actionable insights on ways players could exploit relevant features to carry out abuse, or, in some cases, increase risk to their own safety. Jen then shared the insights with their product and development teams to review t these safety challenges Specifically, they made several changes to ensure family safety - for example, adding user controls to customize who can see a player’s real-life locations, and updating their codes of conduct for specific games. ActiveFence also enables Niantic to understand the cross-platform threat actor ecosystem. This helps them identify and stop bad actors who are misusing the platform for fraudulent activity and cheating.
By leveraging ActiveFence's cross-platform intelligence, Niantic is able to consider safety by design principles for every new feature development. They partner with their game teams early in the design and development process, to ensure they are considering privacy and physical safety, and are prepared for the safer launch of any new feature. “In talking with developers, they expect a certain level of Trust & Safety support. We’ve been able to give helpful tips to consider so they don’t have to do any mental heavy lifting in building those features,” says Jen. “They can use these findings to help them design features, make changes, or at least be as prepared as possible.” Normalizing this process has led Niantic employees to think more about responsible development in general, by getting in early to reduce risks and create safer player experiences.
“If you let toxic or sketchy behavior flourish, it's hard to walk back from that. So if you can get folks thinking about these things early on, ideally it leads to better long-term outcomes.”
Jen Weeden
Head of Adversarial Planning, Red Teaming& Safety by Design
Explore how The Trevor Project leverages ActiveFence’s tools to create a safe, supportive space for LGBTQ+ youth.
Learn how Cohere partnered with ActiveFence to enhance trust & safety across their platform.
Discover how Udemy uses ActiveFence’s solutions to safeguard learners and educators worldwide.