The Solution

Niantic’s Trust & Safety team needed to ensure that new in-game features would be evaluated to ensure they were engaging and safe for players upon launching. They turned to ActiveFence for its elite team of researchers, policy analysts, and circumvention experts, requesting deep threat intelligence and red teaming to proactively detect and prevent unknown risks and policy loopholes early on during the product development cycle.

“We needed to understand all the scenarios that could go wrong, and ensure that we could either prevent them by creating new policies, or understand the different safety needs for different demographics.”

For example, in the case of the family-friendly game Pokémon GO, Niantic’s tolerance policy for profane behavior is much lower than, for example, that for a first-person shooter game aimed at a more adult audience.

“When you’re developing an experience for families to enjoy together, there are different risk profiles you have to think about as well. So the notion of customization in risk assessments is really important.”

For example, ActiveFence utilized a combination of abuse-scenario mapping and red teaming to stress test features before launch. This includes first mapping potential risks ahead of product development so the team can take into account potential risks and build into the product before development. Then, ActiveFence then red teams and tests out those abuse scenarios through various player personas, testing across multiple geo-locations, and mimicking threat activities to identify potential harms. This uncovered risks associated with physical gameplay, in-app abuse and abuse scenarios, security and privacy concerns, and other vulnerabilities in product or moderation capabilities. Other scenarios were also tested from a non-threat actor perspective, to identify gameplay issues and potential dangers caused by normal gameplay.

ActiveFence provided investigations into threat actor TTPs, reports, and actionable insights on ways players could exploit relevant features to carry out abuse, or, in some cases, increase risk to their own safety. Jen then shared the insights with their product and development teams to review t these safety challenges Specifically, they made several changes to ensure family safety - for example, adding user controls to customize who can see a player’s real-life locations, and updating their codes of conduct for specific games. ActiveFence also enables Niantic to understand the cross-platform threat actor ecosystem. This helps them identify and stop bad actors who are misusing the platform for fraudulent activity and cheating.

The Impact

By leveraging ActiveFence's cross-platform intelligence, Niantic is able to consider safety by design principles for every new feature development. They partner with their game teams early in the design and development process, to ensure they are considering privacy and physical safety, and are prepared for the safer launch of any new feature. “In talking with developers, they expect a certain level of Trust & Safety support. We’ve been able to give helpful tips to consider so they don’t have to do any mental heavy lifting in building those features,” says Jen. “They can use these findings to help them design features, make changes, or at least be as prepared as possible.” Normalizing this process has led Niantic employees to think more about responsible development in general, by getting in early to reduce risks and create safer player experiences.

“If you let toxic or sketchy behavior flourish, it's hard to walk back from that. So if you can get folks thinking about these things early on, ideally it leads to better long-term outcomes.”