Privacy & Security

Keeping the Web Open and Private in the Bot Era

If you’ve been running into endless CAPTCHAS or website login requests lately, you’re not imagining things.

Websites, facing a rising tide of abusive traffic from bots, are adopting increasingly aggressive countermeasures, damaging user’s experience of the web, their privacy and open access to the web.

In this post, we’ll talk about a new initiative we’re launching with Cloudflare, other web browsers, and web stakeholders to address this challenge.

Privacy and access in tension

The fight for privacy on the web has made real progress. Browsers that put privacy first are eliminating third-party cookies, restricting fingerprinting, and hiding IP addresses, pushing back against the trackers.

But every step forward has come with a cost.

Users are seeing more CAPTCHAs, more demands to log in, and more outright block pages than ever before. Building privacy into the browser means dismantling the passive signals, like IP addresses and browser fingerprints that are used to profile users, but are also relied on by anti-abuse systems.

At the same time, sites are facing large increases in bot traffic. The response from websites is understandable; volumetric abuse like credential stuffing and spam can do real damage. But the result is a lose-lose: users face mounting friction and reduced privacy, while sites drive away the legitimate visitors they wanted to serve.

If nothing changes, users will increasingly be forced to choose between their privacy and their access to the web.

Proposals have been made to tackle this dilemma, by asking users to prove to sites that their devices and software are ‘trusted’. These proposals, such as Web Environment Integrity (WEI), transfer control of devices away from users and to a small handful of operating system and hardware vendors. This deprives users of choice and control and gives those gatekeepers control over which devices and software can access the web, the opposite of the open web, which Mozilla is working to protect.

Finding a better way forward

We think there’s a better way forward. It starts from a simple observation: bots cause harm because they operate at scale. To stop that kind of abuse, a site doesn’t need to know who you are, or that your device is restricted to running approved software. It only needs to know whether you’re staying within a reasonable rate limit.

To make a rate limit work, it must be hard for attackers to create new identities and reset their allowance. That’s one reason why sites demand an email address, a federated login or a device fingerprint: obtaining a new one is just costly enough to make the rate limit stick. The challenge is whether we can make rate limits work, without giving sites access to hard-to-change identifiers that also enable tracking.

Some sites naturally have a relationship with their users, like a subscription or a long-standing account. What if one of those existing relationships could quietly vouch for you elsewhere, so a site you’ve never visited could trust that you’re a real person within its limits, without learning who you are or even where the vouch came from?

For example, consider a VPN service. Many websites block VPN traffic entirely due to the high rates of abusive traffic blended with legitimate traffic. What if a VPN service could vouch for each of its subscribers? This would let sites manage a per-subscriber rate limit, meaning users get fewer roadblocks and sites get more of the legitimate traffic they want. Of course, this requires that the vouching system doesn’t enable sites to track VPN users, which would otherwise defeat the very purpose of using the VPN.

Enabling this kind of privacy-preserving vouching is already possible in a limited sense. Apple’s Private Access Tokens, built on a cryptographic protocol called Privacy Pass, let Apple devices receive single use tokens they can later present to websites without those visits being linked together.

However, Private Access Tokens have some critical shortcomings. First, like WEI, they rely on device attestation, the very hardware gatekeeping we are determined to avoid. Second, there’s no easy way to open up the system to let more parties vouch for users without compromising on user privacy, which means concentrating control in the hands of a few. To keep the web open, we need a system where any site can vouch for users, and where other sites can decide who they trust to vouch for users responsibly.

This is a much harder problem, but we think the cryptographic foundations exist to deliver it.

Anonymous credentials let one party issue you a credential that you can later present to a site a limited number of times, whilst preventing sites and issuers from tracking its use. It’s even possible to hide which party issued it, proving only that it came from a set of trusted issuers.

A fix is both essential and possible

Building this into a system for the open web, where any site could vouch and any site could set its own limits is challenging, but we believe it’s both possible and essential in order to defuse the tension between privacy and access, while avoiding centralising control in a small number of gatekeepers.

Working with other web stakeholders, including Cloudflare and other browsers, we’ve started designing such a system. For a deeper dive, read our post on Hacks, which goes into more detail about the problem space and the approach we’re working on.

Our goal is simple: fewer CAPTCHAs, fewer unnecessary blocks and fewer demands to identify yourself, without compromising on privacy. This is the kind of web that Mozilla built Firefox to offer: easy to use, private and open to all.