An update on residential proxies and the scraper situation

The following subscription-only content has been made available to you by an LWN subscriber. Thousands of subscribers depend on LWN for the best news from the Linux and free software communities. If you enjoy this article, please consider subscribing to LWN. Thank you for visiting LWN.net!

By Jonathan Corbet July 10, 2026

Our article "Fighting the AI scraper bot scourge", published in early 2025, discussed the problem of widespread scraping of web sites in search of training data for large language models and related projects. This activity overwhelms sites with traffic. Over a year after that article is published, the problem is still growing. The hammering of sites by shadowy actors has reached new heights, and the open web is becoming increasingly difficult to maintain. Where is this traffic coming from, and what can be done about it?

Residential proxies

As was described last year, scraper attacks come from a huge number of sources across the net. It is not unusual to see coordinated requests from millions of unique IP addresses over the course of a few hours, each of which hits the site at most two or three times. Attacker-controlled data, such as the user-agent field, is entirely fictional; each hit is meant to look like just another human with a web browser. There are ways to tell the difference — the bots usually do not fetch images or CSS, for example — but, by the time that determination is made, the address in question will not be used again. Blocking the address at that point is just a waste of time.

This traffic comes predominantly from residential and mobile networks, directed by central command-and-control nodes. Software is installed on ordinary systems that takes orders from a control node, fetches web pages on demand, and forwards the resulting data back to the controller. Much of the time, this activity occurs without the knowledge or consent of the owner of the device in question. The term "residential proxies" is used to describe systems that are used in this way.

There are a few different (on the surface, at least) types of operator running residential-proxy networks to attack web sites. One type is purely criminal, running scrapers on systems that have been compromised with some sort of malware. At the beginning of the year, Google acted to take down a bot network called IPIDEA and provided a lot of information about how these operations work. The shutdown of IPIDEA correlated with a significant reduction in scraper traffic here at LWN; things were relatively peaceful for a few months. That period of peace has since come to an end, though.

More recently, media-streaming devices have been identified as a major carrier of malicious scraping software. Sometimes the devices are compromised at the source; other times, they are just poorly secured and easily compromised after the fact.

The second sort of operator works more overtly, pretending to a degree of legitimacy and offering "ethically sourced" IP addresses. A company called Bright Data is one of the most prominent of these; it happily advertises its prowess at getting around web-site access controls and traffic limits. Bright Data offers a "free" VPN service; all that is needed is for the user to give Bright Data the ability to route traffic through the user's device — to become a part of the company's residential-proxy network, in other words. Every phone or other device that makes use of this VPN becomes yet another endpoint that will be used to attack web sites.

There are many other examples of this type of operator out there; often they offer a library that app developers can link into their offerings and be paid for hijacking their users' network connections. One of them even sent us a query about running an ad for its SDK on LWN; that was, it suffices to say, a short conversation. In general, these companies range from those that aspire toward some appearance of legitimacy, advertising "GDPR compliance" for example, to others that are just overtly sleazy.

While these residential-proxy networks are used for web-site scraping, it is worth emphasizing that these operators have the ability to run code that accesses resources on whatever networks millions of devices happen to be connected to. To assume that this type of access would only be used for scraping would be naive at best.

Then, of course, there are the high-profile companies developing models as their core business. These companies do their own scraping; the traffic that can be easily attributed to them is clearly identified in the user-agent field and, as a general rule, observes measures like robots.txt. They, too, will scrape an entire site, repeatedly, seemingly on the theory that articles written in 2003 might somehow have changed in the last day, but they do not generate overwhelming amounts of traffic from millions of systems and are not the biggest problem.

What isn't clear is who is using the residential proxies; somebody is paying them to run these attacks on web sites. There is no evidence (that I am aware of) that the frontier-model companies are using those networks. If were to turn out that they are doing so, though, the increase in global astonishment would barely register. Those companies are feeding their models somehow, they are not forthcoming about how they get their training data, and they have not distinguished themselves with their level of respect toward content creators — or toward anybody who might have concerns about their operations.

For every public model, though, there must be a vast number of undercover models. Many companies are surely trying to build their own; after all, we are reliably informed that AI is going to take over the world and the companies that come out on top of that race will be worth untold amounts of money. There must be shadowy government agencies in many countries working on their own models and groping for training data wherever they can find it. Large-scale criminal organizations (to the extent that they are distinct from governments) probably also want to have their own models. These tools are seen as weapons, and there is an arms race underway. The Internet as a whole is caught in the crossfire.

Defending the open Internet

In response to all of this, web-site operators have been scrambling to defend their sites while minimizing the effect on their actual users. Anubis, which attempts to fend off scrapers by requiring a proof of work, is now widespread. Other sites use commercial services, which sometimes make themselves known with a "prove you are human" button. Or sites force users to pick out squares containing streetlights (but only those with LED bulbs), place puzzle pieces, or hum a song while holding down the space bar. Many site features have been placed behind login gates or paywalls. Some sites attempt to actively poison the data sent to scrapers with tools like iocaine.

Both the need to set up and maintain these mechanisms, and the requirement that users cope with them to access a web site, constitute a heavy tax placed on the world as a whole by scrapers and those who pay them.

Recently, LWN was subjected what was, by far, the heaviest scraper attack yet. Thanks to the defenses that have been implemented, the site bore the traffic well enough that most actual readers probably did not even notice. There have been requests to describe the measures we have taken to defend the site; for obvious reasons we do not wish to discuss them in any detail. It is an arms race at this level too.

What we can say is that we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when you have millions of other people's machines to do the work on.

There is also a desire to not impede the operation of legitimate search engines, the Internet Archive, and other such groups. Some sites may add explicit allowlists to, for example, give the dominant search engine access to the site. Such measures have the effect of further entrenching a monopoly that already serves us poorly and should be avoided. We have, thus far, succeeded in that.

We have aggressively optimized parts of the site, and found ways to minimize expensive operations during times when the site is under attack. Anonymous readers may occasionally encounter one of those measures; logged-in users will not. Amusingly, the response time when the site is under attack is often better than during the calm times, when the defensive measures are dormant. We have learned better than to think that the problem is solved, though; consideration must be given to our next steps once the current measures are no longer effective.

On July 2, Google announced that it had, in coordination with the US Federal Bureau of Investigation and others, taken down a residential-proxy network called "NetNut". For the time being, that action would, indeed, seem to have succeeded in reducing the level of scraper attacks somewhat. Experience shows, though, that this welcome peace will only last so long. Google takes pains to point out that its Play Store will now check for NetNut-infected apps, but all of the major vendors are silent on the topic of why it is so easy to put apps with residential-proxy functionality into their app stores.

It would be good to find a more lasting solution before the entire Internet is driven behind defensive walls, and the open network that inspired so much creativity is lost. The industry that is driving these attacks seems entirely at ease with turning independent web sites into smoking craters after having pillaged their contents — an attitude that extends to the planet and its economies as well. Some of us, though, object to that idea and will fight against it. Someday, with luck, the world as a whole will decide to hold the companies behind large language models and related technologies to a minimal ethical standard. Until then, though, this behavior will continue, and we will have no choice but to defend ourselves against it.

---

How to avoid running a residential proxy (especially on Android phones)?

Posted Jul 10, 2026 16:25 UTC (Fri) by farnz (subscriber, #17727) [Link]

If you're using Google Play Services, turn on Play Protect and have that scan every so often - this will catch any apps that Google has found contain one of these proxies.

Other than that, you can ask Android to tell you how much mobile data an app used; go to "Settings", then "Apps", and it's in the "App Info" screen for individual apps. You can also disable "Background data" if you're suspicious of an app - that stops it using data when you don't have it open and are on mobile; IIRC, this also covers WiFi networks set as "metered", but ICBW on that one.

How to avoid running a residential proxy (especially on Android phones)?

Posted Jul 10, 2026 20:26 UTC (Fri) by mirabilos (subscriber, #84359) [Link]

How to avoid running a residential proxy (especially on Android phones)?

Posted Jul 10, 2026 21:42 UTC (Fri) by rgmoore (✭ supporter ✭, #75) [Link] (1 responses)

How can one avoid unintentionally running one of these proxies?

Some of the other posters have commented on how to avoid these kinds of proxies on your own system, but I want to point out that individual action won't be enough to solve the broader problem. I don't want to discourage anyone from trying to keep their own system clean, but we will never get rid of these kinds of proxies by encouraging each person with a mobile device to avoid them. You need to block access to a large majority of the available devices to make any real progress, and voluntary programs that involve real effort are never going to get that level of participation. This stuff needs to be taken care of at the system level- getting the apps out of app stores, making this kind of thing clearly illegal and prosecuting the offenders, etc.- not the individual user level.

How to avoid running a residential proxy (especially on Android phones)?

Posted Jul 10, 2026 23:40 UTC (Fri) by KJ7RRV (subscriber, #153595) [Link]

How to avoid running a residential proxy (especially on Android phones)?

Posted Jul 11, 2026 3:04 UTC (Sat) by duelafn (subscriber, #52974) [Link]

How to avoid running a residential proxy (especially on Android phones)?

Posted Jul 11, 2026 3:15 UTC (Sat) by marcH (subscriber, #57642) [Link]

Thanks for your efforts and keep up the good work!

Posted Jul 10, 2026 16:08 UTC (Fri) by wtarreau (subscriber, #51152) [Link] (2 responses)

Thanks for your efforts and keep up the good work!

Posted Jul 10, 2026 17:15 UTC (Fri) by daroc (editor, #160859) [Link] (1 responses)

Thanks for your efforts and keep up the good work!

Posted Jul 10, 2026 17:46 UTC (Fri) by wtarreau (subscriber, #51152) [Link]

Preventing this at the app store level is hard

Posted Jul 10, 2026 20:42 UTC (Fri) by roc (subscriber, #30627) [Link] (2 responses)

Preventing this at the app store level is hard

Posted Jul 10, 2026 22:29 UTC (Fri) by rgmoore (✭ supporter ✭, #75) [Link] (1 responses)

Google's rules already forbid non-user-authorized "residential proxies" But if the user gives genuine consent to it, should they still be banned? That's a tough call.

No, they shouldn't be allowed. Banning them is at least as much about protecting the whole internet from their pernicious effects as it is about protecting the individual user, so it shouldn't be up to the individual user any more than emissions controls should be up to the individual driver.

I would also argue that for any user to give "genuine consent", they would need to be given the option to have the same app without the proxy. If you let the author condition use of their app on accepting the proxy, consent is at least somewhat coerced.

Preventing this at the app store level is hard

Posted Jul 10, 2026 23:11 UTC (Fri) by Wol (subscriber, #4433) [Link]

IP blocks should be neither authentication nor authorization

Posted Jul 10, 2026 20:59 UTC (Fri) by quotemstr (subscriber, #45331) [Link]

Anubis doesn't seem to be working anymore here

Posted Jul 10, 2026 22:34 UTC (Fri) by koverstreet (subscriber, #4296) [Link] (3 responses)

Anubis doesn't seem to be working anymore here

Posted Jul 11, 2026 0:21 UTC (Sat) by joey (guest, #328) [Link] (2 responses)

Anubis doesn't seem to be working anymore here

Posted Jul 11, 2026 0:58 UTC (Sat) by koverstreet (subscriber, #4296) [Link] (1 responses)

Anubis doesn't seem to be working anymore here

Posted Jul 11, 2026 3:32 UTC (Sat) by danielbaumann (subscriber, #38804) [Link]

Have your cake and eat it too

Posted Jul 11, 2026 3:31 UTC (Sat) by marcH (subscriber, #57642) [Link]