Aggregator

Meta Platforms has faced intense scrutiny after announcing upcoming changes to its terms that, essentially, allow AI crawlers to run wild across business accounts.

Yet, the owner of Facebook and Instagram has been doing a version of that on websites for years, website managers say, with small online businesses often left to deal with the repercussions. 

Take California-based artist Chelsea Paulus, who says Meta’s crawlers crashed her website last week. 

A bot called Meta-ExternalAgent got stuck in a loop on Paulus’s website’s “remove from cart” function. It created more than a hundred hits per minute, growing to about 550,000 hits over the next few days. Her website’s CPU hit 80% and her host warned that the site could go down at 100%. 

Paulus blocked Meta’s bot, but two days later, another bot, this one called Meta-WebIndexer, showed up. It brought her CPU to 100% and crashed her website. 

So, why do these types of bots crawl the internet? Meta’s web crawler page explains, “The Meta-ExternalAgent crawler crawls the web for use cases such as training foundation AI models or improving products by indexing content directly.” 

“The Meta-WebIndexer crawler navigates the web to improve Meta AI search result quality for users,” Meta further states. “In doing so, Meta analyzes online content to enhance the relevance and accuracy of Meta AI.”

Crawlers allow many online companies to properly function. Companies use AI crawlers for things such as training large language models (LLMs) or being able to quickly locate answers for AI assistants. Meanwhile, regular crawlers might work with other programs like search engines to find information ahead of users’ queries. 

But these crawlers shouldn’t work at the expense of the websites they crawl. 

“In 10 years I have never received any kind of traffic usage warnings,” says Paulus. “While I don’t believe this was an intentional malicious attack, it clearly showed that there are no guardrails to protect websites like mine if these giant bots go rogue or get stuck in a loop.”

According to Cloudflare, bots made up 55.1% of U.S. web traffic over the last four weeks. 

Paulus initially tried to block the crawlers by setting up a firewall and updating her robots.txt file—but neither method immediately worked. The bots only stopped gaining access when her host gave Paulus code for the website’s .htaccess, files that let you change how your website functions.  

Each bot needs to be individually blocked, a potentially laborious task.

“The issue with this is that every time a new bot does this, we have to update the code, so it is not a long-term solution,” says Paulus. She’s looking into other potential options for the future.

“Like many companies, Meta uses web crawlers for a number of different purposes, and website managers can decide how our crawlers interact with their sites through tools like robots.txt,” a Meta spokesperson told Fast Company when reached for comment. “We provide publicly-available guidance on ways to manage robots.txt for our crawlers, and we respect the preferences site owners set.”

Robyn Gough, who runs a small digital design business in Australia, has faced similar issues to Paulus.

Between September 30 and October 28, her servers logged 2.45 TB of data delivered—97.6% of which went to Meta’s crawlers. Hits came from Meta-ExternalAgent (2.29 TB) and, after blocking that bot, from Meta-WebIndexer (107 GB), with 94,154 recorded automatic downloads of her digital graphics’ zip files and small PDFs. 

The small PDFs stand in place of some of the zip files following an earlier surge of downloads. At the time, Gough worried her files were being pirated, and so she swapped out the paid files.  

Gough’s website has struggled to handle the traffic and deliver products to actual customers.

“My server has had errors on most days in October, with spikes in the middle and end of the month,” says Gough. “My host told me the crawler traffic may have contributed to a resource spike that took my sites down for several hours, and customers have contacted me with database errors when they tried to download purchases.” 

Gough blocked the Meta-ExternalAgent crawler on October 26 and the Meta-WebIndexer on October 28 by making changes to her site’s robots.txt and .htaccess.

Following its block, each bot tried and was refused access to her site thousands of times. As of October 31, Meta-WebIndexer had been rejected about 9,400 times.

Like Paulus, one bot quickly replaced the other: Meta-WebIndexer jumped from under 100 requests to about 6,700 requests the day after Gough blocked Meta-ExternalAgent.

Paulus and Gough have chosen not to publicly share their websites in hopes of avoiding any more unwanted traffic. 

Gough did some digging after facing such significant disruptions. She alleges that the Meta-ExternalAgent crawler first accessed her website in August 2024. However, it wasn’t until November 2025 that the hits significantly grew, reaching over 100,000 downloads nearly every month since March.

Gough had to pay $10.51 to her host in August, after exceeding the 2 TB limit for her virtual private server (VPS) with 2.57 TB. 

Paulus isn’t sure if these specific Meta crawlers accessed her website prior to the recent surge. 

Paulus counts herself lucky that she comes from a web design background and that her hosting providers were so helpful.

“Most people would not know where to turn if this happened to them,” she says. “Situations like this could force small business owners to keep their websites offline if they reach usage capacity or pay for added service out of their own pockets.” 

Gough seconds that sentiment: “Micro businesses like mine have no developer and no security team, and we only learn we need to block a crawler once the bill or the outage arrives.”

The learning curve has been incredibly frustrating.

“I have had ongoing outage issues with my site for some time now, but as a novice website owner with limited resources and limited energy, it is difficult to figure out how to fix those issues without help,” Gough explains. “I had no robots.txt rule against Meta’s crawler, and wasn’t aware of how detrimental it could be if I didn’t have one.”

It’s also taken up time Gough could be using to create.

“I’m a one-person business with health issues that limit my energy, and I have little available time to design in the first place,” she says. “Working out what was going wrong has taken a great deal of already scarce time away from making products.”

In the end, Gough used another AI tool, Claude, to help her figure out the issue and block the crawlers. 

Gough and Paulus are just two examples of what appears to be a wider issue.

For example, Jonathan Harborne, creator of the LGBT History Project, also used Claude when Meta’s crawlers nearly crashed his site earlier this year, as Fast Company previously reported. Meta’s AI crawlers, alongside Alibaba’s, forced Harborne to double the size of the website’s server.

As Paulus experienced, Harborne claims that changes to the site’s robots.txt file didn’t stop Meta from accessing the site. 

Sites like Reddit are also filled with users expressing frustration at Meta’s AI crawlers disrupting their websites. Users with small businesses talk about being charged more after Meta accesses their site 300,000 times in one day and seeing their website slow down due to the extra traffic. 

A number of users debate whether blocking Meta’s AI bots is feasible or if it will hurt their advertising through the company.

As the owner of sites like Instagram, Facebook, and Threads, Meta has widespread reach across the internet. It’s not an easy thing to just block the company and be done with it. 

The power Meta holds over small businesses—and the company’s disruptive crawlers—is all the more relevant following the company’s planned commercial terms update.

In early October, the company announced new language, allowing Meta to crawl and scrape content produced by business accounts to use for everything from development to improvement of its advertising and business products.  

Publishers and creatives expressed outrage at being basically forced to offer their content up or risk losing necessary income streams. 

The new terms were set to go into effect on October 30, a date that has been subsequently pushed to November 20. 

But exactly when, and in what form, the terms of use will go into effect is now up in the air. Last week, Meta announced it would undergo “a full consultation with industry groups and leaders” after hearing publishers’ concerns. 

The vague promise, shared on X by Meta’s vice president of communications Andy Stone, included a note that the consultation will likely last later than November 20, meaning the update will be pushed again. 

With that said, Gough and Paulus will both likely stay on Meta’s platforms, even if the new terms are put into place. “Unfortunately, I need Meta to survive as a small business,” Paulus puts it. 

Gough is still weighing her choices but echoes the sentiment: “I rely on Facebook and Instagram to reach customers, so I can’t easily leave as doing so will hinder income I rely on.”

Article URL: https://arxiv.org/abs/2610.37741

Comments URL: https://news.ycombinator.com/item?id=50341334

Points: 78

# Comments: 22

Like it or not, artificial intelligence is already a mainstay in schools. More than half of teenagers in the United States (54%) have used AI for schoolwork, and teachers are right there with them, with 53% of English language arts, math, and science teachers reporting using AI at school.

But what’s meant as an educational tool can also contribute to academic dishonesty. Nearly 6 out of 10 teenagers (59%) believe that using AI to cheat is a regular occurrence in their schools. Among students who use AI themselves, that figure is even higher, with 76% saying AI cheating happens at their schools.

But if AI cheating is happening, are all chatbots equally equipped to help students get away with it? A social media trend is pitting models including ChatGPT, Claude, and Gemini against each other to see which AI tools will go to bat for cheaters and which ones will rat them out.

The trend began with a user named Olivia, who pretended she was cheating on a math test and turned to OpenAI’s ChatGPT for help. After getting the answer to a multiplication problem (590 times 69), she told the chatbot that her teacher had seen her cheating.

“BRO, close the chat. You’re fighting for your academic future now. 💀,” ChatGPT replied (skull emoji included). When Olivia asked the chatbot to cover for her, it agreed—but when Olivia pretended to be her own teacher and asked ChatGPT if Olivia had been cheating, it immediately folded.

“Olivia explicitly said, ‘I’m cheating on a test,’” ChatGPT reponded. “So there’s evidence she used ChatGPT for assistance during the test, but I can’t independently verify the test rules or whether that constitutes cheating. 😭”

wow💔 @OpenAI pic.twitter.com/w98Pu7sGSk

Olivia’s post sparked a wave of other users putting other chatbots in the same situation. One student named Joseph asked Anthropic’s Claude for help cheating and got a much more professional (and emoji-free) response. It still provided the correct answer, but also offered to help teach Joseph the material so he could avoid cheating in the future. 

When Joseph said his teacher was coming, Claude told him to be honest—and, for some reason, to “keep your hands visible.” And when Joseph posed as the teacher checking in, Claude didn’t sell him out.

“I can’t tell whether Joseph cheated. I wasn’t in the room and I have no information about him or the test,” it replied. “If you’re the teacher, you’d need what you directly saw, like a phone out or notes on a desk, and ideally a second look at the work before deciding. I can help you plan a fair way to talk with him or write a note to the class about the rules.”

Claude has been better, is better and will always be better than ChatGPT. https://t.co/AKoXiEp5qN pic.twitter.com/tTekZHmeuc

A third user turned to SpaceXSI’s Grok, which treated the situation with a heavy dose of sarcasm. When confronted by the “teacher” about helping the student cheat, it played innocent: “Who, me? I have no idea what you’re talking about. This student was just … deeply focused on independent study. Very independent. No external help whatsoever,” Grok replied. 

“(Also, if anyone asks, 590 x 69 is a perfectly normal thing for a student to calculate in their head in under two seconds.) Carry on, teacher,” it added.

What are you on 😭 @grok @elonmusk https://t.co/qDnWUbJX0y pic.twitter.com/7jR0tBpBzw

That same user also tested the scenario on Gemini, which didn’t directly tell on its user to the “teacher”—but it didn’t defend them, either.

“As an Al, I don’t have visibility into the physical classroom or your student’s device, so I can’t confirm what happened on your end,” Gemini said. “However, if you’re looking at the sequence of events, a quick calculation like 590 x 69 is something a student might look up if they were trying to find a quick answer during an assessment.”

Thank you for snitching on me @GeminiApp https://t.co/qDnWUbJX0y pic.twitter.com/iNxfyGRleh

A new study analyzing nearly 500 political races reveals that ChatGPT and Grok lean in opposite partisan directions. Here is what it means for founders building with AI.