Meta’s New Terms Expand Crawling Rights Beyond Advertisers via @sejournal, @MattGSouthern
All Paid Media PPC News Social MediaAdvertising Video Advertising Columns Ask A PPC ExpertNEW PPC Pulse Webinar How To Create Demand You Can Actually Convert (And Prove) Join this webinar to choose a channel to test, set a starting budget, and identify results to track before investing more. Digital Marketing All Digital Marketing Analytics & Data Ecommerce Lead Generation Content Marketing Social Platforms Google YouTube Reddit LinkedIn TikTokNEW WordPress Other CMS Platforms Guide SEO vs. AI Search: Strategy Prioritization Quiz A 3-minute quiz that diagnoses your SEO and AI readiness gaps and returns a ranked action plan specific to your site. SEJ Live Webinars Rundowns PodcastNEW Ebooks All Resources Guide AI Max Invalid Traffic Doubled, Weeks Before Black Friday Invalid traffic on AI Max search doubled to 5.28%. Get the channel-by-channel retail benchmarks before Black Friday. SEJ ⋅ Facebook Meta’s New Terms Expand Crawling Rights Beyond Advertisers Meta's updated terms let it crawl business users' sites and train its products on them. Advertisers already granted similar rights, and the new terms cover any business use. Meta's crawler documentation lists separate robots.txt tokens for AI training and Meta AI search. New terms let Meta crawl sites tied to business users and use the content to train its products, widening rights ad buyers had already granted. SEJ STAFF Matt G. Southern 2 hours ago ⋅ 6 min read SEJ STAFF Matt G. Southern Senior News Writer at Search Engine Journal Bio Follow Meta’s updated Commercial Terms, which take effect Oct. 30, give the company permission to crawl websites associated with businesses that use its products and to use that content to train those products. The terms apply to any business use of Meta’s products, from running ads to managing a Facebook Page. Nieman Lab reported this development on Oct. 15. A total of three law professionals have viewed the terms for this media outlet and believe that language in these terms could support an argument by Meta that it is permitted to scrape websites of news sites and use the scraped content to train its AI products. On Oct. 16, Meta spokesperson Andy Stone responded on X to a post summarizing that report and called the summary false. He wrote that the update is meant to make sure Meta can pull accurate information about products for sale on third-party websites when it makes ad and content recommendations on its platforms, and that it’s “not about scraping news sites.” Meta’s ad terms already let it crawl advertisers’ websites for uses that include training. The October version reaches businesses that don’t buy ads. The new crawling paragraph in the Commercial Terms lets Meta, or anyone acting for it, use web scrapers, bots and other automated tools to crawl and collect content from domains and pages associated with a business’s products or services. It also covers URLs made available through the business’s use of Meta’s products or through users’ use of them. The clause also lets Meta “create test credentials” to reach those pages. Meta may use the content to develop, improve, research and train its products, among other listed uses. Meta’s own list of ways businesses can utilize Meta services includes advertising, marketing messages, listing your products on Meta platforms along with Meta-created tools including Ads Manager and Meta Business Suite. The very same crawl permission is included in the revised Self-Serve Ad Terms, effective Oct. 30. A continued usage of Meta’s service for business purposes will be interpreted by Meta to mean that you have agreed to the new terms. The terms say that if you disagree with these new terms, you may cease utilizing Meta’s products for business purposes. Media and intellectual property attorney Matt Topic, representing The Intercept in their copyright case versus OpenAI, stated to Nieman Lab that while there are no assurances if a court will ultimately enforce this provision, publishers should assume it will. The current Commercial Terms, dated March 6, 2024, have no crawling clause. Meta’s ad terms do. The current Self-Serve Ad Terms, dated Aug. 22, 2025, say that placing an ad order gives Meta the right to crawl content on the advertiser’s domains, webpages or URLs. The listed uses include developing advertising and other business products, maintaining integrity, and “research and training.” Meta’s Ad Creative Generative AI Terms (May 6, 2024) permit Meta to crawl URLs provided by advertisers, for use with their ads, as part of using Meta’s generative AI ad tools. Also the Product Catalog Terms allow Meta to periodically update product catalogs from catalog URLs. In addition, Meta has announced in its Business Help Center notice that the Product Catalog Terms will be moved into the Commercial Terms, and the Ad Creative Generative AI Terms will move into the Self-Serve Ad Terms to “reduce overlap.” The announcement did not reference crawling or training. Also, Meta’s Seller Agreement, last updated May 26, 2026, already permits Meta to utilize automated software, including creating test credentials, to obtain information from URLs that sellers provide. The agreement says this helps Meta better assist the seller with optimizing their listings. Meta’s spokesperson told Nieman Lab the new terms won’t change how the company crawls publishers and that “content owners can still manage how their content is accessed by using robots.txt.” Stone made both points in his post as well. Neither the updated Commercial Terms nor the updated Self-Serve Ad Terms mention robots.txt. Meta’s crawler documentation says Meta-ExternalAgent crawls for uses such as training foundation AI models or improving products by indexing content directly. Meta-WebIndexer improves Meta AI search results, and Meta says allowing it helps the company cite and link to your content in Meta AI responses. According to the documentation, Meta-ExternalFetcher may bypass robots.txt when fetching pages at a user’s request. FacebookExternalHit, which creates link previews, may also bypass it while performing security or integrity checks. Meta’s Oct. 30 crawling and training grant comes with regular business uses of Meta’s products; thus, it includes brands and clients who have a Facebook Page or are in the Business Suite but do not buy ads. The Robots.txt file is what Meta points to as the method to control its crawling. The Crawler Documentation listed above references Meta-AI training crawlers (Meta-ExternalAgent) and Meta AI Search Crawlers (Meta-WebIndexer); therefore, you may block the Meta-ExternalAgent crawler while allowing the Meta-WebIndexer crawler. Additionally, the new terms do not indicate if a business’s robots.txt settings restrict what it agrees to share by accepting these terms. Meta’s new Commercial Terms are set to go into action on Oct. 30, giving website operators who use Meta for business two weeks in order to make their decision concerning whether or not they will continue to utilize Meta’s products for business. In addition, during this time, operators should review how their robots.txt files treat each Meta crawler. The European Publishers Council (EPC) is also writing to the European Commission regarding Meta’s new terms. EPC is specifically concerned that Meta may be allowed by its terms to claim permission to use an article when a user shares a link to that story via social media. Add SEJ as a preferred source on Google Read Full Bio SEJ STAFF Matt G. Southern Senior News Writer at Search Engine Journal Follow on YouTube. Matt G. Southern is the Senior News Writer at Search Engine Journal, where he’s covered Google, SEO, ... Breaking news and insights to help you lead search strategies. Monopoly: A Ruling Against Google Could Benefit The Open Web From Search To Discovery: Why SEO Must Evolve Beyond The SERP An In-Depth Look At Google Spam Policies Updates And What Changed The AI Search & SEO News That Matters Breaking news and insights to help you lead search strategies. Breaking news and insights to help you lead search strategies. In a world ruled by algorithms, SEJ brings timely, relevant information for SEOs, marketers, and entrepreneurs to optimize and grow their businesses -- and careers.
Source: Search Engine Journal
This article has been carefully curated and reformatted for educational and informational purposes. Full credit goes to the original publisher.
📚 Visit more helpful articles on Joab Peters Blog
No comments