X Updates Its Terms To Ban Crawling And Scraping: Protecting Data And AI Models

In a move to protect its data from being used to train AI models, Elon Musk-owned X, formerly known as Twitter, has updated its terms of service effective from September 29. The new terms explicitly prohibit any form of crawling or scraping without prior written consent. The previous version of the terms allowed crawling in accordance with robots.txt, but scraping was prohibited without consent.

New Terms to Prevent Unauthorized Access

The updated terms of service now state, “NOTE: crawling or scraping the Services in any form, for any purpose without our prior written consent is expressly prohibited.” This change aims to prevent unauthorized access to X’s data which has become a concern for the platform.

In addition to the updated terms, X has also modified the robots.txt file, which instructs robot crawlers on what parts of the site they are permitted to visit. The new robots.txt file, however, only allows Google’s crawler bot access to the site, removing instructions for all other crawler bots.

Twitter’s Relationship with Google

In 2015, Twitter had a firehose deal with Google to include tweets in search results. It is unclear if the terms of that agreement have changed with the transition to X. Further details about the nature of the agreement and any potential updates are yet to be clarified.

X has applied further restrictions by disallowing crawlers from retrieving information such as likes, retweets, media, photos, and account-specific data. This indicates a strong stance against companies using Twitter data for training AI models without consent.

Musk’s Fight Against Data Scraping

Elon Musk has been vocal in his opposition to companies scraping data from Twitter/X to train AI models. In April, he even threatened to sue Microsoft for the alleged illegal use of Twitter data. In July, he filed a lawsuit against unknown companies engaged in data scraping.

To address concerns around data privacy, X recently updated its privacy policy to state that it might use public data, including tweets, to train AI models. Musk has mentioned that his latest venture, xAI, founded in July, will utilize public data for model training. The revised privacy policy also includes provisions for the collection of biometric data, educational background, and employment history of users.