BIG ISSUE - Blocking AI Bot Crawlers In Wix

Ask the community
, , ,
1

Recently, I blocked all the LLM user agents in Wix using the robots.txt file. Since my site is content-based, I can’t afford platforms like Perplexity and Claude claiming my content and sharing it with their userbase; it took me hours to write and create it.

These crawlers are notorious for crawling websites despite being blocked. Even after blocking the agents, platforms like ChatGPT, Claude, and Perplexity are stealing my content(So much for respecting the copyright referendum of LLMs).

I urge the Wix Team to address this issue and add a proper Firewall feature against these AI crawlers. They are not only ignoring the robots.txt but also straight-up stealing the content even if the site owners don’t want them to.

Cloudflair Integration could be a good choice as well, but I am afraid that after a year or so, they will start selling data as well.

If anyone wants to test it, this is my website - https://www.utilitytools.online/

And this is the current Robots.txt file - https://www.utilitytools.online/robots.txt

I asked the Chat platforms about my recent page, and they fetched it even after blocking their agent.

2

Hey there,

Currently as part of the LLMs.txt that is rolling out to users, there is an option to opt-out of providing guidance to AI tools.

image
image1066×202 15.6 KB

I checked the robots.txt and currently its set to default information so I’m not sure if you made changes or went back to default.

image
image1920×1080 107 KB

In the interim, setting up the robots.txt file to disallow AI bots from crawling the site might be best until you can access the llms.txt file.

3

Hi Mr. @Rob ,

Thanks for the Update, I really appreciate it.

Will the LLM.txt also block the web_fetch method used by these LLMs?

That’s the biggest issue, they bypass the legality with argument “We are using “web_fetch” ,this functions more like a regular browser visit rather than systematic crawling”

Blocking the web fetch would be a great feature to safeguard the data. I want to block these agents from Fetching/Scrapping my website’s data.

Thanks.