Block Wix Website from Wayback Machine and crawlers

Ask the community
, ,
1

Hi! I want to stop Wayback Machine/crawlers from taking a snapshot of my website. Apparently to do this one must add the following text to the robots.txt file:

User-agent: ia_archiver
Disallow: /

For some of my website pages I’ve disabled the function “Let search engines index this page” under SEO basics. There’s already text in my robots.txt file but when I click on it in the Wix editor all text disappears. There’s doesn’t seem to be a way to copy the code that already there so I’ve typed the code manually below. The text in my robots.txt file looks like this:

User-agent: *
Allow: /

Optimization for Google Ads Bot

User-Agent: AdsBot-Google-Mobile
User-Agent: AdsBot-Google
Disallow: / _api/*
Disallow: / _partials*
Disallow: /pro-gallery-webapp/v1/galleries/*

Sitemap: https://mysite.com/sitemap.xml

Where should I put the code to stop Wayback Machine/crawlers from taking snapshot of my site? Before or after the code that’s already in my robot.txt file?

Or should I delete the code that’s already there and only type in the code to stop crawlers?

Is it possible to exclude Wix pages from showing up in search results and stop crawlers? Or do I have to choose one? From Google developers:

“You can prevent a page or other resource from appearing in Google Search by including a noindex meta tag or header in the HTTP response.“
“For the noindex directive to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler. “

Text from: https://developers.google.com/search/docs/advanced/crawling/block-indexing

Grateful for help, thank you!

2

See the article Editing Your Site’s Robots.txt File for details.

If you need further details or clarification, I would recommend contacting Wix Customer Care , as they’ll be better able to assist you regarding this topic. They’re also smarter and nicer than I am.

3

Hi, thanks for the reply. I’ve already read this article and I couldn’t find answers to my questions.

The article says “Wix Customer Care cannot help you with any changes you make to your robots.txt file.” so I’m asking for help in this forum.

4

@capacath Wix Customer Care won’t necessarily be able to help you with any changes you make to your robots.txt file, but they might be able to provide different methods and techniques to help you out.

And, maybe someone here in the forum has some experience and ideas about your question.