Robots.txt - please help

Ask the community
,
1

I am trying to advertise on gogole merchant centre and am being told that my url for images are disallowed:

Please update the robots.txt file on your web server to allow Google’s crawler to fetch the provided images. The robots.txt file can usually be found in the root directory of the web server (e.g. http://www.example.com/robots.txt ). In order for us to access your whole site, ensure that your robots.txt file allows both user-agents ‘Googlebot-image’ (used for images) and ‘Googlebot’ (used for web pages) to crawl your site. You can do this by changing your robots.txt file as follows:
User-agent: Googlebot
Disallow:
User-agent: Googlebot-image
Disallow:

does anyone know how to do this, as there is no information on wix

thank you, I have been trying to sort this now for 2 weeks and its driving me mad

1 Like
2

https://support.wix.com/en/article/robotstxt-file

You might be thinking that it is also a good way to hide pages or information you’d rather be kept confidential and you don’t want to appear on Google. This is not what the robots.txt file is for, as these pages you want to hide may easily appear by circumventing the robots.txt instructions, if for instance another page on your site links back to the page you don’t want to appear.

While it is important to have this file, your site will still function without it and will still usually be crawled and indexed. An important reason it’s relevant to your site’s SEO because improper usage can affect your site’s ranking .

What’s improper usage?

Your robots.txt might be in conflict with your sitemap.xml file (your robots.txt file contradicts your sitemap – if something is in your sitemap, it should not be blocked by your robots file).

Using it to block private or sensitive pages instead of password protecting them.

robots.txt on Wix

If you are on Wix, you should know that Wix automatically generates a robots.txt file for every site created using its platform. You can view this file by adding ‘/robots.txt’ to your root domain ( www.domain.com/robots.txt ) (replacing domain.com with your actual domain name). If you see what’s in your robots .txt file, you will realise that Wix has added files related to the structure of the sites, such as ‘noflashhtml’ and ‘backhtml’ links. Since they do not contribute to your site’s SEO, they do not need to be crawled.

It is not possible to edit the robots.txt file of your Wix site. However, you may add a ‘noindex’ tag (so it doesn’t appear in search results) to an individual page of your Wix site.

If you don’t want a specific page of your site to appear in search engine results, you can hide it in the Page SEO section:

  • Click the Pages Menu from the top bar of the Editor

  • Click the page you want to hide

  • Click the Show More icon

  • Click Page SEO

  • Click the toggle next to Hide this page from search results .

  • Click Done

If you choose to password protect a page, this too prevents search engines from crawling and indexing that page. This means that password protected pages do not appear in search results

This means that people cannot find your page when searching keywords and phrases in search engines.

3

Thank you for your reply, google have just come back to me to say they cant help and that I have to do this through wix. I don’t want to hide pages, google needs to be able to see the images to advertise them on the google merchant centre, so I need to allow Googlebot and Googlebot-image to crawl my website