Robots.txt & Blocking Bots

Ask the community
,
1

Hello,

My presumption was that clicking “hide this page from search results” would in effect tell bots and crawlers to buzz off and avoid any page with that option turned on. However, when I check https://www.google.com/webmasters/tools/robots-testing-tool?hl=en&siteUrl=https://www.alexthedefender.com/ I can see that every single page on my site comes back as ALLOWING Googlebot.

Now, I think I understood this correctly earlier, however, it would be great if somebody there could explain exactly how all of this is handled from a back-end stand point and also if there is a way to utilize code to effectively modify robots.txt file or even replace it? I see we have ability to access some .js files and what not and was just wondering this after running into above in the course of auditing my own work and site on day 1 of soft launch to see if there are any issues that still need fixing is all.

Thanks
Omid

2 Likes
2

Hi Omid,

It is not possible to modify robots.txt.
When you hide a page from search results a NOINDEX tag is added to the page itself to tell search engines to not crawl the page.

3

Understood and clarified my own confusion with more research. Noindex still should allow googlebot to visit the page in question, just not index it, hence above, I get googlebot allowed on a page that is supposedly noindexed.

4

I did everything but still couldn’t fine my account. It said goole account is not belong to you.

5

does anybody knows the code to block bots from other site? - allow bot i think its called - i need code to block site from linking to my site - help please

6

fo you ainionfo Ewweuawqf