.Google's John Mueller answered a question concerning why Google marks web pages that are actually forbidden from creeping through robots.txt as well as why the it is actually safe to overlook the related Search Console documents about those crawls.Crawler Web Traffic To Concern Specification URLs.The individual inquiring the question chronicled that robots were creating hyperlinks to non-existent concern parameter Links (? q= xyz) to web pages along with noindex meta tags that are actually likewise shut out in robots.txt. What caused the question is actually that Google.com is crawling the hyperlinks to those web pages, getting shut out through robots.txt (without noticing a noindex robots meta tag) then getting turned up in Google Search Console as "Indexed, though blocked out through robots.txt.".The person inquired the following concern:." However below is actually the large question: why will Google.com index pages when they can't also view the web content? What is actually the conveniences in that?".Google.com's John Mueller verified that if they can not crawl the page they can not view the noindex meta tag. He likewise produces an interesting mention of the website: search operator, advising to ignore the end results considering that the "typical" individuals will not see those results.He created:." Yes, you are actually appropriate: if our team can not creep the web page, our experts can't see the noindex. That stated, if we can not creep the web pages, then there's certainly not a whole lot for our team to mark. Therefore while you might view a few of those pages with a targeted site:- question, the typical user won't see all of them, so I would not bother it. Noindex is also fine (without robots.txt disallow), it simply suggests the URLs are going to end up being actually crept (and end up in the Browse Console record for crawled/not listed-- neither of these statuses create problems to the rest of the web site). The important part is actually that you do not create them crawlable + indexable.".Takeaways:.1. Mueller's answer validates the restrictions in operation the Internet site: hunt accelerated search operator for diagnostic explanations. Among those factors is since it is actually certainly not linked to the regular hunt index, it is actually a distinct thing entirely.Google.com's John Mueller discussed the site search operator in 2021:." The brief response is that a web site: question is certainly not suggested to become complete, nor made use of for diagnostics objectives.An internet site query is actually a details sort of hunt that limits the end results to a particular website. It is actually primarily just the word site, a bowel, and afterwards the internet site's domain.This inquiry confines the outcomes to a details web site. It's not suggested to be a thorough compilation of all the webpages coming from that website.".2. Noindex tag without utilizing a robots.txt is actually alright for these sort of circumstances where a robot is connecting to non-existent pages that are receiving uncovered through Googlebot.3. URLs with the noindex tag will definitely create a "crawled/not listed" entry in Explore Console and that those won't possess a bad impact on the remainder of the website.Review the concern and also respond to on LinkedIn:.Why would Google index web pages when they can't even find the information?Featured Photo by Shutterstock/Krakenimages. com.