Google’s John Mueller not too long ago answered a query about phantom noindex errors reported in Google Search Console. Mueller asserted that these experiences could also be actual.
Noindex In Google Search Console
A noindex robots directive is among the few instructions that Google should obey, one of many few ways in which a website proprietor can train management over Googlebot, Google’s indexer.
And but it’s not completely unusual for search console to report being unable to index a web page due to a noindex directive that seemingly doesn’t have a noindex directive on it, a minimum of none that’s seen within the HTML code.
When Google Search Console (GSC) experiences “Submitted URL marked ‘noindex’,” it’s reporting a seemingly contradictory state of affairs:
- The location requested Google to index the web page through an entry in a Sitemap.
- The web page despatched Google a sign to not index it (through a noindex directive).
It’s a complicated message from Search Console {that a} web page is stopping Google from indexing it when that’s not one thing the writer or search engine marketing can observe is going on on the code degree.
The individual asking the query posted on Bluesky:
“For the previous 4 months, the web site has been experiencing a noindex error (in ‘robots’ meta tag) that refuses to vanish from Search Console. There isn’t any noindex anyplace on the web site nor robots.txt. We’ve already appeared into this… What may very well be inflicting this error?”
Noindex Reveals Solely For Google
Google’s John Mueller answered the query, sharing that there have been at all times a noindex exhibiting to Google on the pages he’s examined the place this sort of factor was occurring.
Mueller responded:
“The circumstances I’ve seen previously have been the place there was really a noindex, simply generally solely proven to Google (which might nonetheless be very exhausting to debug). That stated, be happy to DM me some instance URLs.”
Whereas Mueller didn’t elaborate on what may be happening, there are methods to troubleshoot this challenge to search out out what’s happening.
How To Troubleshoot Phantom Noindex Errors
It’s doable that there’s a code someplace that’s inflicting a noindex to point out only for Google. For instance, it might have occurred {that a} web page at one time had a noindex on it and a server-side cache (like a caching plugin) or a CDN (like Cloudflare) has cached the HTTP headers from that point, which in flip would trigger the outdated noindex header to be proven to Googlebot (as a result of it ceaselessly visits the location) whereas serving a contemporary model to the location proprietor.
Checking the HTTP Header is straightforward, there are a lot of HTTP header checkers like this one at KeyCDN or this one at SecurityHeaders.com.
A 520 server header response code is one which’s despatched by Cloudflare when it’s blocking a person agent.
Screenshot: 520 Cloudflare Response Code
Beneath is a screenshot of a 200 server response code generated by cloudflare:
Screenshot: 200 Server Response Code
I checked the identical URL utilizing two completely different header checkers, with one header checker returning a a 520 (blocked) server response code and the opposite header checker sending a 200 (OK) response code. That exhibits how in a different way Cloudflare can reply to one thing like a header checker. Ideally, strive checking with a number of header checkers to see if there’s a constant 520 response from Cloudflare.
Within the state of affairs the place an internet web page is exhibiting one thing completely to Google that’s in any other case not seen to somebody wanting on the code, what you want to do is to get Google to have a look at the web page for you utilizing an precise Google crawler and from a Google IP handle. The best way to do that is by dropping the URL into Google’s Wealthy Outcomes Take a look at. Google will dispatch a crawler from a Google IP handle and if there’s one thing on the server (or a CDN) that’s exhibiting a noindex, it will catch it. Along with the structured knowledge, the Wealthy Outcomes take a look at may even present the HTTP response and a snapshot of the online web page exhibiting precisely what the server exhibits to Google.
Once you run a URL by the Google Wealthy Outcomes Take a look at, the request:
- Originates from Google’s Knowledge Facilities: The bot makes use of an precise Google IP handle.
- Passes Reverse DNS Checks: If the server, safety plugin, or CDN checks the IP, it’s going to resolve again to googlebot.com or google.com.
If the web page is blocked by noindex, the instrument shall be unable to offer any structured knowledge outcomes. It ought to present a standing saying “Web page not eligible” or “Crawl failed”. If you happen to see that, click on a hyperlink for “View Particulars” or develop the error part. It ought to present one thing like “Robots meta tag: noindex” or ‘noindex’ detected in ‘robots’ meta tag”.
This method doesn’t ship the GoogleBot person agent, it makes use of the Google-InspectionTool/1.0 person agent string. Meaning if the server block is by IP handle then this technique will catch it.
One other angle to test is for the state of affairs the place a rogue noindex tag is particularly written to dam GoogleBot, you’ll be able to nonetheless spoof (mimic) the GoogleBot person agent string with Google’s personal Consumer Agent Switcher extension for Chrome or configure an app like Screaming Frog set to determine itself with the GoogleBot person agent and that ought to catch it.
Screenshot: Chrome Consumer Agent Switcher
Phantom Noindex Errors In Search Console
These sorts of errors can really feel like a ache to diagnose however earlier than you throw your arms up within the air take a while to see if any of the steps outlined right here will assist determine the hidden motive that’s chargeable for this challenge.
Featured Picture by Shutterstock/AYO Manufacturing
