Google Search Advocate John Mueller responded to a query in regards to the “Web page Listed with out content material” error in Search Console, explaining the problem usually stems from server or CDN blocking relatively than JavaScript.
The change befell on Reddit after a person reported their homepage dropped from place 1 to place 15 following the error’s look.
What’s Taking place?
Mueller clarified a standard false impression about the reason for “Web page Listed with out content material” in Search Console.
Mueller wrote:
“Often this implies your server / CDN is obstructing Google from receiving any content material. This isn’t associated to something JavaScript. It’s often a reasonably low degree block, typically primarily based on Googlebot’s IP tackle, so it’ll most likely be inconceivable to check from outdoors of the Search Console testing instruments.”
The Reddit person had already tried a number of diagnostic steps. They ran curl instructions to fetch the web page as Googlebot, checked for JavaScript blocking, and examined with Google’s Wealthy Outcomes Check. Desktop inspection instruments returned “One thing went improper” errors whereas cellular instruments labored usually.
Mueller famous that normal exterior testing strategies gained’t catch these blocks.
He added:
“Additionally, this could imply that pages out of your website will begin dropping out of the index (quickly, or already), so it’s a good suggestion to deal with this as one thing pressing.”
The affected website makes use of Webflow as its CMS and Cloudflare as its CDN. The person reported the homepage had been indexing usually with no current adjustments to the positioning.
Why This Issues
I’ve coated this sort of drawback repeatedly over time. CDN and server configurations can inadvertently block Googlebot with out affecting common customers or normal testing instruments. The blocks usually goal particular IP ranges, which implies curl exams and third-party crawlers gained’t reproduce the issue.
I coated when Google first added “listed with out content material” to the Index Protection report. Google’s assist documentation on the time famous the standing means “for some purpose Google couldn’t learn the content material” and specified “this isn’t a case of robots.txt blocking.” The underlying trigger is nearly at all times one thing decrease within the stack.
The Cloudflare element caught my consideration. I reported on an analogous sample when Mueller suggested a website proprietor whose crawling stopped throughout a number of domains concurrently. All affected websites used Cloudflare, and Mueller pointed to “shared infrastructure” because the possible wrongdoer. The sample right here seems acquainted.
Extra just lately, I coated a Cloudflare outage in November that triggered 5xx spikes affecting crawling. That was a widespread incident. This case seems to be one thing extra focused, possible a bot safety rule or firewall setting that treats Googlebot’s IP addresses in a different way from different visitors.
Search Console’s URL Inspection software and Reside URL take a look at stay the first methods to establish these blocks. When these instruments return errors whereas exterior exams cross, server-level blocking turns into the possible trigger. Mueller made an analogous level in August when advising on crawl fee drops, suggesting website house owners “double-check what really occurred” and confirm “if it was a CDN that truly blocked Googlebot.”
Wanting Forward
In case you’re seeing the “Web page Listed with out content material” error, test the CDN and server configurations for guidelines that have an effect on Googlebot’s IP ranges. Google publishes its crawler IP addresses, which might help establish whether or not safety guidelines are concentrating on them.
The Search Console URL Inspection software is essentially the most dependable option to see what Google receives when crawling a web page. Exterior testing instruments gained’t catch IP-based blocks that solely have an effect on Google’s infrastructure.
For Cloudflare customers particularly, test bot administration settings, firewall guidelines, and any IP-based entry controls. The configuration could have modified by computerized updates or new default settings relatively than handbook adjustments.
