I’ve had quite a few web site house owners attain out to ask for assist with indexing points recently. Most often, I’m discovering that Google has categorised its pages as “crawled-not presently listed” within the web page indexing report in Google Search Console.
In nearly each case, I’ve examined these pages have high quality points. They’re often “commodity content material” – basically rehashing what many others have already written on a subject with out providing something new or extra useful than what presently exists on-line.
On this article, I’ll share how I have a look at the crawled-currently not listed report in GSC. I’ll provide you with a software that can assist you discover the pages on this report that it is best to analyze additional. And I’ll provide you with some suggestions for bettering so you’ll be able to probably get better. I have to give honest warning, although. For many websites, if in case you have numerous pages you need listed, however they’re caught in crawled-currently not listed, restoration will probably be tough.
What Google Mentioned About Crawled-At the moment Not Listed At The Google Search Central Occasion In Toronto
I attended the Google Search Central occasion in April of 2026. The organizers requested us to not attribute quotes on to any Googler, however they did give us permission to share what was mentioned.
One presenter shared about how Search works. He mentioned that when Google crawls a web page it basically means they obtain it. Then, “If we predict it’s helpful we would put it in a database,” or in different phrases, in Google’s index.
Then he talked about what sorts of issues Google needs to place within the index. He mentioned that AI has made the brink for creating issues decrease. If anybody can create content material on something, then the kind of content material Google needs so as to add to their index is content material that provides two issues: private expertise, and data nobody else has.
He mentioned that if Google has crawled your web page and has determined to not index it, there could possibly be two causes:
1. There Might Be A Technical Situation
I’ve discovered this to be uncommon. Nonetheless, simply final week I reviewed a web site that had undergone a migration and all of their pages had been caught in crawled-currently not listed. Word: This isn’t the identical as “Found-not presently listed” which implies that Google is conscious of the pages, however has not but crawled them.
My first step was to investigate whether or not Google might see the content material on pages. I used the web page inspection software in GSC by clicking the magnifying glass subsequent to the url within the crawled-currently not listed checklist and clicked, “Take a look at Stay URL” Surprisingly, after I considered the dwell examined web page, all it confirmed was a heading, just a few boilerplate phrases and no content material in any way.
On this case, the location proprietor did certainly have a technical problem. Their robots.txt had this line, Disallow: /*?*. The thought was to dam crawling of urls with parameters like ?replytocom or ?utm_source. However, their new theme relied on these parameters for his or her CSS recordsdata and javascript in order that they had been basically blocking Google and all different search engines like google and yahoo from seeing most of their content material.
We now have since eliminated this block and really slowly, pages are beginning to seem again within the index once more.
In case your dwell check reveals Google can certainly see the content material in your pages, it’s most unlikely to be a technical problem that’s inflicting crawled-currently not listed issues.
I also needs to point out that some pages ought to be in your crawled-currently not listed checklist if they aren’t the canonical model. Should you see /feed/ pages or pagination or pages with url parameters, that is regular.
2. High quality
The Googler in Toronto went on to elucidate one other trigger for Google to crawl a web page and never index it. He mentioned it could possibly be as a result of “we checked out it and located it to not be good.” He mentioned that if hundreds have lined the very same matter they could resolve that your web page is unlikely to be helpful in Search. It could be that there are different choices which are extra in style or of higher high quality.
He additionally mentioned that typically Google experiments by permitting your web page to be listed for some time to see if customers prefer it, “We’re experimenting with seeing which one produces happier customers.” That may be a fairly wild assertion!
My guess is that if in case you have pages that you really want listed, however Google has them within the crawled-not presently listed bucket, then your important problem is said to commodity content material.
Commodity Content material Is The Most Doubtless Trigger
Google talked rather a lot about commodity content material at this occasion.
Commodity content material is content material that just about anybody might write a couple of topic. It’s typically repeating what already exists on-line on different websites. Non-commodity content material brings a singular viewpoint or has content material that others lack or can’t simply replicate. It often demonstrates first-hand data or expertise.
Take this text you’re studying proper now. Anybody might use AI to put in writing a useful article defining crawled-currently not listed pages. My article, nevertheless, talks about my expertise as an expert who’s paid to provide my opinion on this topic. I’ve shared the real-world technical instance above, I’ve shared first-hand data I discovered from attending a Google occasion, and I’m about to share my observations on pages which were deemed undeserving of indexing.
My Observations Of Pages Caught In Crawled-At the moment Not Listed
These pages are often not junk. They’re good, first rate articles – nearly as good because the pages that Google is rating. And that’s simply the purpose. The pages aren’t particular or any extra invaluable than what presently exists.
Right here is the method I take advantage of to investigate these pages.
To seek out the checklist, click on on “Pages” beneath Indexing in GSC. Then click on on crawled-currently not listed:
Under this, you’ll see an inventory of URLs to research. (Under, I’ll share extra a couple of software I’ve created that can assist you filter this checklist to see the URLs that actually matter.)
I’ll discover a URL on this checklist that actually is one which we would like listed.
First, I’ll seek for some queries that you’d count on the web page to rank for. On the SERP, there’s often an AI reply that could be very useful. Usually, a person will discover the reply to their query there. If so, then why would they wish to click on via to your web site to learn the very same factor?
I’ll develop the AI overview after which open up Gemini within the Chrome sidebar. Then I maintain down CTRL/Cmd and click on on the highest web sites linked to from inside the AIO. Should you do that when you’ve Gemini within the Chrome sidebar opened, you’ll discover these tabs get added to your Gemini dialog.
Then I kind “/” which opens up the abilities I’ve saved at chrome://abilities/ and select my Non-commodity verify. (Should you’re a member of my paid neighborhood, you will discover this full talent right here.)
This talent is a really lengthy immediate that appears at among the issues Google tells us its algorithms goal to reward in its documentation on creating useful content material, together with, however not restricted to:
- Does the content material present unique data, reporting, analysis, or evaluation?
- Does the content material present insightful evaluation or attention-grabbing data that’s past the apparent?
- If the content material attracts on different sources, does it keep away from merely copying or rewriting these sources, and as an alternative present substantial extra worth and originality?
- Does the content material present substantial worth when in comparison with different pages in search outcomes?
And Gemini provides me among the explanation why the pages linked to supply worth to the reader. Word: Typically pages are rating not due to their non-commodity worth however as a result of they’re an authoritative supply. If you’re a recognized authority, you may get away with a bit extra “commodity-ness.”
Now, we have to acknowledge that Gemini doesn’t have inside perception into Google’s rating techniques. It doesn’t know why sure pages are rating. What we are attempting to study here’s what varieties of issues could possibly be serving to a web page be worthy of presenting to searchers.
Then, I open up my shopper’s web page and immediate this, “Now analyze this web page in accordance with the identical standards.. This web page will not be rating properly. It’s our shopper. Please share the place you assume it’s missing. No must recommend enhancements at this level.”
Right here is the consequence for one crawled-currently not listed web page I used this immediate on.
John Mueller And Martin Splitt Mentioned Crawled-At the moment Not Listed In A Current Podcast
As I used to be about to publish this, Google revealed a Search Off the File Podcast on “The right way to learn the Indexing Report.” There’s rather a lot in right here, so I bolded the components that I believed had been essential.
This dialogue begins at 20:32 within the video
Chapter 9: Found vs. Crawled Not Listed: Is it a technical or web site high quality problem?
“And likewise, in case you add or change your web site or in case your web site could be very new, then you’ll be able to truly additionally use this report back to see a little bit bit how your web site goes via the completely different phases, as a result of in some unspecified time in the future, you’re going to see pages in Found presently not listed. Which tells you we all know they exist, however we haven’t truly visited them. And if we haven’t visited them, we will’t put them within the index. Crawled-currently not listed, which suggests we visited them and we didn’t put them within the index. And that may have all types of various causes. Would you say that’s typically or solely typically an indication of a high quality problem?
So, it’s positively the case if our techniques are critically frightened concerning the high quality of an internet site, that they are going to scale back the variety of pages that they index. As a result of if we’ve got robust considerations concerning the general high quality, then it doesn’t make a lot sense for our techniques to spend so much of time on the web site.
So, we’ll in all probability crawl rather a lot much less, we’ll index rather a lot much less, after which you’ll see issues like crawled, not listed or found, not listed, which from our perspective is principally our system saying, we learn about this, we checked out it, and as soon as we’re blissful, we are going to take one other look and see if we will index it. It’s not a lot that I’d say it is best to take these conditions and attempt to repair them. From a technical perspective, it’s not that it is advisable to repair this technical problem that Google will not be indexing this web page for the time being, however somewhat you nearly must whenever you acknowledge a much bigger sample like this, that Google will not be indexing quite a lot of your pages, and there’s no technical purpose, you nearly must take a step again and take into consideration the standard general.
And desirous about high quality is actually difficult as a result of quite a lot of occasions, it’s your web site, and it’s your child. And naturally, it’s the most effective child ever. However taking a step again and making an attempt to have a look at it with the eyes of somebody who will not be immediately concerned along with your web site. Typically that opens up some concepts for areas the place you’ll be able to enhance, the place perhaps if most of your web site is AI-generated and it labored for some time, it could be that individuals have a look at this AI-generated web site, they usually’re like, properly, I can inform that is AI-generated. There’s nothing distinctive or invaluable that’s accessible right here for me. That’s to not say that each one AI-generated content material is unhealthy, however typically you simply run throughout web sites the place you’re like, anybody might have written this. This tells me nothing. Yeah, that’s true. And I feel what makes this tough will not be solely the truth that clearly the best way you wrote it’s the method you thought was finest, and that’s why you assume it’s top quality, after all. In order that’s actually, actually exhausting to step out of your individual perspective. However typically, there’s additionally a lot different stuff that’s simply nearly as good. So, why would we add it to the index.
After which that may let you know, like, perhaps this content material isn’t as invaluable as I believed it was as a result of different persons are overlaying the identical factor. After which what’s the worth of this model of it being within the index? Yeah that’s true. I really feel we might have an entire podcast about high quality. I feel perhaps one different factor that’s value mentioning almost about high quality is it’s not simply the textual content. So quite a lot of occasions individuals will say, properly, my textual content is exclusive, or my articles are good, they usually’re packaged in a web page that’s horrible to entry, the place anybody who, after they attempt to load it like their pc fan spins up they usually’re like, “Oh my gosh, I’ve to run away to verify my pc doesn’t explode.” So perhaps that’s an excessive case, however you’ve all seen these pages the place principally the textual content is there, however it’s nearly hidden away, hidden behind adverts, hidden behind interstitials, hidden behind different issues which are shifting and coming and going, perhaps hidden beneath a bunch of filler content material, which we typically see, for instance, with recipes the place there’s this actually lengthy story on high that perhaps most individuals don’t actually care about. After which the recipe comes. These are all of the sorts of issues the place the general high quality is far more than simply that piece of textual content that you just say, that is my important content material. That is what Google needs to be counting for my web site. And from our perspective, we nearly should have in mind the total expertise on a web page, as a result of that’s what customers see. It’s not that customers go to an online web page and activate some magic mode that simply pulls out the textual content, however somewhat they’ve the total expertise of this web site with all the 3D, 4D animations, and all the pieces. I agree very a lot. Agree, oh my God.”
How Can You Repair This Situation?
Oh boy, that is the robust a part of this text, as a result of in quite a lot of circumstances, I really feel that this can be very tough to get pages out of crawled-currently not listed. I imply, if extreme adverts and filler are in charge, there are apparent issues to enhance on there. If there’s a technical problem, repair it and request reindexing by way of GSC – or simply be affected person and wait until Google tries to crawl your pages once more.
If it’s a high quality problem, although, you’re probably going to should put important effort into bettering these pages.
For a lot of websites that I analyze, their superpower prior to now was the power to cowl a subject completely. Currently, there’s a development to not solely cowl a subject, however to anticipate all the fan-out queries and canopy these as properly. This was talked about within the Google Search Central occasion just a few occasions. If you’re creating a great deal of content material based mostly on this technique, you run the chance of dealing with a scaled content material penalty. I can’t show this but, however I think that the June 2026 spam replace impacted quite a few websites that had been creating commodity content material at scale. If that is true, you gained’t see a handbook motion in GSC. You’ll simply see a drop in natural visitors with no clarification.
I concern for lots of search engine marketing companies as a result of for a lot of, your primary software in your toolbox is content material creation. AI has made it a lot simpler to cowl content material on any topic. I’m not towards utilizing AI to assist with content material creation. However, in case your search engine marketing firm can use AI to create content material in your subjects, then it’s probably not unique, insightful, and considerably extra useful than what presently exists. There are exceptions. I do know of some companies that use intelligent AI pipelines to interview a enterprise, extract its related expertise, and switch that into good, unique content material.
Though I don’t suggest utilizing AI to put in writing your content material for you with none human enter, I do assume you’ll be able to brainstorm with AI to assist enhance it. The issue, although, is that the options would require effort. The phrase “effort” is used 120 occasions in Google’s High quality Rater Pointers. It would be best to discover methods to attract out of your expertise to create content material that provides to the physique of information that presently exists in your subjects.
Do that easy immediate. Give your content material to an LLM or open up Gemini within the sidebar and ask this: “Is that this content material prone to be thought-about commodity content material?”
I simply opened up Gemini in Google Docs and requested about this very article you’re studying now:
Subsequent, do this for some concepts.
“Give me 20 concepts that assist me draw from my first-hand expertise to make this text much more useful, and considerably higher than the rest that exists on this matter on the internet.”
Rattling, there are some good concepts in right here.
Some Instruments To Assist You Assess Your Crawled-Not At the moment Listed Pages
I created a few instruments utilizing Google’s Antigravity. You’ll find them at instruments.mariehaynes.com.
There are two new instruments:
1. Filter your crawled-not presently listed URLs. Export your crawled-not presently listed URLs from GSC. Should you export as CSV, open the zip file and discover the desk.csv file. You possibly can add it to this software, and it’ll strip out /feed/ pages and others in an effort to see and click on on the URLs that you just wish to examine.
2. GSC Index Checker. You have to to log in to your Google account to make use of this software, however know that I don’t see any of your knowledge. It’s going to verify an inventory of URLs to see what their indexing standing is. You possibly can select from the latest pages in your sitemap, paste an inventory of URLs in manually, or have the software seize your top-trafficked pages from GSC.
What you’re searching for right here is whether or not these pages that matter to you’re certainly listed, or whether or not they’re caught in crawled-currently not listed.
I hope this text helps! Google does appear to be getting extra strict on what it’s indexing lately.
Extra Assets:
Learn Marie’s publication, AI Information You Can Use. Subscribe now.
Featured Picture: Tetiana Yurchenko/Shutterstock
