Crawl Errors: What They Are and How to Fix Them

A crawl error means a search engine tried to fetch a URL on your site and the request failed. Not that the page ranks poorly, not that it's missing from results for some content reason - the fetch itself broke. The server didn't respond, the URL returned a 404, a redirect looped back on itself, or robots.txt told the crawler to stay away. Search Console groups these failures by type, and each type points to a different fix.

That's worth separating from an indexing problem right away, because the two get treated as one thing constantly and they aren't. A page can be crawled perfectly - server responds, content downloads clean, 200 status - and still never make it into the index, because Google judged the content wasn't worth indexing, or it's a near-duplicate of another URL, or a noindex tag told it not to bother. That's a decision made after a successful crawl. A crawl error happens earlier and is more mechanical: the fetch itself didn't complete. Fixing one doesn't guarantee indexing, but leaving it unresolved means Google may keep retrying a URL that will never load, and on a large site that's effort competing with crawl budget that could go toward pages worth finding.

Where You Actually See Crawl Errors

Three sources, and they don't agree on timing. Search Console's Page indexing report lists statuses including "Not found (404)", "Server error (5xx)", "Redirect error", and "Blocked by robots.txt". But it also lists "Crawled - currently not indexed" and "Discovered - currently not indexed" right alongside those, and those two are indexing decisions, not crawl failures - the fetch either succeeded or hasn't happened yet because of budget, not because it broke. Don't file them under the same fix list.

The Crawl stats report, under Settings, is closer to a server-eye view: total requests over time, a breakdown by response code, and by purpose - discovering new URLs versus refreshing known ones. It's aggregate rather than per-URL, so it's better for spotting a spike than tracking down one broken link.

Raw server logs are the most honest source. Search Console data lags a few days before an error even surfaces, filtered through its own reporting logic. Your access logs show every crawler hit as it happens, the exact response code and timing, and enough detail to confirm a request actually came from Google's IP ranges rather than a bot spoofing the user agent. For anything urgent, logs beat waiting for the report to catch up.

Server Errors (5xx): Fix These First

A 5xx means the server failed to produce a response at all - not a problem with the page's content, a problem with the request never completing. The crawler asked, and something on your end broke, timed out, or refused to answer.

The usual causes are capacity-related: a process crashing under load, a database connection pool exhausted during a traffic spike, hosting resource limits getting hit, or a plugin throwing an intermittent 500. Start diagnosis by matching timestamps from Search Console or your access log against your application error log - it almost always names the actual failure, whether that's a timeout, a memory limit, or an unhandled exception.

How urgent this is depends on the pattern. A cluster of 500s during a deploy that clears up on its own isn't worth chasing. A steady trickle over days or weeks is different: persistent server errors can erode Google's confidence in the site's reliability, and it may respond by crawling more cautiously, slowing discovery of new content sitewide, not just on the failing URLs. That's why this class gets fixed before individual broken links.

DNS and Connectivity Failures

A step earlier than a server error - the crawler couldn't reach the server at all. DNS didn't resolve, the connection timed out before any response came back, or a TLS handshake failed. Google often reports these simply as the site being unreachable rather than a per-URL error.

Causes include a DNS provider outage, a misconfigured record after a migration, propagation lag after a nameserver change, an expired certificate, or - more common than it should be - a firewall or bot-management rule overzealous enough to block Googlebot's own IP ranges along with the traffic it was meant to stop.

Check DNS resolution from more than one location, confirm the certificate is valid, and if there's a CDN or WAF in front of the site, check its bot rules specifically. These failures are rarer than the other classes, but the most severe when they hit, because they don't affect one page - they can stop crawling of the whole site until resolved.

404s: Most Are Fine, Some Aren't

A 404 means the page genuinely doesn't exist there, and in most cases that's exactly correct - not a problem to fix. A discontinued product, an old campaign page, content removed on purpose: Google drops these from the index on its own, and chasing every dead URL on the list wastes time reviving content nobody needs back.

The ones worth acting on are the ones something still points at. Search Console's report for a 404 URL includes a referring page when one is known. If that referrer is your own site, it's a broken internal link - a stale href, an old sitemap entry - and it's worth fixing regardless of what happened to the destination. If the referrer is external, someone usually linked to a URL that moved without a redirect, which is link equity going nowhere for the cost of a single 301.

No referrer at all means an old URL nobody links to anymore, internally or externally. Leave it. It isn't costing anything, and inventing a redirect target for it doesn't serve a real intent.

• Broken internal link → fix the link on your own site, regardless of the destination
• Dead URL with external backlinks or past traffic → 301 redirect to the closest live equivalent
• Old URL with no referrer and no traffic → leave it as a 404, this is normal background noise

Redirect Chains and Loops

Two problems get filed under "redirect error". A chain redirects through several hops before the final page - A to B to C to D - and each hop costs a separate crawl request and dilutes the signal along the way. A loop is worse: A sends to B, B sends back to A, and the crawler never resolves to real content before giving up.

Chains usually build up over time: a site moves http to https, then www to bare domain, then restructures its URLs, and each migration stacks its own redirect on the last instead of anyone updating the original link to point straight at the current destination. Loops tend to come from two rules fighting each other - a trailing-slash rule sending traffic one way and a non-trailing-slash rule sending it back, or a redirect defined in the server config conflicting with a separate one defined in the application, with nobody remembering the first one exists.

Trace the actual path with a header-checking tool rather than assuming, and check both layers where redirects tend to live, since they're often maintained by different people at different times. The fix is the same either way: point every redirect straight at its true final destination in one hop, and remove leftovers from old migrations once nothing depends on the intermediate step.

Blocked by Robots.txt

This status means the crawler saw the URL, checked robots.txt, and was told not to fetch it. Sometimes that's the intent - staging environments, admin paths, internal search results, or the endless combinations a faceted filter can generate are reasonable things to keep a crawler away from.

It's also an easy accident. A broad Disallow rule written for one folder can unintentionally match a path prefix shared by content you actually want crawled - a rule meant to catch /blog-drafts/ written broadly enough to also catch /blog/. A wildcard left over from staging and never removed after launch is another common one. Test the specific URL against the specific rule rather than eyeballing the file, since the syntax is easy to misread by hand.

One nuance worth knowing: robots.txt blocks crawling, not indexing. A blocked URL can still appear in results, as a bare link with no snippet, if enough other pages link to it - Google knows the URL exists even though it was never allowed to fetch what's on it. To actually keep something out of results, that needs a noindex tag on a page the crawler can reach, or removing the content outright.

Soft 404s

A soft 404 returns a normal 200 OK status while the content is effectively nothing - an out-of-stock notice with no other content, a zero-result search page with a mostly blank template, a lone "this item is no longer available" message. The server says everything's fine; the page says there's nothing here. Google evaluates the actual content, not just the status code, and treats a page like this the same as a real 404.

It's the sneakiest class on this list because status-code monitoring won't catch it - uptime checks and scripts that only look at the response code report a clean 200 and move on. Finding these takes either Search Console flagging them or someone actually looking at what renders.

The fix depends on intent. If the content is genuinely and permanently gone, send a real 404 or 410 instead of a 200, so the signal matches reality. If the URL should stay live - a product that will restock, a category that's temporarily empty - give the page enough real content to not be functionally blank: related items, a restock notice, links back into the category.

Prioritizing a Long List, and Confirming a Fix Took

A site of any size will always show some number of crawl errors, and driving that to zero isn't the goal - most of it is background noise from content that was deliberately retired. Treat the list as something to triage, not a checklist to clear out.

Work sitewide problems first: a spike in 5xx errors or a connectivity failure affects crawling everywhere, so it outranks any single broken URL. After that, filter 404s and redirect errors by whether the URL has a referring internal link, an external backlink, or a history of organic traffic - those represent real cost. Anything with none of the three can wait indefinitely. Blocked-by-robots.txt entries only need attention when the blocked URL is one you actually want found; if it's doing its job on a staging path, it's correct as-is. Soft 404s are worth a look wherever the affected page type should convert or rank.

Once a fix is live, don't take your own testing as proof. Open the URL in the Page indexing report and use Validate Fix, which tells Search Console to re-check it and report back - that can take days to a couple of weeks, and a passed validation means Google re-fetched the URL itself and got the expected result, not just you eyeballing a browser. For a faster signal, check server logs for the next crawler hit on that exact URL and read the response code directly - that confirms the fix at the source, ahead of Search Console's own cycle.